Change:
This test is failing in gke serial suite: http://k8s- /sig node /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign also cc in case you already have some idea xref
/milestone v1.10
this also failed in upgrade jobs: http://k8s- xref
cc please update
/priority critical-urgent
Both are failing because the runtime cpu usage is exceeding our limit. In both cases, we are seeing 350m usage and expecting <200m usage
is this indicative of a general increase in resource usage, as per
I don't think so. This is a specific increase in the runtime (docker in this case) cpu usage
This seems like a legit issue. Docker's 95th% cpu usage has increased on all nodes since 2/27, when no test/user pods are running. The tests started failing when this PR landed: The PR bumped the fluentd version to v3.0, and I kept seeing fluentd liveness probes failures in the kubelet log. Not sure if it's related, but it's worth looking into.
^^^^^^^^ /cc
we should investigate this with according to this report, the latest version of fluentd using our image is using more CPU that is acceptable. I didn't see this behavior in my tests, I wonder if it has anything to do with the parser fixes we when we moved the multiformat into a plugin block.
Just ran my perf tests against the latest configuration, and I'm seeing 600m usage for CPU. Will investigate how we can optimize this with Ling today.
the test checks "docker daemon's" cpu usage when only system add-on pods are running, and the usage increases on all nodes. fluentd is one of the few pods that run on every node and the failure coincided with the fluentd version upgrade. On top of that, fluentd frequently fail its own liveness checks. Here are a few things that are worth checking IMO: Revert the PR (or run the test in a separate test cluster) to confirm whether fluentd is the culprit. Confirm whether the increase of "docker daemon" cpu usage is intended. Why is fluentd failing the liveness checks so frequently? Is that expected. I've seen a few consecutive failures (although they were under the threshold, which is 3). Is the newer version of fluentd logging more? Docker daemon is responsible for writing the container stdout/stderr logs to the disk, and the increase of logs could cause cpu spikes. Just check the fluentd logs and see if there is anything abnormal.
/assign (since I can't assign it to )
After doing some investigation, we discovered that this was due to an update in Fluentd v0.14 in how it manages its buffers. I've submitted a PR that resolves the liveness probe which should prevent the extreme spikes in CPU when the fluentd process is forcefully shut down.
update? cc AI needed here
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
The two PRs that should resolve this issue have been approved and will be merged soon.

Discussion:
then exit 1; fi; <del> LAST_MODIFIED_DATE=`stat /var/log/fluentd-buffers | grep Modify | sed -r "s/Modify: (.*)/1/"`; LAST_MODIFIED_TIMESTAMP=`date -d "$LAST_MODIFIED_DATE" +%s`; if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $STUCK_THRESHOLD_SECONDS` ]; </del> <ins> touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [[ -z "$(find /var/log/fluentd-buffers -type f -newer /tmp/marker-stuck -print -quit)" ]]; </ins> then rm -rf /var/log/fluentd-buffers; exit 1; fi; <del> if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $LIVENESS_THRESHOLD_SECONDS` ]; </del> <ins> touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [[ -z "$(find /var/log/fluentd-buffers -type f -newer /tmp/marker-liveness -print -quit)" ]]; </ins> then exit 1; fi;