Change:
// initializeRuntimeDependentModules will initialize internal modules that require the container runtime to be up. func (kl *Kubelet) initializeRuntimeDependentModules() { if err := kl.cadvisor.Start(); err != nil { <del> kl.runtimeState.setInternalError(fmt.Errorf("failed to start cAdvisor %v", err)) </del> <ins> // Fail kubelet and rely on the babysitter to retry starting kubelet. // TODO(random-liu): Add backoff logic in the babysitter glog.Fatalf("Failed to start cAdvisor %v", err) </ins> } // eviction manager must start after cadvisor because it needs to know if the container runtime has a dedicated imagefs if err := kl.evictionManager.Start(kl, kl.getActivePods, evictionMonitoringPeriod); err != nil {

Discussion:
master-informing ci-crio-cgroupv1-node-e2e-conformanceci-crio-cgroupv1-node-e2e-conformance.Overall kubetest.Node Tests E2eNode Suite.[It] [sig-node] Container Runtime blackbox test on terminated container should report termination message from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] E2eNode Suite.[It] [sig-node] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance] E2eNode Suite.[It] [sig-node] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance] E2eNode Suite.[It] [sig-node] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance] ! 2023-05-15 12:08 PDT E2eNode Suite: [It] [sig-node] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance] E2eNode Suite: [It] [sig-node] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance] E2eNode Suite: [It] [sig-node] Container Runtime blackbox test on terminated container should report termination message from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] E2eNode Suite: [It] [sig-node] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance] Node Tests No response /sig node cc
~partially blocked on edit: I got the compatibility matrix confused, this is not blocked on that
/reopen Better to close once we see it green
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /triage accepted Head over to the Slack thread if you want to see more discussion:
Working on a fix in
Green tests:

Recently we observe increased number of large-cluster startup failures. After very little digging it seems to be a problem in kubelet/cAdvisor/docker. We're seeing the following log repeat itself until the timeout: P0 as it happened at least 3 time in last 2 days. cc
Those particular logs come from kubelet on master machine in this run:
- wild theory. is it possible that the kube-logrotate.service has to be running before Looking at: If so, we could add a in the for Right?
caused kubelet not being able to sync pods. Could it be possible that kubelet-logrotate get removed while cadvisor was traversing the directory recursively to add all of them? If so, we should ignore such errors since the file doesn't exist anymore. /cc
I didn't see in my cluster. Only
logrotation is triggered by and only runs hourly. I assume it was run while cadvisor was starting up, so cadvisor saw the short-lived cgroup. All my comments above are just speculation :)
Offline discussed with and should be the root cause. In this master, we can see: It means that cadvisor did see during . However, because is only a short lived service, when cadvisor tried to , it has already been deleted. That's why we get the error: I've checked some normal clusters, cadvisor doesn't see at all, because it is too short lived. This is only about timing. :) Anyway, cadvisor shouldn't fail when the cgroup is missing at watching, which sometimes does happen. I'll send a fix soon.
Thanks a lot for this debugging! And waiting for the fix.
FYI, I've sent a PR to fix cadvisor side Once that PR is included in kubernetes, we can close this one. :)
Awesome - thanks
Shouldn't we run into the same issue for 1.3 GKE release? If we do, we need to patch 1.3. cc/