Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On cgroup v1 when using systemd cgroup driver, kubelet regularly freeze and thaw its control groups, and that is causing issues further down. No freeze Running the command below should never print anything, since should always be . However, when using master/v1.22.0, this returns regularly. When using runc v1.0.0 or earlier, this can cause containers to end in a permanently frozen state. This state is tricky to debug, since no log output is given. To see if an environment is having problems, one can use: More info in and Kubernetes version (use ): : introduced in Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /priority critical-urgent
Thanks for opening this. IIUC, this issue is limited to: cgroup v1 systemd cgroup driver runc 1.0.1+
systemd cgroup driver runc 1.0.1+ Well, yeah, kinda. It is a bit of a matrix As long as cgroup version = v1 cgroup driver = systemd the freezing will occur. The freezing is unnecessary, and creates a big overhead on machines with huge number of processes. However, if runc (the binary used by the container runtime) is v1.0.0 or before, the permanent freeze of containers can occur. If using runc (the binary) v1.0.1, the permanent freeze is mitigated and does not occur. So in one way, as long as the user is using runc binary v1.0.1 or later, this is not completely critical.
I am seeing a lot of flakiness lately in systemd node e2e jobs using crio, Almost all of them fail with container not being able to come up with following error, That seems to be coming from I am wondering if that flakiness is due to frozen cgroups issue reported in this issue. Unfortunately I do no have access to the system to query sysfs to see if cgroups is in FROZEN state.
I think is right and the flakes we're seeing in the CRI-O jobs seems to be related. runc considers a frozen cgroup state as paused container: This means that in our test, the container creation works as intended, but the fails sometimes which causes test to flake and container to get stuck in the container creation error. The code in runc seems intentional and exists since a long time, are we aware of any recent changes in k/k to this kind of cgroup handling?
cc
Yes that looks like a result of this issue. Thanks for the detailed information! This was issue in k8s (the freezing) was introduced in As disucssed here: , freezing would cause issues, and the runc bump was merged before it was properly tested. If you are talking about runc as in the binary, there was previously an issue with the definition of a "paused/frozen" container; should give some more context. That issue is fixed in runc v1.0.1 (the binary).
/triage accepted
This issue will be fixed by or any similar way to avoid the freeze.
cc:

Related discussion:
text: err := kl.syncPod(pod, dockerContainers) if err != nil { glog.Errorf("Error syncing pod, skipping: %v", err) <ins> record.Eventf(pod, "", "failedSync", "Error syncing pod, skipping: %v", err) </ins> } }) }