Change:
Looks like we just got a spike of a new run failure message in master: Seen in } pod "oidc-discovery-validator" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-03-31 05:35:20 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-03-31 05:35:20 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-03-31 05:35:20 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-03-31 05:35:20 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:172.18.0.2 HostIPs:[{IP:172.18.0.2}] PodIP:10.244.1.130 PodIPs:[{IP:10.244.1.130}] StartTime:2022-03-31 05:35:20 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:oidc-discovery-validator State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:128,Signal:0,Reason:StartError,Message:failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write : write /sys/fs/cgroup/rdma/kubelet/kubepods/besteffort/pod4c5127ae-797f-4b89-9aa9- no such device: unknown,StartedAt:1970-01-01 00:00:00 +0000 UTC,FinishedAt:2022-03-31 05:35:21 +0000 UTC,ContainerID:containerd://61b3e1f7568f23bb1503c2309e9e254c1ac0103d0de059958f9555ff6548b5c8,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 ContainerID:containerd://61b3e1f7568f23bb1503c2309e9e254c1ac0103d0de059958f9555ff6548b5c8 Started:0xc000d223ea}] QOSClass:BestEffort EphemeralContainerStatuses:[]}` /milestone v1.24 /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kubelet log: containerd log: I prefer to think it is a container-related issue: .
cc
/triage accepted /priority critical-urgent /cc
Was the runc (or containerd) binary updated on these jobs?
Support for RDMA controller was indeed to runc 1.1 (in ). The error message says that write failed. That write happens right after mkdir, which (apparently) succeeded.
kubelet log also shows (this is the earliest mention of rdma): Note that support for hybrid unified hierarchy was also appeared in runc 1.1.
The removal failure happens because allegedly there is a process(es) left in rdma and unified cgroups which prevent the cgroup removal. I can't figure out why this can ever happen (kubelet does not know anything about rdma or unified, but it should not break things). My preliminary theory is, inability to write a pid to rdma is caused by too many rdma cgroups. In any case, we should figure out why rdma and unified are not empty upon removal. Following the source code, kubelet kills all the processes in these cgroups before trying to remove them, so I am puzzled.
Any possibility of processes stuck in 'D' state?
So, I some debug in to see what is going on. Here is an excerpt from the kubelet log: Apr 05 01:54:38 kind-worker kubelet[261]: time="2022-04-05T01:54:38Z" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/unified/kubelet/kubepods/burstable/pod1883213d8fec799ee2b7bf9f2185a5c7/5b078c521eefb476090c430cac51c128fabcd9094ebcaf0fa225d2b366c13c39: device or resource busy" Apr 05 01:54:38 kind-worker kubelet[261]: time="2022-04-05T01:54:38Z" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/rdma/kubelet/kubepods/burstable/pod1883213d8fec799ee2b7bf9f2185a5c7/5b078c521eefb476090c430cac51c128fabcd9094ebcaf0fa225d2b366c13c39: device or resource busy" Apr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "Failed to delete cgroup paths" cgroupName=[kubelet kubepods burstable poda67d4606-bf19-4290-b354-] err="unable to destroy cgroup paths for cgroup [kubelet kubepods burstable poda67d4606-bf19-4290-b354-] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubelet/kubepods/burstable/poda67d4606-bf19-4290-b354- rdma:/sys/fs/cgroup/rdma/kubelet/kubepods/burstable/poda67d4606-bf19-4290-b354-]" pids=[] Apr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "KKK subsystem info" name="" path="/sys/fs/cgroup/unified/kubelet/kubepods/besteffort/pod15fc6eea-8cb0-4d22-8f38-" pids=[0 0] pErr=<nilApr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "Failed to delete cgroup paths" cgroupName=[kubelet kubepods burstable pod1e2d8aa5-e95f-4ccd-8a58-] err="unable to destroy cgroup paths for cgroup [kubelet kubepods burstable pod1e2d8aa5-e95f-4ccd-8a58-] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubelet/kubepods/burstable/pod1e2d8aa5-e95f-4ccd-8a58- rdma:/sys/fs/cgroup/rdma/kubelet/kubepods/burstable/pod1e2d8aa5-e95f-4ccd-8a58-]" pids=[] Apr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "KKK subsystem info" name="rdma" path="/sys/fs/cgroup/rdma/kubelet/kubepods/burstable/pod1883213d8fec799ee2b7bf9f2185a5c7" pids=[] pErr=<nilApr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "KKK subsystem info" name="rdma" path="/sys/fs/cgroup/rdma/kubelet/kubepods/pod9daba603-dc32-4c18-bb17-" pids=[] pErr=<nilApr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "KKK subsystem info" name="" path="/sys/fs/cgroup/unified/kubelet/kubepods/burstable/pod1883213d8fec799ee2b7bf9f2185a5c7" pids=[0 0] pErr=<nilApr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "KKK subsystem info" name="" path="/sys/fs/cgroup/unified/kubelet/kubepods/pod9daba603-dc32-4c18-bb17-" pids=[0 0] pErr=<nilApr 05 01:54:38 kind-worker kubelet[261]: I0405 01:54:38. 261 ] "Failed to delete cgroup paths" cgroupName=[kubelet kubepods besteffort pod15fc6eea-8cb0-4d22-8f38-] err="unable to destroy cgroup paths for cgroup [kubelet kubepods besteffort pod15fc6eea-8cb0-4d22-8f38-] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubelet/kubepods/besteffort/pod15fc6eea-8cb0-4d22-8f38- rdma:/sys/fs/cgroup/rdma/kubelet/kubepods/besteffort/pod15fc6eea-8cb0-4d22-8f38-]" pids=[] All this means that there are no processes in RDMA cgroup, yet it can not be removed; for some reason returns the list with two in it for unified controller. This is probably not related to this issue, but I am looking into it; despite no processes in cgroups, they can not be removed. Adding more debug to My next two suspects are KIND and the kernel. As for KIND, I looked at the sources of the script that prepares cgroups, and found nothing bad.
Looks like it's not it, show no entries (nor any in the subdirectories) -- see the previous comment.
the release 1.24 bug triage shadow. While the test freeze phase is cut off tomorrow, do you think this issue will still be included in the current release?
Until the issue is understood, it should remain in the milestone
/assign
lets catch up on why rdma is even an available controller on this host. rdma isnt in this allowed list:
This "allowed list" is merely a way to specify controllers that must be present (I guess its naming is slightly misleading). IOW, the code you refer to ensures that memory, cpu etc paths do present. It has nothing to do with rdma or unified. runc creates cgroups for all supported controllers/subsystems (and add containers to all of them). What's unclear is why these cgroups can't be removed during destroy. I am still looking at it in (feeling under the weather today so it takes longer).
One thing worth trying may be to see if we don't join the rdma controller, do we still hit the issue. Maybe runc can skip joining it unless rdma settings are specified? (It is a relatively newer controller which hasn't been tested as much within containers so we could avoid potential bugs there.)
understood. rdma as an enabled cgroup controller on a target host for kubelet execution is what was new to me so I was wondering if there was a change to the test operating system configuration beyond just runc adding awareness.
RDMA cgroup requires a kernel config parameter to be set. It is obviously set in Ubuntu kernels. In Fedora 35 kernels, is not set. Here's from my machine: On CentOS Stream 9 kernel, it is set:
Not joining RDMA in vendored libcontainer did not help. It might help in runc binary (which I haven't done). Looking into the underlying cause.
OK, my current theory is KIND + Docker is to blame. It seems that cgroups created are leaked to a parent namespace (which is only true for and , as others are properly fixed in entrypoint). A workaround can be to not mount /sys/fs/cgroup/rdma and /sys/fs/cgroup/unified to inside KIND. Is there someone working on KIND who can look into it? The modifications should belong to , and it seems that has recently modified this.
In I ran some test jobs in non-KIND environment, and there are no failures to remove rdma or unified cgroups (see e.g. in ). One more non-KIND test job is still running (). What we have here is a chicken and egg problem. Docker, which is used to run a KIND container, is not aware of rdma or unified hierarchies, and thus putting the container into the root rdma and unified cgroups. A way to work around it is to not expose /sys/fs/cgroup/rdma and /sys/fs/cgroup/unified inside the KIND container -- I hope that the entrypoint script can do that. The alternative is to not do any testing with KIND until docker and kind are rdma- and unified- aware.
how does one typically disable these two cgroups? and ? are there any examples of doing something similar in an entrypoint?
No messages about cgroup removal errors from this one either, so this indeed looks like KIND cgroup setup
I look into what is done for cgroup v1 in entrypoint, and I see that is determined from the cpu contrroller (), assuming all the other controllers have a similar paths. list is determined from the above path () is done with every subsystem from the . I suspect that list does not include neither rdma nor unified, because Docker is not aware of these subsystems and therefore does not create special mounts for it. Here's how it looks on Ubuntu 21.10 (used by kind tests) with default installed from Ubuntu repos: As you can see, no subdirectory is created for rdma controller (and thus step 2 above does not result in having in . It's more complicated for unified (the one that starts with ), but it is not mounted either. Therefore, no setup is done for there cgroups. The proper fix to this situation is to make Docker rdma- and unified-aware, which is definitely out of scope for this release (and/or project). My guess is we can try something simple like
I have semi-tested this on a Ubuntu 21.10 box (although I do not see inside a container), and it seems my proposal should work.
cc: for the long term solution
Could you open an issue (in moby/moby?) with details?
Redid the tests with the version of docker as used in kind tests (apparently the one from Docker repos, although for some reason the client is different), same results (i.e. it looks like it should work). The above umount/rmdir lines can be either to the beginning or the end of . Here's a PR:
My guess is, using newer runc will automatically fix this (I am going to check it), so no changes to moby will be necessary.
Testing shows my guess was was right -- once docker is switched to use runc 1.1, the rmda and unified subsystems become properly scoped. Here's what I did (Ubuntu 21.10, docker installed from repo): Note that and are not properly scoped (like is, under ), also shows unified hybrid cgroup () scoped to containerd (i.e. it is also not properly scoped). Now let's try the same with runc 1.1.1: As you can see, now rdma and unified hybrid are properly scoped. Note that is still not, as runc is not aware of it. </details
Now that we've narrowed this down to an issue with kind/our test infrastructure setup, it looks like this isn't a regression that would block 1.24. can you confirm? If so I will decrease the priority and remove this from the milestone.
So, the alternative to kubernetes-sigs/kind is to use runc 1.1.x when creating a KIND container (i.e. on the host). Apparently runc comes from package from repo (), and comes with runc 1.0.3.
Forgive my ignorance, but does this go beyond test setup and mean consumers of 1.24 need to ensure they are running runc 1.1.x? Wondering if we need to include this in release notes or (less as a release blocker and more as a potential action-required for users/admins)
I have vague memories of pinning to newer runc versions in the past ( and )... not sure if that's an option here
No, I believe we're still compatible with runc 1.0. The only case is, if someone wants to run kubernetes inside a container (like we do in KIND tests here), they have to ensure that the runtime outside of that container is not older than the runtime inside the container.
Yes, to my best understanding this is an issue with the CI env used, not a kubernetes issue.
Oh, and the issue will re-occur if/when we add cgroup support to runc (the workaround in should prevent this though).
I agree we should include this in known issues; I had just suggested that in our internal slack. I'm not sure how common as a production use case it is to run Kubernetes inside a container, where the runtime on the inside is newer than the runtime on the outside. We'll try to draft something for SIG Release/Docs. /milestone clear /remove-priority critical-urgent /priority important-soon
One thing that is important to mention. This issue is not related to vendor bump of runc 1.1, it is related to switching to use runc 1.1 binary (brought together with containerd 1.6) in CI. In other words, this is not related to kubernetes v1.24, and older versions (and newer versions) would fail the same way. Proof is e.g. this CI job: It uses k/k master at revision (see ), which has runc 1.0.3 in Now, it says containerd 1.6.2 (in ), which means the runc binary used in 1.1.x. Finally, if we take a look at (e.g. ), we can see a bunch of warnings like this: (interestingly, there are no issues with unified ... must be to inside the container later by some change in kind I guess). Once runc 1.1 is used inside the container created using runc 1.0, we have this issue with inability to remove rdma and unified cgroups (which is noted in logs but is not fatal so not resulting in any failures), which eventually can lead to the issue described here -- inability to put a process into an RDMA cgroup (because, I guess, we hit some kind of a kernel limit on a number of cgroups). HTH
slightly catching up on this thread Yes, from currently bundles runc, and for that defaults to using the version from in the containerd repository. Technically we can override this version when building the packages but (I'll spare the details), it's a bit complicated to do this without a corresponding containerd release. The "good news" is that I have opened backports for both the containerd v1.6 and v1.5 release branches to bring runc to version v1.1.1 (, ). Those backports have been merged, but not yet released in a containerd release. Once that happens, we will publish updated packages of on As an extra note; we currently publish packages of containerd v1.5.x, but once the next () release of Docker arrives, will be switching to containerd v1.6.x (a draft PR to verify that Docker 20.10 also works with that version can be found here; , and a similar PR to verify against containerd v1.5 with runc v1.1.1 in )
I'm not sure this is eliminated: but CI should be using kind @ HEAD and merged two days ago
reopening per to make sure this is resolved
should that have actually closed this issue? not seeing how the linked commit in Ben's fork modified kind bringup
Oh no, that's that GitHub "feature", I merely synced my fork to upstream but the commit contains "fixes". Not sure why CI didn't block this with the invalid-commit label.
Ah, yes, those are always a pain.
Potentially related: ... We should probably update the docker-in-docker in Kubernetes CI, it's going to have an outdated docker install and it's generally not well done, I've been meaning to clean that up ... is still based on Debian Buster ...
the tests that execute command on pods seems to be affected by this
KIND's CI image is now on docker 20.10.15 / runc v1.1.1-0-g52de29d. Tentatively after this change we don't see any more loges about rdma cgroups, I've spot checked with a few for CI logs from before and after the change. Which makes sense since3 we should be on runc v1.1.1 on both dind and kind within that. (the CI dind is still naively done and I'm not sure what the underlying hosts are running currently, need to get back to that ...)
is indeed empty. on all jobs has a few, but those are:
The csi-driver-hostpath jobs are due to kubekins-e2e image not having the updated docker (and possibly not updated kind). CAPI is probably the same thing. These remaining flakes are rare and not affecting CI for this repo. /close
Closing this issue. <detailsIn response to : The csi-driver-hostpath jobs are due to kubekins-e2e image not having the updated docker (and possibly not update kind). /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
return Unknown, fmt.Errorf("missing exec parameters") } data, err := e.runner.RunInContainer(podFullName, podUUID, container.Name, container.LivenessProbe.Exec.Command) <del> glog.V(1).Infof("container %s failed health check: %s", podFullName, string(data)) </del> <ins> glog.V(1).Infof("container %s health check response: %s", podFullName, string(data)) </ins> if err != nil { return Unknown, err }