Change:
After we updated two of our clusters from 1.22 to v1.24.4 we noticed a huge amount of logs (tens of entries per second) in kubelet: Neither does the subdir exist, nor the corresponding subdir for the pod ( in this case). Kubelet currently produces multiple gigabytes of logs each day since the upgrade. No log spam (at least not with verbosity=0). Not really sure. We changed nothing in the setup. We just upgraded to v1.24.4. The cluster does not have any volume providers running. Just a 'vanilla' kubernetes (ubuntu 20.04, kubeadm, cillium, cri-o). No response <details</details<detailsbare metal </details<details</details<detailskubeadm </details<detailsCRI-O Version: 1.24.2 GitCommit: GitTreeState: clean BuildDate: 2022-08-09T16:42:39Z GoVersion: go1.18.2 Compiler: gc Platform: linux/amd64 Linkmode: dynamic BuildTags: apparmor, excludegraphdriverdevicemapper, containersimageostree_stub, seccomp SeccompEnabled: true AppArmorEnabled: true </details<detailscilium v1.12.0 </details
cleanupOrphanedPodDirs clean up volume dir. cleanupOrphanedPodCgroups check that the directory exists. Maybe need some clean up.
/sig node
Looks like it's a problem with runc <= 1.1.3 After upgrading runc everyhting works again. edit: nope, loglines are still present with k8s v1.24.4 and runc 1.1.4. But the memory leak and cgroup error messages seem to be gone. edit: typo in runc versions. 1.1.3 and 1.1.4, not 1.3/1.4.
Do you mean contained v1.3 here? runc's latest version is v1.1.4 now
Refer to For v1.24, suggested containerd versions are 1.6.4+, 1.5.11+.
Sorry, Typo. runc 1.1.3.
After upgrading runc to 1.1.4 the memory leaks disappeared and the amount of loglines went down from ~ 100/s to ~1/s, but it still generates a lot of useless logs.
cc
Sorry, I can't see how it relates to runc in any way possible. I am not sure what you mean here, the issue description does not mention any memory leak or cgroup error messages, only the "Path does not exist" error from .
Sorry for mixing multiple issues. We've run into this, I think: Currently we're on v1.24.4 with runc 1.1.4 and still have multiple issues: amount of log lines (10s per second): and consuming all available memory (around 60G) until it crashes
Ah, OK, this might be related to runc, since the error comes from runc code. Having said that, I am not aware of any bugs in runc 1.1.4 that can result in this error. Now, what runc does is it sends the command to systemd (this is the same thing that happens when you issue ) and waits for up to 30 seconds for systemd to respond. Generally, 30s it is enough for systemd to remove the unit (or report an error), but it does not happen in this case and so the timeout is hit. What this means, practically, is systemd is currently having some major issues and is not very responsive. I would start to debug this particular issue by reproducing it and when looking into system logs (journalctl) to see what's going on. Checking how systemd copes in would make sense, too. Hope that's useful.
Definitely helpful, thanks! Now I need to figure out whats causing the memory consumption and hanging system.
Apart the underlying problem, I want to understand if kubelet can be improved to handle this error. you said kubelet consumes all memory and crashes. Do you mean kubelet as a process, or just the log file?
/triage needs-information Moving to needs information for now to get the full scope of the issue, but it feels like there is an error in kubelet with the log verbosity at least.
I am not the reporter of the issue, but someone having the same. In my case it is kubelet itself consuming the memory. I have four nodes, out of three clusters, going into the downward spiral as we speak. in each case it is the kubelet process that takes the memory until the OOM reaper kicks in. I have a script to drain and reboot the node when it reaches 3 GB free just to make things gentler. in each case the kubelet process seems to log exponentially, so over time it becomes more severe, doubling over time, and exhausting the journal. However, kubelet is also taking all of the available memory that the server has until it reaches OOM. restarting the kubelet itself only acts as a short bandaid, it helps memory usage, but not cpu usage, and the the issue comes back in fairly short order (as well as side effects along the way). but a full shut down, clearing everything out that it is complaining about, will last for awhile longer (i get about 3 days from a control pane node. a week or so from a worker node). I have been trying to experiment on a 32gb worker node i have in its death throes right now (no actual workloads have been on it beyond the builtins, and its gone from 3GB to 24 GB usage on kubelet process itself in around 4 hours right now), but i can find no fault in systemd itself. it responds as snappy as always, journal is quite besides the kubelet errors reported above, and the scopes that kubelet says timed out, did actually get removed, so kubelet both tries to keep removing them via systemd (when they are already gone, but also still says they time out), then also saying that the volumes dont exist (because they dont, they did get removed), and as i mentioned, it seems to get to a point where it almost appears exponential, and the node OOMs. so i can find no evidence of an actual timeout by systemd right now. however, despite the scopes actually being removed, kubelet refuses to believe they are gone, and keeps trying to remove them, complaining in both a timeout of systemd as well as the volume not existing, and that appears to grow exponetially until the node OOMs. as i said i have 3 clusters exhibiting these same symptoms and log messages, and lose nodes frequently, so i can gather any specific information rather easily. unfortunatly the journal has nothing of interest beyond what said, both the volume not found (because it is gone), and the systemd timeout removing the cgroup (which it did successfully remove the first time) I specifically am running OKD 4.11, which appears to be runc 1.1.3 right now, and k8s 1.24.0
Can confirm everything said. We have currently 3 clusters with this problem. All started with the upgrade to 1.24.* and all of them are running runc (1.1.4). Another cluster with the same setup and k8s version does not show this problem and uses crun.
/cc Can you please take a look?
I opened a PR to fix the reported issue of log spam.
/triage accepted
/kind bug
Will the OOM issue that we are seeing?
likely not /reopen
Reopened this issue. <detailsIn response to : Will fix the OOM issue that we are seeing? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
one piece of information we are looking for is whether the same ID will keep showing up for hours. It is not clear whether this is recoverable and expected race that we just need to hide or this is an unrecoverable problem. OOM reports makes think that it might be unrecoverable, but it is hard to imagine what will take that much memory on this clean up even if it stuck in this state and not recoverable.
This is an about 16 hours log It contains more than 4 million records Some "path does not exist" pods disappear eventually Others may hang around for hours
In my case the same ids kept showing up for hours. This was due to the misc cgroup not being properly cleaned up. And because not everything was cleaned up the pods kept getting picked up by the housekeeping loop. On the first run the directory is removed. all consecutive runs this log is produced. So while this log isn't the problem it is (the only) indication that the kubelet fails to completely delete pods from the system.
We are running k8s 1.24.9 with cri-o 1.24.4 and runc 1.0.1 and 1.1.4 on ubuntu 20.04 with cgroups v1 in use and experinece this issue. Apart from high amount of such messages we also experienced high CPU & memory consumption. I created some simple deployment with 20 nginx pods which I rotated very often (every 10 secs) to create a lot of garbage in cgroups. Kubelet's CPU usage: ! Kubelet's memory usage: ! This was generally fixed by switching to ubuntu 22.04 which enables cgroups v2 by default.
I also saw the issue disappear after switching to ubuntu 22, however I used Kubernetes 1.25 on ubuntu 22, whereas I had the problem with 1.24 on ubuntu 20.
this was because you switched from cgroups v1 (ubuntu 20 default) to v2 ( Ubuntu 22 default).
Obviously ;)
The misc controller was introduced in 5.13, my guess is that there's a discrepancy between some code that creates the cgroups (handles misc) and some code that cleans up the cgroups (doesn't handle misc) , causing the misc cgroups to be left behind.
In the case of linux kernel 5.15.x and using cgroups v1 and 1.25.x kubernetes, it seems like the runc () doesn't include the removal of "misc" cgroups and upon checking of the code it seems like "misc" cgroups implementation seems to be missing, is there any reason for this or perhaps someone who has more knowledge could point me in the correct direction ? As the result of this (leftover of misc cgroups dir in ), () will always include it in the , thus the spam of "Path does not exist path=..." errors. Btw, if anyone is interested to give it a try, by disabling "misc" cgroups it seems like we can get rid of this spam of "Path does not exist path=..." messages.
Although this patch doesn't seem to be the "complete" implementation of "misc" cgroups but it does seem to properly cleaned up "misc" cgroups leftover after a (note: if you have a previously left over "misc" cgroups directory leftover in then you might want to reboot the system after applying the patch) - this is only relevant if you are using linux kernel 5.13.x onwards (with "misc" cgroups) and cgroups v1.
Hi, We are observing the same issue on Bottlerocket official AMI images. Suddenly a node during the night gets flooded with those errors and fills up the disk. Does anyone run Bottlerocket OS ?
looks like you found the correct bottle rocket issue. leaving the link here for others who may run into it.
I stumbled upon this issue, as we have the same problem on all our Rancher RKE (k8s v1.24 and v1.25) clusters where the container is spamming messages - this resulted in gigabytes of logs from one of our clusters which is running dozens of kubernetes-jobs on ~3 nodes. For a quick and ugly fix I created the script below that periodically cleans the corresponding -folders via a simple cronjob () - not sure if it's super smart what I'm doing and the script can surely be done in a much better way, but we needed a quick fix, the containers on the nodes caused extremely high CPU-usage writing those logs and our ELK also complained about space-usage ;) I haven't seen any issues yet on the clusters where this script is active and docker logs for the container also look much much cleaner. Log-space in our ELK was reduced from for example ~15G per hour from one of the nodes to ~3G per hour. CPU usage also dropped significantly from 20% to 4% after executing the script manually on one of the nodes.

Discussion:
if pathExists, pathErr := mount.PathExists(podVolDir); pathErr != nil { return volumes, fmt.Errorf("error checking if path %q exists: %v", podVolDir, pathErr) } else if !pathExists { <del> klog.InfoS("Path does not exist", "path", podVolDir) </del> <ins> klog.V(6).InfoS("Path does not exist", "path", podVolDir) </ins> return volumes, nil }