Code change:
I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes. I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ). The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet? and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes. It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is . However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.

Related discussion:
text: // Add each PodStats to the result. result := make([]statsapi.PodStats, 0, len(podToStats)) for _, podStats := range podToStats { <del> // Lookup the volume stats for each pod. podUID := types.UID(podStats.PodRef.UID) var ephemeralStats []statsapi.VolumeStats if vstats, found := p.resourceAnalyzer.GetPodVolumeStats(podUID); found { ephemeralStats = make([]statsapi.VolumeStats, len(vstats.EphemeralVolumes)) copy(ephemeralStats, vstats.EphemeralVolumes) podStats.VolumeStats = append(append([]statsapi.VolumeStats{}, vstats.EphemeralVolumes...), vstats.PersistentVolumes...) } logStats, err := p.hostStatsProvider.getPodLogStats(podStats.PodRef.Namespace, podStats.PodRef.Name, podUID, &rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch pod log stats", "pod", klog.KRef(podStats.PodRef.Namespace, podStats.PodRef.Name)) } etcHostsStats, err := p.hostStatsProvider.getPodEtcHostsStats(podUID, &rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch pod etc hosts stats", "pod", klog.KRef(podStats.PodRef.Namespace, podStats.PodRef.Name)) } </del> <ins> makePodStorageStats(podStats, &rootFsInfo, p.resourceAnalyzer, p.hostStatsProvider, false) </ins> <del> podStats.EphemeralStorage = calcEphemeralStorage(podStats.Containers, ephemeralStats, &rootFsInfo, logStats, etcHostsStats, false) </del> <ins> podUID := types.UID(podStats.PodRef.UID) </ins> // Lookup the pod-level cgroup's CPU and memory stats podInfo := getCadvisorPodInfoFromPodUID(podUID, allInfos) if podInfo != nil {