Change:
func (m *podContainerManagerImpl) Destroy(podCgroup CgroupName) error { // Try killing all the processes attached to the pod cgroup if err := m.tryKillingCgroupProcesses(podCgroup); err != nil { <del> klog.V(3).Infof("failed to kill all the processes attached to the %v cgroups", podCgroup) </del> <ins> klog.Warningf("failed to kill all the processes attached to the %v cgroups", podCgroup) </ins> return fmt.Errorf("failed to kill all the processes attached to the %v cgroups : %v", podCgroup, err) }

Discussion:
What happened: Housekeeping sync loop finds an orphaned pod and tries to clean it up. The kubepod slice, as seen below, is not removed by with no indication as to why. This leads to the docker scope timing out and entering a failed state. The returned error(s) from runc are swallowed making debugging difficult. What you expected to happen: As per other areas of the sync loop, useful errors are logged at debug level due to the async calls. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: N/A Environment: Kubernetes version (use ): master
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Have you seen the comment above pcm.Destroy() call ? Did you have a chance to look at kubelet log to see if the above was true ?
/remove-lifecycle stale
This would give us some clue about the error:
yes, definitely looked! In this specific situation, the pods are never retried. What we see happen with these pods: ADD -RECONCILE -Started Volumes -Created slice -No sandbox for pod -Started libcontainer -ContainerStarted -ContainerStarted -RECONSILE -ContainerDied -RECONCILE -Unmounts -Killing unwanted pod -Orphaned removing pod cgroups -container kill failed because of 'container not found' or 'no such process' -cleanup: failed to unmount secrets -ContainerDied -Container garbage collection succeeded -SyncLoop (SYNC): 141 pods (including this one) -stopping timed out. Killing.

-Stopped libcontainer -Removed slice -DELETE -REMOVE -err: pod not found Destroy all cgroups of the pod that should not be running, by first killing all the attached processes to these cgroups. We ignore errors thrown by the method, as the housekeeping loop would again try to delete these unwanted pod cgroups Despite running, the slice is not removed and we do not log or indicate why. Subsequently, the pod is NOT determined to be either an active pod or for some reason not in the cgroup list that is passed into kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) during housekeeping loops.

A more detailed look, which I believe is accurate ;): housekeepingCh -HandlePodCleanups -cgroupPods=UID+CGroupName=GetAllPodsFromCgroups -allPods=GetPodsAndMirrorPods -activePods=filterOutTerminatedPods -cleanupOrphanedPodCgroups I do not think desiredPods and runningPods are relevant here.. So either: a) the pod is not found and to allPods (seems unlikely) but we can't know for sure.. b) the pod is incorrectly determined to be active via activePods which again we can not determine.. Any pod that is not Failed, Succeeded or has a DeletionTimestamp set and a status of Terminated or Waiting - anything else returns false and is deemed active c) the pod cgroup is not found in m.subsystems.MountPoints and to cgroupPods - unlikely, but could be a bug but the root of this issue is the fact that all of the scopes and slices exist and keep building up until things start failing... GetAllPodsFromCgroups could use some logging... Maybe a V 5 log that prints out foundPods at the end... From there, they're passed into the cleanup: kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) 1) All activePod UIDs are put into a podSet 2) A range loop of uid and cgroup name from cgroupPods looks to see if the uid from cgroupPods also exists in the podSet (active pods) 3) If the uid from cgroupPods exists in the podSet we skip out of the loop 4) If it doesn't, the pod is orphaned (not active w/ cgroups on the filesystem) and we call pcm.Destroy(val) on the cgroup name. Doesn't make sense why pcm.Destroy(val) ignores errors... it would be helpful!

tryKillingCgroupProcesses logs any error and also returns the an error which is always swallowed.. we don't see any issues here More problematic, the call to cgroupManager.Destroy only returns an error which would be swallowed... Could probably use a similar level 3 log here like what you have for tryKillingCgroupProcesses because as you can see, were totally blind here...
your change is of a similar nature to what I proposed in my PR for this issue, just a level higher. I think there are a few areas noted above the could help further debug this.. Still haven't been able to get my PR merged with additional logging.
The following code was relatively new (compared to when you logged this issue):
Indeed... I looked into it a while ago and those changes are new via: - Could definitely make a different if the issue is related to pod getting filtered out from activePods when they have not been fully cleaned up yet on the subsequent housekeeping syncloop runs.
I think is promising, but I still think it makes sense to add some logging here. Do you want want to proceed with my open PR, your suggestion in or both?
You can proceed with your PR.