Code change:
When both and parameters are set on Kubelet, they act as the upper and lower thresholds that define how the state is entered and exited. When trying to remediate a , Kubelet performs different operations to free up space. In order to decide if it has freed up enough, it compares the currently available disk space with the triggering threshold instead of the exiting threshold. Kubelet ends up in a long lasting state which can only be left with Kubelet restart, or if by luck the Pods that were using disk space are rescheduled somewhere else. Kubelet periodically does some housekeeping in order to ensure its node has enough resources available. If that isn’t the case, it tries to free up resources, updates the node’s condition (e.g. ) and in last resort starts evicting Pods. By default, the housekeeping is done every 10 seconds for a set of “signals” which represent node level resources (e.g. , , , ). There are a couple of parameters that comes into play in this process and I'll ignore all but two for simplicity: Signal thresholds: defines at which point there is not enough of a resource available, e.g. Signal minimum reclaim: how much resources Kubelet should aim to free up before a threshold is considered as not met. It’s the lower bound of an hysteresis, the threshold being the upper bound (e.g. ) The housekeeping is implemented in Here is my understanding of what this method does, filtering out what I think does matter in this issue: the signals (e.g. how much is available) a list of thresholds that are met (e.g. observed < ) there were thresholds met on a previous run: a. Check if those previous threshold are still met taking the into account. If = 15% and , our value to unmet the threshold is actually 25% percent b. Merge them with (2), the thresholds that are met from the new observation (1) out thresholds for which the individual grace period is not reached. They might be taken into account in another run if observed again. the previously met threshold with the new computed list of thresholds. Those met threshold are stored across runs. no threshold is not actually met, exits. If at least one is met, Kubelet has work to do! Try to reclaim node level resources. In other terms, try to free up resources without having to evict any Pod. For that’s done by removing stopped containers, and unused images. a new observation. If the threshold are not met anymore, ignoring , exit with Pods as last resort Ignoring the in (7) seems wrong. If the amount of resources freed is somewhere between and , Kubelet can get stuck. On the next run, as part of (3a) the threshold taking into account is not unmet. Therefore Kubelet tries again to free up node level resources in (6), eventually does nothing but exits (6) pretending it free up enough resources without doing anything. after having cleaned up the node and in order to decide if Pod eviction is the step, seems to do the trick. Kubelet should try to get out of a by evicting almost all Pods. In this case, it means it should try to evict Pods until the threshold is reached and the can be removed. I expect Kubelet to stay in permanent only if the resources that are used are beyond its control. Preparation: a node with a different from (the problem might be the same when they are mixed but I didn't try) to 15% and to 30% in the Kubelet’s configuration into a Pod running on that node, and use to write a file inside the container. The amount of data should be enough to bring the underlying disk at 86% usage for Kubelet to notice the threshold and to cleanup stopped containers and remove unused images Remove your file. The node is now clean. Restart Kubelet and notice how it sees everything is normal Getting Kubelet stuck: into your Pod and write a file to reach 86% usage again for Kubelet to notice, cleanup stopped containers and images. Since the cleanup was done in the preparation step a minute ago, there is almost nothing to be removed and your usage should stay more or less the same the Pod's file by a new one to reach between 55% and 85% disk usage. how every 10 seconds, Kubelet pretends it was able to . The Node is stuck in until you restart Kubelet and it forgets it had to ensure there is at least available in . Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: vSphere OS (e.g: ): 18.04.2 LTS Kernel (e.g. ): 4.15.0-130-generic We have been running a slightly modified version of Kubelet on one of our clusters for the past 5 days, and since then we had no node staying in state for more than 20 minutes. From the disk usage graph, it looks like both image garbage collection and the disk pressure remedy operations are working properly. I'll open a PR with the little patch we're using.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node

Related discussion:
text: t.Errorf("Manager should not report disk pressure") } <ins> // synchronize manager.synchronize(diskInfoProvider, activePodsFunc) // we should not have disk pressure if manager.IsUnderDiskPressure() { t.Errorf("Manager should not report disk pressure") } // induce hard threshold fakeClock.Step(1 * time.Minute) summaryProvider.result = summaryStatsMaker(".9Gi", "200Gi", podStats) // make GC return disk usage bellow the threshold, but not satisfying minReclaim diskGC.summaryAfterGC = summaryStatsMaker("1.1Gi", "200Gi", podStats) manager.synchronize(diskInfoProvider, activePodsFunc) // we should have disk pressure if !manager.IsUnderDiskPressure() { t.Errorf("Manager should report disk pressure since soft threshold was met") } // verify image gc was invoked if !diskGC.imageGCInvoked || !diskGC.containerGCInvoked { t.Errorf("Manager should have invoked image gc") } // verify a pod was killed because image gc was not enough to satisfy minReclaim if podKiller.pod == nil { t.Errorf("Manager should have killed a pod, but didn't") } // reset state diskGC.imageGCInvoked = false diskGC.containerGCInvoked = false podKiller.pod = nil // remove disk pressure fakeClock.Step(20 * time.Minute) summaryProvider.result = summaryStatsMaker("16Gi", "200Gi", podStats) manager.synchronize(diskInfoProvider, activePodsFunc) // we should not have disk pressure if manager.IsUnderDiskPressure() { t.Errorf("Manager should not report disk pressure") } </ins> // induce disk pressure! fakeClock.Step(1 * time.Minute) summaryProvider.result = summaryStatsMaker("400Mi", "200Gi", podStats)