Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: unused images does not get deleted What you expected to happen: unused images get deleted How to reproduce it (as minimally and precisely as possible): configure kubelet with these args: Anything else we need to know?: when using above config, the node's events history showing: If I change to it works, node's events history showing these: The difference is there is an extra event, which is not seen with cofnig. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux 192-168-1-1 4.4.0-63-generic -Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig area/kubelet
Maybe we are with the same problem. Kubelet Parameters for tests: When we reach a disk consumption of 20% (for debug) - after 2 minutes, the deletion of logs, image docker, and then PODS should begin. But nothing happens. The debug logs kubelet saw the following: Problems are observed in the versions 1.6.X, 1.7.X In version 1.5.7, it looks like this: Can you help us? Or we help you, with Debug? Environment: Kubernetes version (use kubectl version): OS (e.g. from /etc/os-release): ubuntu 16.04.2 LTS Kernel (e.g. uname -a): Linux node2 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Docker: Docker version 1.11.2, build b9f10c9_ PS We tried the rollback option from , but it did not change anything.
hello! Can you spare a little time to this problem?
Well, here is what we have found so far. In our setup, eviction.managerImpl.synchronize() frequently runs with a set of signalObservation objects, which contains the same data as in previous runs, even for "time" field. Probably this is due to slow cadvisor. Unfortunately, eviction.thresholdsUpdatedStats() considers such data expired and removes it from the list of observed thresholds. This was always the case, but before commit "time" field wasn't initialized, and thresholdsUpdatedStats allowed it to persists between iterations of synchronize() by explicit check for After neither check for zero, no check for one time being more recent then another, doesn't work as iterations are frequently performed with the same summary.Node.Fs object. And here we are with blinking "node conditions - observed: [DiskPressure]" in logs: one run of synchronize sets it, and next one discards it (runs are delimited with "synchronize housekeeping" message): The question is whether this logic in thresholdsUpdatedStats is correct now or not. Should garbage collection work even with slow cadvisor (if this is the reason of slow updates of summary.Node.Fs)? Probably frequency of synchronize should be adjusted, or summary.Node.Fs updates should be synchronized with execution of synchronize()? We can correct the problem by monkey-patching makeSignalObservations and get rid of summary.Node.Fs.Time usage in signalObservation, but is it right?
The 2 minute grace period has not been met, so no action is taken. This is the difference between a "soft" and "hard" threshold: The soft only takes action after the threshold has been met for the specified period of time. If no new stats exist, the correct behavior is not to take action. This could delay action by 10 seconds, but it is better not to be overactive when evicting (we have had this problem in the past). The eviction manager assumes that disk pressure still exists when it does not have new stats. It basically just "skips" that synchronize call when that is the case.
dekhtyarev@ it looks like your node goes in and out of disk pressure. I see messages woven into your logs.
Hmmm, or maybe there is something here. Looks like the timer is reset for soft eviction thresholds.
No, On seeing this, I just ran several PODs on the node and did to achieve the conditions of eviction. I used ~22% of the disk, at the threshold of 80%. And disk consumption on the node did not change.
We just think that the timer resets, as the timestamp of the received data has not changed and this is taken into account in the eviction manager logic. Look at the field here: wrote about it, I think.
I see the issue. Ill try and get a bugfix into 1.8.
cc Soft Eviction thresholds have not been functional since 1.6...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details

Related discussion:
text: } debugLogThresholdsWithObservation("thresholds - reclaim not satisfied", thresholds, observations) <del> // determine the set of thresholds whose stats have been updated since the last sync thresholds = thresholdsUpdatedStats(thresholds, observations, m.lastObservations) debugLogThresholdsWithObservation("thresholds - updated stats", thresholds, observations) </del> // track when a threshold was first observed now := m.clock.Now() thresholdsFirstObservedAt := thresholdsFirstObservedAt(thresholds, m.thresholdsFirstObservedAt, now)