Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --20K job and each job is a java application running data monitor and analysis all jobs after they have been completed memory usage 120GB memory: kubelet should use and retain less memory Through pprof,we find the inusespace size is 360MB.And 67% of 120GB is Anonymous(through mmap?). Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"", GitTreeState:"archive", BuildDate:"2020-09-22T06:47:35Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Centos 7.5 Kernel (e.g. ): 3.10.0-957.21.3.el7.x8664 Install tools: Network plugin and version (if this is a network-related bug): Others: pprof png: !
/sig node
Looks like this is the same
This issue is duplicated
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
not fixs this issue.
/triage accepted
/cc
cadvisor has memory leak: cadvisor watch all cgroup director. when file or directory delete, inotify will recevie INDELETE event, when cadvisor recevie INDELETE, cadvisor will call inotify RemoveWatch to remove watch descriptor. When watch descriptor delete before function called, function will return EINVAL, may lead this path remain in watches and paths map, and also not delete path in containersWatched, finally lead to memory leak before kubernetes 1.19, kubelet use ref manager to manage the reference, this reference used for reporting events such as creation,failure, etc. In job completed scenes, container exit with exit code 0, kubelet not to remove reference in ref manager, so lead to memory leak i submit to ignore EINVAL error when calling RemoveWatch, fixed cadvisor memory leak.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
not actually fixed in k8s, the PR above should not have closed this bug
cc
Can u provide info of kubelet?
/reopen Waiting for to be merged and cadvisor to be updated.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
google/cadvisor has merged. It will be included in next cAdvisor release and next k8s version (1.24). If this problem is more widespread we can consider to backport it... is it common to get into case of inotify handles being leaked?
We have a number of issues that might be the symptom of this issue. Namely, a few customers are reporting memory is full on the node, but all the RSS doesn't add up to the in-use space. With the anonymous mmapped data being retained, this issue seems like the culprit. Can we get a backport to 1.23 and 1.22?
I think we can expect quite a big performance boost from this issue. I'll backport to 1.23 and 1.22.
Agree, sounds like we should backport it. I don't think we need to bump cAdvisor, we just need to bump k/utils to pick up the fix PRs from to bump k/utils should be sufficient to backport the fix. Thanks for creating them
I created a 1.21 backport linked to this issue:
The 1.20 backport Closed the 1.20 backport since that release is not supported.
I think we can close this, as all relevant prs have been merged. /close
Closing this issue. <detailsIn response to : /reopen Waiting for to be merged and cadvisor to be updated. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
} if err = watcher.AddWatch(path, inotify.InOpen|inotify.InDeleteSelf); err != nil { klog.ErrorS(err, "Unable to watch lockfile") <ins> watcher.Close() </ins> return err } go func() {

klog.ErrorS(err, "inotify watcher error") } close(done) <ins> watcher.Close() </ins> }() return nil }