Change:
/kind bug What happened: Start a cluster using ./hack/local- Now just monitor the memory on kube-apiserver and kube-controller-manager. Without doing anything else note the memory on each process has grown by 500 MB each day. Tried this for 5 days and saw this steady growth. What you expected to happen: Memory to remain relatively flat after some initial growth. How to reproduce it (as minimally and precisely as possible): Run ./hack/local- and wait. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10 Cloud provider or hardware configuration: local OS (e.g. from /etc/os-release): Debian GNU/Linux Kernel (e.g. ): Debian 4.9.65-3+deb9u2 Install tools: Others:
/sig api-machinery
/milestone v1.10
/priority important-soon
!
Copying some of what was discussed in Slack - we need to redo a test with set when running . This disables the cache mutation detector, which holds on to "all the things" .
What does the chart mean? yeah determined that most of it is from the mutation detector (it really makes me nervous to have test code linked in to the production binaries).
It was a differential pprof memory chart of hack/local-cluster- But as noted, we have to run it without mutation detection. I did that for an hour yesterday and did not notice much memory leaking.
After the suggested fix (turn of mutation detection), there was no noticeable increase on controller manager memory and the api-server memory increase was down to 8 MB per day. (on my setup)
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 2 days, the issue will be moved out of the v1.10 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
closing this to remove it from the release radar. Feel free to reopen if you think we still have an issue.

Discussion:
# Install a default storage class (enabled by default) DEFAULT_STORAGE_CLASS=${KUBE_DEFAULT_STORAGE_CLASS:-true} <del> # start the cache mutation detector by default so that cache mutators will be found KUBE_CACHE_MUTATION_DETECTOR="${KUBE_CACHE_MUTATION_DETECTOR:-true}" </del> <ins> # Do not run the mutation detector by default on a local cluster. # It is intended for a specific type of testing and inherently leaks memory. KUBE_CACHE_MUTATION_DETECTOR="${KUBE_CACHE_MUTATION_DETECTOR:-false}" </ins> export KUBE_CACHE_MUTATION_DETECTOR # panic the server on watch decode errors since they are considered coder mistakes

cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt cluster/kubectl.sh config set-context local --cluster=local --user=myself cluster/kubectl.sh config use-context local <del> fi </del>  No newline at end of file <ins> fi </ins>

"sync" "time" <ins> "github.com/golang/glog" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/diff" )

if !mutationDetectionEnabled { return dummyMutationDetector{} } <ins> glog.Warningln("Mutation detector is enabled, this will result in memory leakage.") </ins> return &defaultCacheMutationDetector{name: name, period: 1 * time.Second} }