Change:
What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed. After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again. I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery

Discussion:
prometheus.MustRegister(watchDuration) prometheus.MustRegister(itemsPerWatch) prometheus.MustRegister(lastResourceVersion) <del> cache.SetReflectorMetricsProvider(prometheusMetricsProvider{}) </del> } type prometheusMetricsProvider struct{}

"net" "net/url" "reflect" <del> "strconv" </del> "strings" "sync" <del> "sync/atomic" </del> "syscall" "time"

return NewNamedReflector(naming.GetNameFromCallsite(internalPackages...), lw, expectedType, store, resyncPeriod) } <del> // reflectorDisambiguator is used to disambiguate started reflectors. // initialized to an unstable value to ensure meaning isn't attributed to the suffix. var reflectorDisambiguator = int64(time.Now().UnixNano() % 12345) </del> // NewNamedReflector same as NewReflector, but with a specified name for logging func NewNamedReflector(name string, lw ListerWatcher, expectedType interface{}, store Store, resyncPeriod time.Duration) *Reflector { <del> reflectorSuffix := atomic.AddInt64(&reflectorDisambiguator, 1) </del> r := &Reflector{ <del> name: name, // we need this to be unique per process (some names are still the same) but obvious who it belongs to metrics:       newReflectorMetrics(makeValidPrometheusMetricLabel(fmt.Sprintf("reflector_"+name+"_%d", reflectorSuffix))), </del> <ins> name:          name, </ins> listerWatcher: lw, store:         store, expectedType:  reflect.TypeOf(expectedType),

// to be served from cache and potentially be delayed relative to // etcd contents. Reflector framework will catch up via Watch() eventually. options := metav1.ListOptions{ResourceVersion: "0"} <del> r.metrics.numberOfLists.Inc() start := r.clock.Now() </del> if err := func() error { initTrace := trace.New("Reflector " + r.name + " ListAndWatch")

return fmt.Errorf("%s: Failed to list %v: %v", r.name, r.expectedType, err) } initTrace.Step("Objects listed") <del> r.metrics.listDuration.Observe(time.Since(start).Seconds()) </del> listMetaInterface, err := meta.ListAccessor(list) if err != nil { return fmt.Errorf("%s: Unable to understand list result %#v: %v", r.name, list, err)

return fmt.Errorf("%s: Unable to understand list result %#v (%v)", r.name, list, err) } initTrace.Step("Objects extracted") <del> r.metrics.numberOfItemsInList.Observe(float64(len(items))) </del> if err := r.syncWith(items, resourceVersion); err != nil { return fmt.Errorf("%s: Unable to sync list result: %v", r.name, err) }

TimeoutSeconds: &timeoutSeconds, } <del> r.metrics.numberOfWatches.Inc() </del> w, err := r.listerWatcher.Watch(options) if err != nil { switch err {

// Stopping the watcher should be idempotent and if we return from this function there's no way // we're coming back in with the same watch interface. defer w.Stop() <del> // update metrics defer func() { r.metrics.numberOfItemsInWatch.Observe(float64(eventCount)) r.metrics.watchDuration.Observe(time.Since(start).Seconds()) }() </del> loop: for {

watchDuration := r.clock.Now().Sub(start) if watchDuration < 1*time.Second && eventCount == 0 { <del> r.metrics.numberOfShortWatches.Inc() </del> return fmt.Errorf("very short watch: %s: Unexpected watch close - watch lasted less than a second and no items received", r.name) } klog.V(4).Infof("%s: Watch close - %v total %v items received", r.name, r.expectedType, eventCount)

r.lastSyncResourceVersionMutex.Lock() defer r.lastSyncResourceVersionMutex.Unlock() r.lastSyncResourceVersion = v <del> rv, err := strconv.Atoi(v) if err == nil { r.metrics.lastResourceVersion.Set(float64(rv)) } </del> }