Code change:
The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact. What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).

Related discussion:
text: // For reference: https://github.com/kubernetes/kubernetes/pull/81387 legacyregistry.RawMustRegister(grpcprom.DefaultClientMetrics) dbMetricsMonitors = make(map[string]struct{}) <ins> l, err := logutil.CreateDefaultZapLogger(etcdClientDebugLevel()) if err != nil { l = zap.NewNop() } etcd3ClientLogger = l.Named("etcd-client") } // etcdClientDebugLevel translates ETCD_CLIENT_DEBUG into zap log level. // NOTE(negz): This is a copy of a private etcd client function: // https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/logger.go#L47 func etcdClientDebugLevel() zapcore.Level { envLevel := os.Getenv("ETCD_CLIENT_DEBUG") if envLevel == "" || envLevel == "true" { return zapcore.InfoLevel } var l zapcore.Level if err := l.Set(envLevel); err == nil { log.Printf("Deprecated env ETCD_CLIENT_DEBUG value. Using default level: 'info'") return zapcore.InfoLevel } return l </ins> } func newETCD3HealthCheck(c storagebackend.Config, stopCh <-chan struct{}) (func() error, error) {