Code change:
The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact. What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).

Related discussion:
text: dbMetricsMonitorJitter = 0.5 ) <ins> // TODO(negz): Stop using a package scoped logger. At the time of writing we're // creating an etcd client for each CRD. We need to pass each etcd client a // logger or each client will create its own, which comes with a significant // memory cost (around 20% of the API server's memory when hundreds of CRDs are // present). The correct fix here is to not create a client per CRD. See // https://github.com/kubernetes/kubernetes/issues/111476 for more. var etcd3ClientLogger *zap.Logger </ins> func init() { // grpcprom auto-registers (via an init function) their client metrics, since we are opting out of // using the global prometheus registry and using our own wrapped global registry,