Change:
Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29. At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:

Discussion:
"github.com/coreos/go-systemd/v22/daemon" "github.com/spf13/cobra" "github.com/spf13/pflag" <ins> "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/metric/noop" </ins> "google.golang.org/grpc/codes" "google.golang.org/grpc/status" jsonpatch "gopkg.in/evanphx/json-patch.v4"

func init() { utilruntime.Must(logsapi.AddFeatureGates(utilfeature.DefaultMutableFeatureGate)) <ins> // Prevent memory leak from OTel metrics, which we don't use: // https://github.com/open-telemetry/opentelemetry-go-contrib/issues/5190 otel.SetMeterProvider(noop.NewMeterProvider()) </ins> } const (

go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful v0.42.0 go.opentelemetry.io/otel v1.28.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 <ins> go.opentelemetry.io/otel/metric v1.28.0 </ins> go.opentelemetry.io/otel/sdk v1.28.0 go.opentelemetry.io/otel/trace v1.28.0 go.opentelemetry.io/proto/otlp v1.3.1

go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect <del> go.opentelemetry.io/otel/metric v1.28.0 // indirect </del> go.starlark.net v0.0.0-20230525235612-a134d8f9ddca // indirect go.uber.org/multierr v1.11.0 // indirect golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect

go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 go.opentelemetry.io/otel v1.28.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 <ins> go.opentelemetry.io/otel/metric v1.28.0 </ins> go.opentelemetry.io/otel/sdk v1.28.0 go.opentelemetry.io/otel/trace v1.28.0 go.uber.org/zap v1.26.0

go.etcd.io/etcd/pkg/v3 v3.5.13 // indirect go.etcd.io/etcd/raft/v3 v3.5.13 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect <del> go.opentelemetry.io/otel/metric v1.28.0 // indirect </del> go.opentelemetry.io/proto/otlp v1.3.1 // indirect go.uber.org/multierr v1.11.0 // indirect golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect

"net" "github.com/spf13/pflag" <ins> "go.opentelemetry.io/otel" </ins> "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" <ins> "go.opentelemetry.io/otel/metric/noop" </ins> "go.opentelemetry.io/otel/sdk/resource" "go.opentelemetry.io/otel/semconv/v1.12.0" "google.golang.org/grpc"

) func init() { <ins> // Prevent memory leak from OTel metrics, which we don't use: // https://github.com/open-telemetry/opentelemetry-go-contrib/issues/5190 otel.SetMeterProvider(noop.NewMeterProvider()) } func init() { </ins> install.Install(cfgScheme) }