Change:
Issue originating from: The label seems to be unused, but it can cause issues with high cardinality. On k3s this label has unbound cardinality due to the inclusion of url param like timeout. Examples from my small k3s cluster: ! query returns: Metrics at fault here seems to be: I didn't expect these metrics to generate this many series. Execute the following promQL query or its equivalent for your monitoring platform scraping kubelet: We could mitigate this issue by pruning the query parameter from the label. I am no kubelet expert so I don't know if it is normal that the timeout query parameter is variable, but if not that might be an issue as well. v1.22.3 <detailsNone </details<details</details<details</details<details</details<details</details
/sig instrumentation /sig node /cc
/triage accepted What timeseries is this coming from?
Updated the issue. It should be:
Since these metrics are coming from component-base, I guess we could remove sig-node, wdyt /assign
These metrics appear to have existed for 2 years, suggesting that this bug has been present for many, many releases... is this k3s specific? Edit: rate_limiter metrics weren't registered before 1.22:
/assign FYI
I think these metrics are broken for all components, not just kubelet, so I'd prefer that we try to fix the cardinality issue rather than just deregister from kubelet. I'd suggest we fix them by paring down the URL to something that's actually aggregatable, e.g. becomes That also looks a little like an URL quoting issue, I'd expect the particular namespace is substituted in... Not a release blocker since this has been present since 1.22. /priority important-soon
/remove-sig node
Maybe I misunderstood your point but IIUC the actual namespace value is replaced by the generic value to avoid excessive cardinality.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
were you able to find a work around. We just had the same issue occurred for us on multiple clusters. It brought down Prometheus :(

Discussion:
) var ( <del> // requestLatency is a Prometheus Summary metric type partitioned by // "verb" and "url" labels. It is used for the rest client latency metrics. </del> <ins> // requestLatency is a Prometheus Histogram metric type partitioned by // "verb", and "host" labels. It is used for the rest client latency metrics. </ins> requestLatency = k8smetrics.NewHistogramVec( &k8smetrics.HistogramOpts{ Name:    "rest_client_request_duration_seconds", <del> Help:    "Request latency in seconds. Broken down by verb and URL.", </del> <ins> Help:    "Request latency in seconds. Broken down by verb, and host.", </ins> Buckets: k8smetrics.ExponentialBuckets(0.001, 2, 10), }, <del> []string{"verb", "url"}, </del> <ins> []string{"verb", "host"}, </ins> ) rateLimiterLatency = k8smetrics.NewHistogramVec( &k8smetrics.HistogramOpts{ Name:    "rest_client_rate_limiter_duration_seconds", <del> Help:    "Client side rate limiter latency in seconds. Broken down by verb and URL.", </del> <ins> Help:    "Client side rate limiter latency in seconds. Broken down by verb, and host.", </ins> Buckets: k8smetrics.ExponentialBuckets(0.001, 2, 10), }, <del> []string{"verb", "url"}, </del> <ins> []string{"verb", "host"}, </ins> ) requestResult = k8smetrics.NewCounterVec(

} func (l *latencyAdapter) Observe(ctx context.Context, verb string, u url.URL, latency time.Duration) { <del> l.m.WithContext(ctx).WithLabelValues(verb, u.String()).Observe(latency.Seconds()) </del> <ins> l.m.WithContext(ctx).WithLabelValues(verb, u.Host).Observe(latency.Seconds()) </ins> } type resultAdapter struct {