Repository: kubernetes/kube-state-metrics

Issue #2408: kube_endpoint_address duplicates with Prometheus 2.52
Created: 2024-05-31T13:03:35Z
After upgrading to Prometheus 2.52, we had some alerts about dropped duplicates samples.


The prometheus log shown the following warning:
 scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://100.91.220.12:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=1



Setting the log level to debug shown the concerned series:
scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://100.91.220.12:8080/metrics msg="Duplicate sample for timestamp" series="kube_endpoint_address{namespace=\"monitoring\",endpoint=\"prometheus-operated\",ip=\"100.91.68.8\",ready=\"true\"}"



Checking the indicated series indeed shown the following duplicates:
kube_endpoint_address{namespace="monitoring",endpoint="prometheus-operated",ip="100.91.68.8",ready="true"} 1
kube_endpoint_address{namespace="monitoring",endpoint="prometheus-operated",ip="100.91.68.8",ready="true"} 1



The prometheus-operated endpoint has the following subsets:
subsets:
  - addresses:
      - ip: 100.91.43.113
        hostname: prometheus-kube-prometheus-istio-0
        nodeName: ip-100-91-48-253.eu-west-3.compute.internal
        targetRef:
          kind: Pod
          namespace: monitoring
          name: prometheus-kube-prometheus-istio-0
          uid: 1180e2a5-75e4-4098-961c-940264115438
      - ip: 100.91.68.8
        hostname: prometheus-kube-prometheus-stack-prometheus-0
        nodeName: ip-100-91-212-113.eu-west-3.compute.internal
        targetRef:
          kind: Pod
          namespace: monitoring
          name: prometheus-kube-prometheus-stack-prometheus-0
          uid: 257bdfed-e2b4-49c7-aaea-1b7bee1a520d
    ports:
      - name: http-web
        port: 9090
        protocol: TCP
  - addresses:
      - ip: 100.91.68.8
        hostname: prometheus-kube-prometheus-stack-prometheus-0
        nodeName: ip-100-91-212-113.eu-west-3.compute.internal
        targetRef:
          kind: Pod
          namespace: monitoring
          name: prometheus-kube-prometheus-stack-prometheus-0
          uid: 257bdfed-e2b4-49c7-aaea-1b7bee1a520d
    ports:
      - name: grpc
        port: 10901
        protocol: TCP


We can see the 2 entries on the same IP (100.91.68.8) but on different ports.
Grpc is enabled only by the Thanos sidecar container, and it's enabled only on one Prometheus instance.
I think there wouldn't have been duplicates if both instances had the same config (there would only have been one subset with both addresses and ports).
The only way I see to fix this would be to add a port label on the kube_endpoint_address metric.
Is there something else I can do or would this be considered as a bug ?
Thanks !
Environment:

kube-state-metrics version: 2.12.0
Kubernetes version: 1.28
Cloud provider or hardware configuration: AWS EKS
Other info:

Duplicate issue #2445:  Duplicate sample for ingress path metrics
Created: 2024-07-11T19:28:05Z
What happened:
After upgrading Prometheus to 2.52, we started receiving warnings about duplicate metrics, which were ultimately tied to kube-state-metrics. There are a couple forms of this, most of which area already reported in other tickets (#2405, #2408, #2390), but this is a different variation.
The metric kube_ingress_path is resulting in duplicates. In our case, we do have ingress path entries which are completely redundant (the result of some shoddy helm templates), but it's still a legal Kubernetes state resulting in illegal Prometheus scrape results.
What you expected to happen:
Either the data is de-duplicated, or the duplicates are somehow differentiated (with an index, perhaps).
How to reproduce it (as minimally and precisely as possible):
Create an ingress with duplicate paths, then create metrics for it with kube-state-metrics.
Anything else we need to know?:
Environment:

kube-state-metrics version: 2.12
Kubernetes version (use kubectl version): 1.28.9
Cloud provider or hardware configuration: AWS
Other info:

Closing comment (2024-07-25T16:57:24Z): Closing as a duplicate of #2408.