Repository: kubernetes/kube-state-metrics

Issue #694: kube-state-metrics keep serving stale metrics after extended apiserver outage
Created: 2019-03-04T16:33:48Z
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
We wanted to check the metrics for some pods and realised we don't have metrics for the pod in question, then check the deployment and other resources, no metrics are there, but some other pod metrics were available.
We checked the metrics exposed from the /metrics endpoint and we saw metrics exposed for pods which are deleted 2+ weeks ago.
So kube-state-metrics were reporting stale metrics for last 3 weeks.
Here are some relevant logs from kube-state-metrics pod:
2019-02-10T21:56:52.294138617Z E0210 21:56:52.293444       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=124961, ErrCode=NO_ERROR, debug=""
2019-02-10T21:56:52.295193557Z E0210 21:56:52.293956       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=124961, ErrCode=NO_ERROR, debug=""
2019-02-10T21:56:52.295465367Z E0210 21:56:52.294137       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=124961, ErrCode=NO_ERROR, debug=""
2019-02-10T21:56:52.29578788Z E0210 21:56:52.294170       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=124961, ErrCode=NO_ERROR, debug=""
... There are total of 17 `Unable to decode an event` logs within same second

2019-02-10T21:56:52.321674569Z W0210 21:56:52.321518       1 reflector.go:341] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: watch of *v1.Endpoints ended with: very short watch: k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Unexpected watch close - watch lasted less than a second and no items received
2019-02-10T21:56:52.322241891Z W0210 21:56:52.322003       1 reflector.go:341] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: watch of *v1beta1.CronJob ended with: very short watch: k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Unexpected watch close - watch lasted less than a second and no items received
... There are total of 17 `Unexpected watch close` logs within same second 

# here the logs pivoted to `TLS handshake timeout`
2019-02-10T21:57:03.330774532Z E0210 21:57:03.330640       1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Pod: Get https://msa-dev-az-xxxxxx.hcp.eastus.azmk8s.io:443/api/v1/pods?limit=500&resourceVersion=0: net/http: TLS handshake timeout
...
... # there are quite many `net/http: TLS handshake timeout` messages here until the logs stop
...
2019-02-10T21:57:25.366656235Z E0210 21:57:25.366563       1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.PersistentVolumeClaim: Get https://msa-dev-az-xxxxxx.hcp.eastus.azmk8s.io:443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: net/http: TLS handshake timeout
2019-02-10T21:57:25.367127153Z E0210 21:57:25.366998       1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v2beta1.HorizontalPodAutoscaler: Get https://msa-dev--xxxxxx.hcp.eastus.azmk8s.io:443/apis/autoscaling/v2beta1/horizontalpodautoscalers?limit=500&resourceVersion=0: net/http: TLS handshake timeout
# this is the very last log line, current date is 2019-03-04 and still no logs, pod is still up and reporting stale metrics from that time, so last log line was 3 weeks ago

What you expected to happen:
kube-state-metrics should not be serving stale metrics more than a certain time (I'd say at most couple of minutes for our case).
Seems like the kube-state-metrics didn't try to reconnect to the kube-apiserver after having some issues with the kube-apiserver.
Instead of keep serving stale metrics kube-state-metrics should at least panic out our report unhealthy status.
How to reproduce it (as minimally and precisely as possible):
We didn't reproduce it yet. I'll update if we are able to reproduce.
Anything else we need to know?:
Here is a summary of the deployment status:
$ ptc msai msa-dev:seed insight deployment/msa-monitoring-kube-state-metrics -n monitoring   -r
Deployment/msa-monitoring-kube-state-metrics[monitoring], created 2 months ago
  desired:1, existing:1, ready:1, updated:1, available:1
  Available:True for 2 months MinimumReplicasAvailable :'Deployment has minimum availability.' last update was 2 months ago
  Progressing:True for 2 months NewReplicaSetAvailable :'ReplicaSet "msa-monitoring-kube-state-metrics-6bb98dc558" has successfully progressed.' last update was 2 months ago
Pod/msa-monitoring-kube-state-metrics-6bb98dc558-tlfvb[monitoring], created 2 months ago
  Running BestEffort
  Initialized:True for 2 months
  Ready:True for 4 days
  ContainersReady:True
  PodScheduled:True for 2 months
  Container: kube-state-metrics deploys quay.io/coreos/kube-state-metrics:v1.4.0
    ready:True
    state: running for 2 months

Environment:

Kubernetes version (use kubectl version): v1.11.5 --> Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Kube-state-metrics image version: quay.io/coreos/kube-state-metrics:v1.4.0

Duplicate issue #1157: Pod metrics reported for deleted pod
Created: 2020-06-12T20:03:57Z
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
Running kube-state-metrics, and a different pod is deleted. After the other pod is removed kube-state-metrics continues to report metrics for that pod indefinitely.
What you expected to happen:
Metrics pertaining to a deleted pod to no longer be updated soon after deletion.
How to reproduce it (as minimally and precisely as possible):
I don't know how to reproduce this deterministically. It has shown up in a couple environments, and restarting the kube-state-metrics pod will fix it.
It it is possibly related to kube-state-metrics startup. When we have seen this behavior usually the pod with metrics reported indefinitely was deleted shortly after the kube-state-metrics pod was started.
Anything else we need to know?:
I am running with auto-sharding turned on. I don't know if this is related or not.
Environment:

Kubernetes version (use kubectl version): 1.14.10
Kube-state-metrics image version: v1.9.5

Closing comment (2020-07-08T08:43:45Z): I have a feeling that this is a dupe of #694. I have some idea what might be wrong, but currently no time to look deeper into it.