Repository: prometheus-operator/kube-prometheus

Issue #881: KubeClientCertificateExpiration always alert
Created: 2018-09-03T07:34:31Z
What did you do?
wget https://codeload.github.com/coreos/prometheus-operator/tar.gz/v0.23.1 install prometheus-operator use kubectl create -f prometheus-operator-0.23.1/contrib/kube-prometheus/manifests/ || true
What did you expect to see?
all components work correctly.
Environment


K8s version: v1.11.0


Prometheus Operator version: v0.23.1


Kubernetes cluster kind:
      install k8s cluster use binary package  https://storage.googleapis.com/kubernetes-release/release/v1.11.0/kubernetes.tar.gz



Manifests:



[2] Firing
--
Labels
alertname = KubeClientCertificateExpiration
job = apiserver
prometheus = monitoring/k8s
severity = critical
Annotations
message = Kubernetes API certificate is expiring in less than 1 day.runbook_url = https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpirationSource

Labels
alertname = KubeClientCertificateExpiration
job = apiserver
prometheus = monitoring/k8s
severity = warning
Annotations
message = Kubernetes API certificate is expiring in less than 7 days.runbook_url = https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration



I used cfssl generate pem and keys
# openssl x509 -in /etc/kubernetes/ssl/ca.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6f:b9:70:eb:80:73:e6:73:f9:c8:29:98:99:5e:b5:f2:6d:a3:0e:49
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Shanghai, L=Shanghai, O=k8s, OU=System, CN=kubernetes
        Validity
            Not Before: Aug  8 09:54:00 2018 GMT
            Not After : Aug  7 09:54:00 2023 GMT
        Subject: C=CN, ST=Shanghai, L=Shanghai, O=k8s, OU=System, CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c8:ae:16:d6:0c:5b:30:95:97:a2:5b:16:cf:db:
                    f1:bd:68:8c:c6:0c:84:5b:a4:46:b4:79:0b:2b:c4:
                    b2:c0:5f:ab:e4:4a:33:46:d3:82:a3:33:bf:a7:f7:
                    ec:a3:4e:b3:70:34:e8:15:24:8e:56:b7:4d:68:9b:
                    e0:dc:0a:3a:3c:36:3e:f2:5c:be:d1:5d:fa:fa:e0:
                    7d:5b:2a:5d:e2:fc:94:9f:ea:a9:ce:ca:ad:2f:fd:
                    16:bc:fb:83:f6:45:fd:2f:9a:ac:94:e3:fd:49:90:
                    a1:31:95:cd:f2:30:2b:cd:31:34:69:b1:3a:b8:6a:
                    b8:7a:ef:f1:e9:ee:a2:5d:81:a8:59:80:77:c1:43:
                    85:3c:29:d8:02:fb:24:b9:9a:1f:e4:61:82:ec:8d:
                    49:3d:91:f7:0a:50:25:b1:a4:51:ba:f3:d6:77:07:
                    e2:50:ed:b8:af:30:18:d8:23:d6:e9:17:b1:a0:1c:
                    8c:74:f3:87:56:08:c7:49:86:c0:90:5e:16:a4:1e:
                    07:49:ef:b2:dc:9e:22:4c:b9:9b:7f:38:47:d7:26:
                    17:15:92:79:51:cc:a9:3f:4b:a1:6d:03:94:5b:9c:
                    03:c0:19:7e:d1:4e:c9:77:84:b1:e4:5b:a6:2b:54:
                    95:d0:a3:ef:39:d6:c3:88:77:af:4f:31:cd:ba:f7:
                    cc:3b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:2
            X509v3 Subject Key Identifier: 
                BC:9F:D1:BD:4C:26:E1:77:C0:7F:CF:04:3E:DF:64:86:BE:23:F3:7F
            X509v3 Authority Key Identifier: 
                keyid:BC:9F:D1:BD:4C:26:E1:77:C0:7F:CF:04:3E:DF:64:86:BE:23:F3:7F

    Signature Algorithm: sha256WithRSAEncryption
         78:b7:65:4d:53:e1:0c:7d:d6:9e:d5:aa:f8:1a:34:e4:1d:c0:
         22:4b:42:72:86:86:e9:73:e2:fd:89:90:e1:10:56:a7:f2:15:
         71:14:79:ce:67:9a:ca:5d:4d:e8:25:3d:70:2a:0a:3b:08:09:
         02:8a:d9:2d:ed:85:cd:10:38:60:75:d7:f5:a7:b2:ee:86:05:
         dd:50:38:04:a4:7a:bc:f5:02:b2:a5:d9:a2:a1:71:7d:e5:ce:
         dd:c8:5a:a7:25:61:de:c3:76:c3:87:3e:5a:4c:eb:36:91:51:
         8b:fc:ef:9d:aa:35:58:3a:ba:fc:2a:3c:4f:b3:54:e8:0d:a5:
         32:25:91:dd:93:75:33:53:2b:94:9e:f1:cb:e9:58:17:a6:dc:
         07:1c:96:5e:93:40:d6:c8:2b:67:49:3b:3f:1f:a8:3a:41:65:
         29:03:f3:18:f9:d3:66:a8:49:14:1e:7f:cb:6b:f6:26:1d:7b:
         6f:46:c6:27:a1:69:fe:62:7f:da:fb:41:7d:fc:ab:12:77:b8:
         b3:4c:92:a5:5c:d2:8c:25:a1:aa:1e:2f:a2:de:38:e5:9a:96:
         2f:b2:bb:3c:32:de:db:7f:80:eb:f0:01:be:2d:ff:00:09:35:
         ea:2b:8d:33:6e:6c:2c:6d:37:a2:c4:b3:c9:eb:ac:3f:ec:e5:
         5d:61:50:66


# openssl x509 -in /etc/kubernetes/ssl/kubernetes.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            76:64:c7:59:95:aa:fb:9b:8c:b2:26:c0:82:24:c5:0a:8d:95:a2:1e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Shanghai, L=Shanghai, O=k8s, OU=System, CN=kubernetes
        Validity
            Not Before: Aug  8 09:54:00 2018 GMT
            Not After : Aug  5 09:54:00 2028 GMT
        Subject: C=CN, ST=Shanghai, L=Shanghai, O=k8s, OU=System, CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b9:7e:1b:a9:9a:95:21:42:5a:e8:3e:79:94:e6:
                    c1:35:87:93:22:3d:3c:c9:65:be:b6:99:4b:47:25:
                    1a:22:db:4a:a5:b8:59:0d:2d:a0:0d:e5:c6:35:3b:
                    8e:2c:e3:fe:3a:d9:bc:63:9b:a0:98:c2:26:98:4c:
                    be:8b:71:20:37:a3:19:21:34:03:0b:10:d7:cb:7c:
                    b6:d8:68:90:1b:e1:6b:ee:b8:0e:6f:3d:33:2b:3f:
                    87:9a:4f:6c:59:08:f4:22:a6:2a:b6:d5:d6:00:b8:
                    7e:3c:90:aa:99:5c:6e:7c:93:f2:6b:6a:6f:5b:c6:
                    35:60:e0:14:62:5e:91:cc:20:eb:88:ea:cc:7a:10:
                    d7:f1:5f:b3:fb:aa:c4:a7:f5:95:3e:8a:44:ee:09:
                    12:6b:aa:29:05:40:df:1e:54:25:05:e2:8c:cb:d7:
                    32:e8:c5:ff:0c:48:11:27:c9:52:81:f2:53:b0:82:
                    b0:1b:7f:ad:08:fd:cd:b6:c1:4e:43:da:2d:f0:90:
                    90:cb:97:a2:2a:31:bc:65:2c:9f:a9:72:90:dd:b0:
                    5e:3b:7d:1c:37:d6:ca:22:13:2a:da:27:1d:61:94:
                    8f:36:9f:9d:6a:d1:6c:b9:17:58:5d:9c:0d:b1:d8:
                    2a:98:f1:54:d7:87:c6:da:ff:05:c9:a2:c5:91:5a:
                    77:23
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                B3:1C:65:F4:DA:61:57:1F:68:06:05:46:36:31:BC:AF:E1:D5:06:7C
            X509v3 Authority Key Identifier: 
                keyid:BC:9F:D1:BD:4C:26:E1:77:C0:7F:CF:04:3E:DF:64:86:BE:23:F3:7F

            X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.2.93, IP Address:10.100.0.1, IP Address:192.168.2.86, IP Address:192.168.2.87, IP Address:192.168.2.88
    Signature Algorithm: sha256WithRSAEncryption
         2d:a6:ee:28:71:0f:ea:69:ff:90:25:d6:04:4e:4c:e1:3d:ff:
         34:f1:64:67:4f:ab:80:ee:f5:d9:16:53:48:0c:c4:fd:9a:f0:
         09:13:71:b1:ba:52:b0:36:38:6b:51:be:ac:cc:14:30:2b:e7:
         a9:87:00:76:fe:1a:58:72:45:27:0a:59:51:74:65:6a:30:ea:
         37:f3:c9:79:59:f0:09:87:e9:94:99:00:11:d7:20:9c:90:5c:
         de:ee:09:ff:53:07:41:06:4c:91:8d:8a:d1:d5:ff:30:06:3b:
         53:32:4c:dd:70:f0:22:7f:7d:e6:02:f2:eb:a6:fd:5a:de:d6:
         0d:fa:b5:e9:f0:95:5a:79:bb:f9:b5:a5:47:01:13:3f:b0:12:
         c6:35:11:45:2f:6b:f3:71:26:92:8f:34:90:0f:42:d8:2a:12:
         0f:ad:96:1f:60:54:5c:27:f3:0f:c3:4e:f5:ef:58:75:51:7a:
         df:8c:f3:b2:d4:b8:70:99:ff:e3:5a:ee:a9:00:69:84:a3:c2:
         df:7e:9b:55:e1:ab:92:bb:55:8b:54:6c:aa:05:c4:ea:29:8e:
         56:72:15:11:c2:6e:49:72:b5:d7:30:06:7b:c4:a2:0a:82:87:
         19:83:b7:1e:3a:86:02:35:f5:21:e8:e6:bf:5e:51:c0:ec:f0:
         c1:3d:15:35


# openssl x509 -in /etc/kubernetes/ssl/admin.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            30:7e:a9:d4:1c:0a:04:d7:3b:2a:38:7a:b3:ca:25:fb:65:e3:e6:72
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Shanghai, L=Shanghai, O=k8s, OU=System, CN=kubernetes
        Validity
            Not Before: Aug  8 09:55:00 2018 GMT
            Not After : Aug  5 09:55:00 2028 GMT
        Subject: C=CN, ST=Shanghai, L=Shanghai, O=system:masters, OU=System, CN=admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:cb:10:41:82:61:ec:93:e8:4d:bf:3e:2d:88:45:
                    ce:e8:57:ee:c6:90:8c:a2:e7:7b:16:ae:9e:fc:6e:
                    60:25:5c:f4:26:c2:50:c7:b5:1e:d3:91:d8:54:e9:
                    5b:6f:85:0e:0a:56:2c:e8:4d:69:dc:06:1e:94:92:
                    29:b9:7c:6f:cd:bd:25:13:bf:c9:9b:98:dd:81:f2:
                    0e:df:27:17:75:c9:4f:d8:9a:9c:5c:b0:db:9c:ed:
                    bb:a5:1f:c1:df:85:9a:f9:62:6b:a8:7a:96:69:30:
                    93:2f:e9:e3:16:dc:74:5f:4d:68:5d:e3:05:ae:01:
                    bd:60:72:d0:30:7c:3b:01:7a:13:9f:4c:ef:62:f2:
                    6c:47:6a:25:6f:b4:0c:7a:53:db:78:a4:71:00:c8:
                    6c:a7:c6:39:42:cf:da:e0:20:ce:66:02:36:43:13:
                    5a:56:7d:da:77:ad:01:4f:ab:56:54:6d:b9:27:08:
                    4e:d6:95:8b:cd:90:5f:28:c2:63:de:d8:f9:77:4f:
                    6d:35:02:9b:6c:cf:27:43:8a:47:b0:74:7e:25:c5:
                    6c:2d:7a:4b:e1:49:af:e7:28:d1:e0:3b:2a:21:1d:
                    bd:09:80:f7:4f:ee:a9:23:50:8c:65:55:0b:fd:d8:
                    4b:4b:b3:82:cb:2a:9f:33:c7:d3:88:63:91:ca:f9:
                    e1:a7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                DA:B4:8B:36:C7:E9:9C:C0:6E:AC:8D:1F:D6:18:93:76:4D:6E:78:1F
            X509v3 Authority Key Identifier: 
                keyid:BC:9F:D1:BD:4C:26:E1:77:C0:7F:CF:04:3E:DF:64:86:BE:23:F3:7F

    Signature Algorithm: sha256WithRSAEncryption
         2f:69:9c:6f:53:bb:7a:42:e3:4e:8f:b4:17:00:10:90:c3:1c:
         be:68:05:f3:15:6a:aa:0c:53:eb:89:c6:0c:2e:c2:0a:75:14:
         16:09:7e:68:0e:83:5c:c9:79:e0:ab:86:ee:93:d7:de:50:66:
         98:3d:5a:43:e0:7f:dd:dc:8a:b8:83:84:84:d4:0f:a5:c5:a1:
         b2:4a:65:76:15:e7:85:f3:7d:37:ee:e2:50:70:28:85:e8:05:
         05:d1:60:74:40:e2:67:7a:31:32:39:e3:96:e3:5b:fe:5e:eb:
         36:ef:cf:fa:95:37:9c:f1:3a:f5:11:80:e8:80:f9:1c:39:04:
         a0:14:af:e0:e7:ac:ce:6f:ad:4a:f3:e8:24:13:20:72:46:15:
         da:9a:e3:1d:88:c5:3d:93:12:7c:71:d3:77:95:5b:cd:f7:3b:
         b3:33:5d:10:31:7e:d9:ba:0e:ed:c8:61:9a:e7:df:fa:75:f1:
         f4:e5:67:81:be:3b:4a:5d:1e:82:1e:64:f7:16:14:4c:d9:e1:
         09:56:81:f4:64:21:47:79:f2:50:55:bb:e1:28:21:40:22:7d:
         f6:b7:f1:cd:3f:99:e5:96:c9:ee:76:be:03:68:da:7a:94:f5:
         ad:bb:40:66:cc:8c:85:36:91:3d:6a:5e:f6:d8:71:23:9e:f1:
         97:ff:73:ea


my k8s cluster  and prometheus  seems fine. but  KubeClientCertificateExpiration always trigger alert, how do I fix it ?

Duplicate issue #2133: KubeClientCertificateExpiration sudden alarm
Created: 2023-06-06T09:58:08Z
Hello,
I have 3 kubernetes 1.18 cluster. Prometheus is v2.20.0, Prometheus Operator is v0.42.1.
KubeClientCertificateExpiration had a sudden alarm, I check k8s Certificate, they don't going to  expire in one week or one day:


KubeClientCertificateExpiration appeared for some time and then disappeared:


There is no error logs in prometheus:

The KubeClientCertificateExpiration formula is
apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by(cluster, job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 604800

and
apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by(cluster, job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 86400

I think this issue maybe is same as #1089. But there is no root cause yet.

Closing comment (2023-06-15T09:41:04Z): Also related to #881