Portal:Toolforge/Admin/Kubernetes/Certificates

This page contains information on certificates (PKI, X.509, etc) for the Toolforge Kubernetes cluster.
== General considerations ==
Kubernetes includes an internal CA which is the main one we use for cluster operations.
By default, kubernetes issued certificates are valid for 1 year. After that period, they should be renewed.
The internal kubernetes CA, generated at deployment time by kubadm expires after 10 years. The current CA is good until Nov 3 14:13:50 2029 GMT
Worth noting that etcd servers don't use the kubernetes CA, but use the puppetmaster CA instead.
Most certs can be checked for expiration with sudo kubeadm certs check-expiration on a control plane node.
== External API access ==
We have certain entities contacting external the kubernetes API. The authorization/authentication access is managed using a kubernetes ServiceAccount and a x509 certificate.
The x509 certificate encodes the ServiceAccount name in the Subject field.
Some examples of this:
* tools-prometheus uses this external API access to scrape metrics.
* TODO: any other example?
=== Operations ===
Certificates for this use case can be generated using a custom script we have: wmcs-k8s-get-cert .
Usually, the generated cert will be copy&pasted into the private puppet repo to be used as a secret in a puppet module or profile.
Renewing the certificate is just generating a new one and replacing the old one.
Example workflow for replacing tools-prometheus k8s certificate:
user@cloudcumin1001:~# sudo cumin "O{project:tools name:tools-prometheus}" 'disable-puppet "T12345 refreshing certificates"'
user@tools-k8s-control-3:~$ sudo -i wmcs-k8s-get-cert prometheus
/tmp/tmp.9k9N7ksn6K/server-cert.pem
/tmp/tmp.9k9N7ksn6K/server-key.pem
user@tools-k8s-control-3:~$ sudo cat /tmp/tmp.9k9N7ksn6K/server-cert.pem
-----BEGIN CERTIFICATE-----
MIIDYTCCA[...]
-----END CERTIFICATE-----
user@tools-k8s-control-3:~$ sudo cat /tmp/tmp.9k9N7ksn6K/server-key.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBA[...]
-----END RSA PRIVATE KEY-----
root@tools-puppetmaster-02:/var/lib/git/labs/private# vim modules/secret/secrets/ssl/toolforge-k8s-prometheus.key
# copy paste here the private key
root@tools-puppetmaster-02:/var/lib/git/labs/private# git commit -a
# Write the task you are working on in the commit and any details you find relevant you are done!
user@laptop:~/git/wmf/operations/puppet$ nano modules/profile/files/ssl/toolforge-k8s-prometheus.crt
create a patch similar to https://gerrit.wikimedia.org/r/c/operations/puppet/+/926484
user@cloudcumin1001:~# sudo cumin "O{project:tools name:tools-prometheus}" 'enable-puppet "T12345 refreshing certificates"'
== Internal API requests ==
Sometimes the Kubernetes API server makes requests to other services. For example:
* custom webhooks
* the internal metrics server (i.e, what kubectl top uses)
In general Kubernetes requires those requests to be encrypted and verified via TLS certificates. Historically we used to generate certificates for those using the Kubenetes internal CA, because that was possible and the easiest method. However due to changes in the Kubernetes certificates API, that is no longer possible. These days the modern approach for these is to use cert-manager to generate those certificates.
=== cert-manager ===
You will need to generate a certificate for the service. Self-signed certificates will work fine. Something like this:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: metrics-server-api-tls
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: metrics-server-api-tls
spec:
 dnsNames:
 - "metrics-server.metrics.svc"
 secretName: metrics-server-api-tls
 revisionHistoryLimit: 1
 issuerRef:
 name: metrics-server-api-tls
 kind: Issuer
 group: cert-manager.io
This will generate a certificate for that DNS name and save it to a secret called metrics-server-api-tls. To see the status of the certificate, use kubectl describe certificate:
taavi@tools-sgebastion-11:~ $ kubectl describe certificate -n metrics metrics-server-api-tls
Name: metrics-server-api-tls
Namespace: metrics
Labels: app=raw
 app.kubernetes.io/managed-by=Helm
 chart=raw-0.3.0
 heritage=Helm
 release=metrics-server-api-certs
Annotations: meta.helm.sh/release-name: metrics-server-api-certs
 meta.helm.sh/release-namespace: metrics
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
 Creation Timestamp: 2023-02-17T11:15:56Z
 Generation: 1
 Managed Fields:
 API Version: cert-manager.io/v1
 Fields Type: FieldsV1
 fieldsV1:
 f:status:
 f:revision:
 Manager: cert-manager-certificates-issuing
 Operation: Update
 Time: 2023-02-17T11:15:56Z
 API Version: cert-manager.io/v1
 Fields Type: FieldsV1
 Manager: helm
 Operation: Update
 Time: 2023-02-17T11:15:56Z
 Resource Version: 1059036361
 UID: 2e6e087e-06d9-4da5-b8ed-6109dbc38d6e
Spec:
 Dns Names:
 metrics-server.metrics.svc
 Issuer Ref:
 Group: cert-manager.io
 Kind: Issuer
 Name: metrics-server-api-tls
 Revision History Limit: 1
 Secret Name: metrics-server-api-tls
Status:
 Conditions:
 Last Transition Time: 2023-02-17T11:15:56Z
 Message: Certificate is up to date and has not expired
 Observed Generation: 1
 Reason: Ready
 Status: True
 Type: Ready
 Not After: 2023-05-18T11:15:56Z
 Not Before: 2023-02-17T11:15:56Z
 Renewal Time: 2023-04-18T11:15:56Z
 Revision: 1
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Issuing 17m cert-manager-certificates-trigger Issuing certificate as Secret does not exist
 Normal Generated 17m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "metrics-server-api-tls-hv7gg"
 Normal Requested 17m cert-manager-certificates-request-manager Created new CertificateRequest resource "metrics-server-api-tls-cm8gn"
 Normal Issuing 17m cert-manager-certificates-issuing The certificate has been successfully issued
On the api configuration (so usually either the webhook or APIService object), use cert-manager's CAInjector feature:
 annotations:
 # syntax: namespace/secret-name
 cert-manager.io/inject-ca-from: "metrics/metrics-server-api-tls"
Cert-manager will automatically renew the certificate when it has 1/3 of its lifetime remaining. If the service does not automatically re-load the changed certificate, you can use stakater/reloader to restart the deployment when the certificates change.
== Node/kubelet certs ==
This information has been moved to Portal:Toolforge/Admin/Kubernetes
== Tool certs ==
This information has been moved to Portal:Toolforge/Admin/Kubernetes
== Etcd certs ==
All etcd servers use puppetmaster issued certificates (puppet node certificates). The etcd service will only allow communication from clients presenting a certificate signed by the same CA.
This means kubernetes components that contact etcd should use puppet node certificates.
In the puppet profile controlling this, we have a mechanism to refresh the certificate and restart the etcd daemon if the puppet node certificate changes (it is reissued or whatever).
== See also ==
Some other interesting docs:
* Toolforge Kubernetes RBAC and PSP
* Toolforge Maintenance Script Documentation
* Upstream docs on kubeadm for certs management