Portal:Toolforge/Admin/Kubernetes

Kubernetes (often abbreviated k8s) is an open-source system for automating deployment, and management of applications running in containers. Kubernetes was selected in 2015 by the Cloud Services team as the replacement for Grid Engine in the Toolforge project.[[mailarchive:labs-announce/2015-September/000071.html|[Labs-announce] [Tools] Kubernetes picked to provide alternative to GridEngine]]
 Usage of k8s in Tools began in mid-2016,[Labs-announce] Kubernetes Webservice Backend Available for PHP webservices
 and the current cluster design dates back to early 2020.News/2020 Kubernetes cluster migration
== Subpages ==
== About this document ==
This document tries to document the Kubernetes cluster used in Toolforge, and its direct support services (e.g. etcd). It does not cover specifics about services running in the cluster (e.g. the Jobs framework and build service), nor does it cover Toolforge services that are fully unrelated to the Kubernetes cluster (e.g. Redis).
The four main sections of this document correspond to the four categories of documentation in The Grand Unified Theory of Documentation system in a structure inspired by how the Tor Project Admins do it.
== Tutorial ==
=== Access kubectl ===
kubectl is the official Kubernetes command line interface tool. Assuming you are listed as a maintainer of the admin tool (or the toolsbeta equivalent) you will automatically have superuser credentials provisioned in your NFS home directory.
To use the CLI tool, log in to a bastion host on the project where the cluster you want to interact with is located. If you want to just experiment, you should use the toolsbeta cluster for that. Most read-only commands can be used out of the box, for example to list pods in the tool-fourohfour namespace used by the 404 handler:
$ kubectl get pod -n tool-fourohfour
NAME READY STATUS RESTARTS AGE
fourohfour-7766466794-gtpgk 1/1 Running 0 7d20h
fourohfour-7766466794-qctt8 1/1 Running 0 6d18h
However, all write actions and some read-only actions (e.g. interacting with nodes or secrets) will give you a permission error:
$ kubectl delete pod -n tool-fourohfour fourohfour-7766466794-gtpgk
Error from server (Forbidden): pods "fourohfour-7766466794-gtpgk" is forbidden: User "taavi" cannot delete resource "pods" in API group "" in the namespace "tool-fourohfour"
If you're sure you want to continue, you need to use kubectl sudo:
$ kubectl sudo delete pod -n tool-fourohfour fourohfour-7766466794-gtpgk
pod "fourohfour-7766466794-gtpgk" deleted
=== Manage pods ===
Pods are the basic unit of compute in Kubernetes. A pod consists of one or more OS-level containers that share a network namespace.
==== List pods ====
Pods can be listed with the kubectl get pod command. Log in to a toolsbeta bastion, become fourohfour and run:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fourohfour-bd4ffc5ff-479sj 1/1 Running 0 43s
fourohfour-bd4ffc5ff-4lhcf 1/1 Running 0 35s
The -o (--output) flag can be used to customize the output. For example, -o wide will display more information:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fourohfour-bd4ffc5ff-479sj 1/1 Running 0 91s 192.168.120.158 toolsbeta-test-k8s-worker-nfs-1 <none> <none>
fourohfour-bd4ffc5ff-4lhcf 1/1 Running 0 83s 192.168.145.16 toolsbeta-test-k8s-worker-nfs-2 <none> <none>
Or -o json will display the data in JSON:
$ kubectl get pods -o json | head -n5
{
 "apiVersion": "v1",
 "items": [
 {
 "apiVersion": "v1",
So far we have only been accessing data in the namespace we are in. To access data in any namespace, we need to switch back to our user account. Now we can use the -n (--namespace) flag to specify which namespace to access.
$ kubectl get pod -n tool-fourohfour
NAME READY STATUS RESTARTS AGE
fourohfour-bd4ffc5ff-479sj 1/1 Running 0 4m27s
fourohfour-bd4ffc5ff-4lhcf 1/1 Running 0 4m19s
$ kubectl get pod -n tool-admin -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
admin-cb6d84bd8-pshh7 1/1 Running 0 5d21h 192.168.25.21 toolsbeta-test-k8s-worker-nfs-4 <none> <none>
Or we can use -A (--all-namespaces) to list data in the entire cluster:
$ kubectl get pod -A | head -n5
NAMESPACE NAME READY STATUS RESTARTS AGE
api-gateway api-gateway-nginx-6ddddd6f64-mbnlg 1/1 Running 0 12d
api-gateway api-gateway-nginx-6ddddd6f64-tdl6c 1/1 Running 0 8d
builds-admission builds-admission-7897cf7759-jtxb5 1/1 Running 0 28h
builds-admission builds-admission-7897cf7759-nvmzt 1/1 Running 0 26h
==== View logs for pod ====
To view the combined standard output and standard error for a pod, use kubectl logs:
$ kubectl get pod -n maintain-kubeusers
NAME READY STATUS RESTARTS AGE
maintain-kubeusers-55b649885c-px8c6 1/1 Running 0 87m
$ kubectl logs -n maintain-kubeusers maintain-kubeusers-55b649885c-px8c6 | wc -l
176
Some useful flags for this command are:
* --tail NUMBER to only show the specified number of most recent lines
* --follow to do, well, exactly what it says
==== Restart a pod ====
=== Manage workers ===
The main Toolforge cluster consists of a bit over 50 "normal" NFS-enabled workers, and some special workers used for specific purposes. These workers can be added and removed using cookbooks. Both adding and removing a node is fairly straightforward, although due to the time it takes to replace the entire cluster we prefer to update existing nodes instead of replacing the entire cluster during most routine maintenance (e.g. Kubernetes upgrades or node reboots). It is however totally fine to replace nodes in Toolsbeta if you want to try the process.
==== Add a worker ====
These cookbooks can be run from the cloudcumin hosts (recommended) or from your laptop if you have them set up locally. Use of screen or tmux is recommended.
To create a normal worker_nfs in toolsbeta, use:
$ sudo cookbook wmcs.toolforge.add_k8s_node --cluster-name toolsbeta --role worker_nfs
==== Remove a worker ====
Removing a worker is equally straightforward. To remove the oldest worker_nfs node in toolsbeta, use:
$ sudo cookbook wmcs.toolforge.remove_k8s_node --cluster-name toolsbeta --role worker_nfs
If you have a specific node that you want to remove, pass that as a parameter:
$ sudo cookbook wmcs.toolforge.remove_k8s_node --cluster-name toolsbeta --role worker_nfs --hostname-to-remove toolsbeta-test-k8s-worker-nfs-1
==== Drain and undrain a node ====
Sometimes a node is misbehaving or needs maintenance done on it, and needs to be drained from all workload. This is easiest done with the cookbook:
$ sudo cookbook wmcs.toolforge.k8s.worker.drain --cluster-name toolsbeta --hostname-to-drain toolsbeta-test-k8s-worker-nfs-1
To "uncordon" (allow new pods to be scheduled to it again) the node, run the following on a bastion in the relevant project:
$ kubectl sudo uncordon toolsbeta-test-k8s-worker-nfs-1
node/toolsbeta-test-k8s-worker-nfs-1 uncordoned
You can also just "cordon" a node which will prevent new workloads from being scheduled but won't drain existing ones:
$ kubectl sudo cordon toolsbeta-test-k8s-worker-nfs-1
node/toolsbeta-test-k8s-worker-nfs-1 cordoned
That is also reversed with the uncordon command.
== How-to ==
=== Cluster management ===
==== Build a new cluster ====
We have not built a new cluster since the 2020 cluster redesign. The documentation written during the 2020 redesign is at Portal:Toolforge/Admin/Kubernetes/Deploying, although it is likely somewhat outdated.
==== Upgrade Kubernetes ====
Kubernetes upstream releases new versions about three times a year.Kubernetes releases We cannot skip any upgrades and thus must upgrade sequentially. This process is documented at Portal:Toolforge/Admin/Kubernetes/Upgrading Kubernetes.
==== Upgrade Calico ====
==== Upgrade ingress-nginx ====
==== Upgrade worker operating system ====
We have upgraded the cluster OS once, from Buster to Bookworm, and during the same time changed the container runtime from Docker to containerd.Toolforge k8s: Migrate workers to Containerd and Bookworm There is no set process or specific automation for this, but the approach taken last time was:
# Pick which Debian release you're going to upgrade to
# Ensure the container runtime version in that release is supported by Kubernetes, Calico and cadvisor
# Import kubeadm packages for the new Debian release
# Add a new worker in toolsbeta
## Test carefully that it works
## Do this for all types to test out all configuration combinations (with/without NFS, with/without extra volume)
## Remove matching number of old workers
# Replace a control node in toolsbeta
# Add a few new nodes in tools
# Wait a few days
# Replace all tools workers
## In paraller, replace remaining toolsbeta workers
# Replace tools controls
==== Roll reboot cluster ====
The wmcs.toolforge.k8s.reboot cookbook can be used to reboot the entire cluster, for example to apply Kernel or container runtime updates, or in case the NFS server is having issues. Start from reading the --help output for the cookbook. For example, in the NFS issue case in toolsbeta, you could run:
$ sudo cookbook wmcs.toolforge.k8s.reboot --cluster-name toolsbeta --all-workers
==== Renew kubelet certificates ====
All kubeadm-managed kubelet client certs are rotated when upgrading Kubernetes, but they can be manually rotated with kubeadm as well. It is possible to configure the kubelet to request upgraded certs on its own when they near expiration. So far, we have not set this flag in the config, expecting our upgrade cycle to be 6 months, roughly.
To renew certificates run the wmcs.toolforge.k8s.kubeadm_certs_renew cookbook. You would usually run the cookbook for all control nodes in a given cluster. The cookbook is idempotent and can be run at any time safely. For example, to renew the certs on all toolsbeta control nodes, you would run>
$ sudo cookbook wmcs.toolforge.k8s.kubeadm_certs_renew --project toolsbeta --control-hostname-list toolsbeta-test-k8s-control-4 toolsbeta-test-k8s-control-5 toolsbeta-test-k8s-control-6
See also upstream docs: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal
=== etcd ===
==== Add etcd node ====
==== Remove etcd nodes ====
==== Upgrade etcd ====
We run etcd from the Debian packages, so an etcd upgrade is automatically a Debian upgrade and vice versa.
We have not upgraded etcd yet since the 2020 cluster redesign. This section should be filled when we do that for the first time.
=== Component system ===
In the Toolforge Kubernetes component workflow improvements enchancement proposal we introduced a standard "components" system for various components that run in the Kubernetes cluster.
==== Deployment ====
This is documented in the toolforge-deploy.git README file.
=== Manage (tool) users ===
==== Modify quotas ====
Tool quotas are managed by maintain-kubeusers and configured in in the values file in toolforge-deploy.git.Track and apply Toolforge quota changes via a Git repository To update quotas for a specific tool:
# Send a patch to the values file changing the quotas. The format should be relatively self-explanatory, and the defaults and supported keys are listed in the default values file. Always change the version when making any kind of change or it will not be applied.
# Merge the patch to main and deploy it like any other component change.
==== Regenerate .kube/config ====
In case something goes wrong with the credentials for a certain tool user, you can delete the maintain-kubeusers configmap which will cause maintain-kubeusers to re-generate the credentials for that user. On a bastion in the relevant project, run:
$ kubectl sudo delete cm -n tool-$TOOL maintain-kubeusers
Please have a look at the logs for maintain-kubeusers and file a bug so the issue can be fixed.
==== Enable observer access ====
Requests for observer access must be approved by the Toolforge admins in a Phabricator task. Once approved, they can be implemented on a control plane node with:
$ sudo -i wmcs-enable-cluster-monitor <tool-name>
=== Manage user workloads ===
==== Find newly added workloads ====
The Kubernetes capacity alert runbook documents how to find where a sudden increase in workload has come from.
==== Pod tracing ====
Given all tools running on a single worker share that worker's IP address, occasionally you need to figure out which tool on a given worker is misbehaving. That process is documented on Portal:Toolforge/Admin/Kubernetes/Pod tracing.
=== Update prebuilt images ===
This has been moved to the Jobs framework documentation.
== Reference ==
=== Admission controllers ===
 Custom admission controllers in the Toolforge cluster
Repository 	Related to functionality	Description
builds-admission 	Build Service 	Validate build service user-created pipelines
envvars-admission 	Envvars Service 	Inject configured envvars to pods
ingress-admission 	Webservice 	Validate created ingress objects use the domain allowed for that tool
registry-admission	Jobs framework 	Validate new pods use images in the Toolforge docker registry or Harbor
volume-admission 	Jobs framework 	Inject NFS mounts to pods that are configured to have them
=== Authentication, authorization, certificates and RBAC ===
==== cert-manager ====
==== External certificates ====
==== maintain-kubeusers ====
maintain-kubeusers is responsible for creating Kubernetes credentials and a namespace (tool-[tool name]) for each tool, and removing access for disabled tools. It is also in charge of maintaining quotas and other resources for each tool (like kyverno security policies, etc). In addition, it creates admin credentials all maintainers of the admin tool.
The service is written as a long-running daemon, and it talks to LDAP directly for tool data. It exports Prometheus metrics, but those are not used for any alerts or dashboards at this moment.
==== kubelet certificates ====
Kubelet has two certs:
# A client cert to communicate with the API server
# A serving certificate for the Kubelet API
At this time the serving certificate is a self-signed one managed by kubelet, which should not need manual rotation. Proper, CA-signed rotating certs are stabilizing as a feature set in Kubernetes 1.17, and we should probably switch to that for consistency and as a general improvement. The client cert of kubelet is signed by the cluster CA and expires in 1 year. They are automatically renewed when the cluster is upgraded, but can be renewed manually as well.
==== Observer access ====
Some tools (e.g. k8s-status) need more access to the Kubernetes API than what the default credentials require. For these tools, an "observer" role has been created that grants read-only access to non-sensitive data about the cluster and workloads that run on it.Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection The role is deployed from a file deployed from Puppet (although phab:T328539 proposes moving it to maintain-kubeusers), and role bindings are created manually using a script.
Using observer status in a job with serviceAccountName: ${tool}-obs is not supported by the Jobs framework or webservice. The k8s-status tool uses a custom script for managing a web service with such access included.
Requests for such access should be approved by the Toolforge admins before access is granted.
=== Backups ===
The main thing worth backing up is the contents of the etcd cluster. It is not currently backed up.
=== Bastion nodes ===
The Toolforge bastion nodes have kubectl installed. As the bastion nodes have NFS mounts, and maintain-kubeusers provisions certificates to NFS, everything will then work out of the box.
=== Kubernetes design ===
The Kubernetes documentation is both more detailed and up-to-date. Here is, however, a quick overview of the major Kubernetes components.
==== Control plane ====
===== etcd =====
Kubernetes stores all state in etcd - all other components are stateless. The etcd cluster is only accessed directly by the API Server and no other component. Direct access to this etcd cluster is equivalent to root on the entire k8s cluster, so it is firewalled off to only be reachable by the rest of the control plane nodes as well as etcd nodes, has client certificate verification in use for authentication (puppet is CA) and secrets are encrypted at rest in our etcd setup.
We currently use a 3 node cluster, hosted on VMs separate from the main control plane. They're all smallish Debian Buster instances configured largely by the same etcd puppet code we use in production. The main interesting thing about them is that they're localdisk instances as etcd is rather sensitive to iowait.
===== API server =====
The API server the heart of the Kubernetes control plane. All communication between all components, whether they are internal system components or external user components, must go through the API server. It is purely a data access layer, containing no logic related to any of the actual end-functionality Kubernetes offers. It offers the following functionality:
* Authentication & Authorization
* Validation
* Read / Write access to all the API endpoints
* Watch functionality for endpoints, which notifies clients when state changes for a particular resource
When you are interacting with the Kubernetes API, this is the server that is serving your requests.
The API server runs as a static pod on the control plane nodes. It listens on port 6443/tcp, and all access from outside the Kubernetes cluster should go via HAProxy. Requests are authenticated with either tokens (mostly for internal usage) or client certificates signed via the certificates API.
===== controller-manager and scheduler =====
The controller-manager and scheduler contain most of the actual logic. The scheduler is responsible for assigning pods to nodes and the controller-manager is for most other actions, for example launching CronJobs at scheduled times or ensuring ReplicaSets have the correct number of Pods running. The general idea is one of a 'reconciliation loop' - poll/watch the API server for desired state and current state, then perform actions to make them match.
==== Worker ====
The primary service running on each node is the Kubelet, which is an interface between the Kubernetes API and the container runtime (containerd in our case). Kubelet is responsible for ensuring the pods running on the node match with what the API server wants to run on that node, and reports back metrics to the API. It also proxies logs requests when necessary. Pod health checks are also done by the Kubelet.
In addition, there are two networking-related services running on each node:
* kube-proxy manages iptables NAT rules for Service addresses.
* The container network interface (or CNI, Calico in our cluster) manages the rest of the cluster networking. In practice this means an overlay network where each pod is assigned an cluster-internal IP address which can be routed across the entire cluster.
=== Labels ===
A reference of various used Kubernetes labels and their meanings is available on Portal:Toolforge/Admin/Kubernetes/Labels.
=== Monitoring and metrics ===
==== Alert runbooks ====
* Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity
* Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady
==== Kubernetes metrics stack ====
The Kubernetes cluster runs multiple pieces of software responsible for cluster monitoring:
* metrics-server (per-container and node metrics for Kubernetes internal use)
* cadvisor (per-container metrics for Prometheus)
* kube-state-metrics (cluster-level metrics from Prometheus)
These are all deployed via the wmcs-k8s-metrics component using the standard component deployment model.
==== Prometheus integration ====
Toolforge Prometheus servers scrapes cadvisor, kube-state-metrics and Prometheus exporter endpoints in the apps that have them. For this, the Prometheus server have an external API certificate provisioned via Puppet that needs to be renewed yearly. The scrape targets are defined in the profile::toolforge::prometheus Puppet module.
Alerts are managed via the Alerts GitLab repo and sent via metricsinfra infrastructure.
=== Networking ===
==== Calico ====
We use Calico as the Container Network Interface (CNI) for our cluster. In practice Calico is responsible for allocating a small private subnets in 192.168.0.0/16 for each node, and then routing those subnets to provide full connectivity across all nodes.
We deploy Calico following their self-managed on-premises model. We do not use their operator deployment - instead, we take their manifest deployment and build a Helm chart from it. Instructions for upgrading Calico in this setup are in the upgrade Calico section.
==== DNS ====
In the cluster we use the default CoreDNS DNS plugin. It resolves cluster-internal names (e.g. services) internally and forwards the remaining queries to the main Cloud VPS recursor service. CoreDNS configuration is managed by Kubeadm and generally works well enough, although we should consider increasing the number of replicas.
==== Ingress ====
We use kubernetes/ingress-nginx to route HTTP requests to specific tools inside the Kubernetes cluster. Ingress objects are created by webservice (soon jobs-api), and the ingress admission controller restricts each tool to [toolname].toolforge.org.
==== HAProxy (external service access) ====
=== NFS and LDAP ===
The worker nodes are Puppetized which means they have the standard Cloud VPS SSSD setup for using LDAP data for user accounts.
In addition, most (as of February 2024) worker nodes have the shared storage NFS volumes mounted, and these nodes have the kubernetes.wmcloud.org/nfs-mounted=true and toolforge.org/nfs-mounted=true for tools to run NFS-requiring workloads on them. The volume-admission-controller admission controller mounts all volumes to pods with the toolforge: tool label.
There are plans to introduce non-NFS workers to the pool once the Bookworm OS upgrades have finished. These would be used by tools with build service images, buildservice builds and infrastructure components with no need for NFS connectivity. Given the reliability issues with NFS, new features should be designed in a way that they at least do not make it harder to move away from NFS.
=== Pod isolation ===
We use kyverno to enforce a set of security and isolation constraints to all tool account Pod workloads running in the cluster.
Examples of things we ensure:
* pods have a limited set of privileges and cannot escalate
* pods cannot "hijack" files from other tools via jumping NFS data dir
* pods don't run as root
With kyverno, we not only validate that pods are correct, but we also mutate (modify) them to inject some values we want them to be set to something in particular, like the uid/gid of each tool account.
Each tool account has a Kyverno policy resource created by maintain-kubeusers.
Privileged workloads, like custom components we deploy, or internal kube-system components, are not subject to any Kyverno policy enforcement as of this writing.
=== Testing and local deployments ===
We have a testing deployment in the toolsbeta Cloud VPS project. It is almost identical to the tools cluster except it is much smaller.
The lima-kilo project can be used to run parts of a Toolforge Kubernetes cluster on a local machine.
=== User workloads ===
==== Jobs framework ====
The recommended way for someone to run a workload on Toolforge is to use the Jobs framework (admin docs). The framework will create deployment, cronjob and job objects in tool namespaces.
==== Raw Kubernetes API users ====
Before the Jobs framework was introduced, many users used the Kubernetes API directly to run their tools. This is now deprecated, but tools are still using it because it works.
==== Build service builds ====
The Build service (admin docs) runs builds in the image-build namespace. All of this is managed via the build service API, users do not have direct access to that namespace. These builds run without NFS access.
=== Worker types ===
There are a few different types of workers.
Cookbook name	Name prefix	Description
worker 	worker 	Normal workers. As of February 2024, these do not have NFS access.
worker_nfs 	worker-nfs 	Normal workers with NFS.
control 	control 	Special purpose nodes for the Kubernetes control plane.
ingress 	ingress 	Special purpose workers exclusively for the web ingress and the API gateway.
Addition and removal of all of these types is fully automated via the cookbooks.
== Discussion ==
=== Bring-your-own-image ===
We only allow running images from the from the Toolforge Docker registry (for "pre-built" images) and from the Toolforge Harbor server. This is for the following purposes:
# Making it easy to enforce our Open Source Code only guideline
# Make it easy to do security updates when necessary (just rebuild all the containers & redeploy)
# Faster deploys, since this is in the same network (vs dockerhub, which is retreived over the internet)
# Access control is provided totally by us, less dependent on dockerhub
# Provide required LDAP configuration, so tools running inside the container are properly integrated in the Toolforge environment
This is enforced with an admission controller.
The decision to follow this approach was last discussed and re-evaluated at Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T302863_toolforge_byoc.
=== GitOps tools ===
=== Puppet ===
We use Puppet to provision the Kubernetes nodes, and also related non-K8s managed infrastructure such as etcd and HAProxy. However, configuration for what's inside the cluster should not be managed by Puppet for several reasons:
* We already have a deployment management system for what's inside the cluster (toolforge-deploy.git), and we should not introduce two systems for the same purpose
* Puppet cannot be used to provision a local environment as is
* puppet.git merges require global root, which not all Toolforge admins have
=== Single cluster reliance ===
== References ==