nyuuzyou/wikis huggingface dataset:
Portal:Toolforge/Admin/Runbooks/HarborComponentDown

This happens when one of the Harbor components is down or not reporting being up.
This might make harbor unusable by toolforge and prevent any image from being pulled/pushed, making any non-cached run fail to start or build a new image for a tool.
It might prevent also from deploying some of the toolforge system components.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for, the harbor instance, and the component that is failing, or a message saying that none of them is reporting anything.
If none is reporting anything, it might be an issue with prometheus/alertmanager.
== Debugging ==
You can ssh to the harbor instance directly and check there how/if it's running (use the instance of the project the alert is from):
You can try to restart/start it again, with docker-compose restart and docker-compose up -d.
You can also check the logs of each component with docker logs harbor-portal, where harbor-portal is the name of the component.
== Common issues ==
Add new issues here when you encounter them!
=== Issue 1 ===
...
== Related information ==
* Harbor admin docs
* Harbor original implementation docs
* Toolforge admin docs
== Old incidents ==
* T354714 - Trove DB filled disk and caused toolforge-build to fail as a result
Portal:Toolforge/Admin/Runbooks/HarborDown

This happens when the prometheus host is not able to do an https request to the harbor instance.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for, and the url of the harbor instance that fails.
Note that this request goes through the proxies and it might be an issue there instead of just harbor.
== Debugging ==
* You can ssh to the harbor instance directly and check there how/if it's running (use the instance of the project the alert is from): 
* You can try to restart/start it again, with docker-compose restart and docker-compose up -d.
* You can also check the logs of each component with docker logs harbor-portal, where harbor-portal is the name of the component.
* Logs may also be found in /var/log/harbor/
== Common issues ==
Add new issues here when you encounter them!
=== Issue 1 ===
...
== Related information ==
* Harbor admin docs
* Harbor original implementation docs
* Toolforge admin docs
== Old incidents ==
* T354714 - Trove DB filled disk and caused toolforge-build to fail as a result
Portal:Toolforge/Admin/Runbooks/HarborProbeUnknown

This happens when prometheus has no data from the blackbox exported on the harbor instance for the project.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmcloud.org/tools/classic/graph?g0.range_input=1h&g0.expr=probe_success%7Bjob%3D%22probes%2Fpingthing%22%7D&g0.tab=1
== Common issues ==
Add new issues here when you encounter them!
=== Issue 1 ===
...
== Related information ==
* Karma UI for cloud vps (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/JobsApiDown

This happens when the jobs-api pod in the jobs-api namespace of tools/toolsbeta k8s cluster is down or can't be reached.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
The first most likely step is to ssh to tools/toolsbeta (depending on the project the alert is from) k8s-control servers (i.e toolsbeta-test-k8s-control-4.toolsbeta.eqiad1.wikimedia.cloud). From there you can:
* check that the pods are running:
* You can also check the log of the pod's deployment with kubectl logs -n jobs-api deploy/jobs-api.
* You can force a restart of all the pods with a rollout kubectl rollout restart -n jobs-api deployment/jobs-api
* It might also make sense to check if there has been any recent code change and re-deployment attempts. Again a good place to start is by looking at the recent commits in jobs-api gitlab repo, or the toolforge-deploy gitlab repo.
* If the pods don't exist or the deployment does not exist, you can try redeploying the jobs-api by following the instructions in the toolforge repo (it will do nothing if there's nothing to do).
=== Doing a manual curl for the stats ===
You can try doing a curl directly to the pods for the statisticts, by checking the configuration of prometheus, you'll get the cert, key and url:
Then you can curl directly the pods by name, like:
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If jobs-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* [API docs]
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
Portal:Toolforge/Admin/Runbooks/JobsApiUpMetricUnknown

This happens when prometheus has no data from k8s on the jobs-api pod.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmflabs.org/tools/graph?g0.expr=up%7Bjob%3D%22jobs-api%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If jobs-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though) Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
== Related information ==
* Karma UI (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown

== Source code ==
The source code for the maintain-kubeusers daemon itself is at:
* https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers
The deployment of the code is done using toolforge-deploy, at:
* https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy
The prometheus alerts are defined via Toolforge alerts repository:
* https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts
== Logs ==
This is how you can get the logs for maintain-kubeusers:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers logs deploy/maintain-kubeusers --timestamps=true
In case you want to check the logs for a previous container restart:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers logs deploy/maintain-kubeusers --timestamps=true --previous
== Get a shell inside the pod ==
This is how you can get a shell inside the maintain-kubeusers pod:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- bash
root@maintain-kubeusers-76d6df5465-jgdwt:/app#
[..]
== See prometheus metrics inside the container ==
This is often useful to verify what is being scraped by prometheus:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- apt-get update
[..]
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- apt-get install curl -y
[..]
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- curl localhost:9000/metrics
[..]
maintain_kubeusers_run_seconds_count 32.0
maintain_kubeusers_run_seconds_sum 4373.660967367701
maintain_kubeusers_run_finished 1.7175907241729112e+09
== Dashboards ==
See grafana dashboard:
* https://grafana.wmcloud.org/d/E9HyRBySk/maintain-kubeusers?from=now-24h&to=now&orgId=1
Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon

This happens when the certificate that prometheus uses to fetch data from k8s is about to expire.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) and which instance (VM) is the cert about to expire (all should have similar times though, but maybe they got out of sync).
== Debugging ==
We don't have yet a way to autorefresh the certs prometheus uses to authenticate against k8s, so they need renewal.
When they expire prometheus is not able to get metrics from it (so any k8s related metric will just not be there).
You can check manually if the certificates that prometheus uses to connect to k8s have expired:
If that's the case, you can follow this guide.
== Related information ==
* Karma UI (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/Redis

This runbook explains how to debug and fix common issues with Toolforge Redis.
== Error / Incident ==
* checker.tools.wmflabs.org/toolschecker: Redis set/get is a paging alert that triggers where toolschecked is unable to talk to Redis.
== Debugging ==
SSH to Redis hosts at tools-redis-X.tools.eqiad1.wikimedia.cloud and check the output of redis-cli info replication.
The Redis service is using a non-standard unit name in Systemd: redis-instance-tcp_6379.service
== Common issues ==
=== max number of clients reached ===
When this happens, the following message will be logged: ERR max number of clients reached. Check in the logs of all servers with sudo journalctl -g "max number of clients". If you find this message appearing repeatedly, restart the Systemd unit on the hosts where the message is logged:
$ sudo systemctl restart redis-instance-tcp_6379.service
== Related information ==
Portal:Toolforge/Admin/Redis
== Support contacts ==
#wikimedia-cloud-admin is the main communication channel for Toolforge admins.
If Redis is down, you should follow the Wikimedia Cloud Services team/Incident Response Process
== Old incidents ==
* Incidents/2024-04-28_WMCS_Toolforge_Redis_refusing_connections
Portal:Toolforge/Admin/Runbooks/TektonDown

This is when the tekton-pipelines-controller pod in the tekton-pipelines namespace of tools/toolsbeta k8s cluster is down or can't be reached.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
The first most likely step is to ssh to tools/toolsbeta (depending on the project the alert is from) cloudcontrol servers (i.e toolsbeta-test-k8s-control-4.toolsbeta.eqiad1.wikimedia.cloud). From there you can:
* check that the pods are running:
* You can also check the log of the pod's deployment with kubectl logs deploy/tekton-pipelines-controller -n tekton-pipelines.
* It might also make sense to check if there has been any recent code change and re-deployment attempts. Again a good place to start is by looking at the recent commits in builds-builder repo, or the toolforge-deploy gitlab repo.
* If the pods don't exist or the deployment does not exist, you can try redeploying the jobs-api by following the instructions in the toolforge repo (it will do nothing if there's nothing to do).
=== Doing a manual curl for the stats ===
You can try doing a curl directly to the pods for the statisticts, by checking the configuration of prometheus, you'll get the cert, key and url:
Then you can curl directly the pods by name, like:
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
If tekton seems up, you can check if the certificates that prometheus uses to connect to k8s have expired:
To refresh and fix the issue follow Portal:Toolforge/Admin/Kubernetes/Certificates#Operations.
== Related information ==
* Build services original implementation docs
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
* phab:T338025 - [T338025] [tools] Prometheus k8s cert expired
Portal:Toolforge/Admin/Runbooks/TektonUpMetricUnknown

This happens when prometheus has no data from k8s on the tekton-pipelines-controller pod.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmflabs.org/tools/classic/graph?g0.range_input=1h&g0.expr=sum(up%7Bjob%3D%22tekton-pipelines-controller%22%7D)&g0.tab=1
== Common issues ==
Add new issues here when you encounter them!
=== Prometheus k8s cert expired ===
We don't have yet a way to autorefresh the certs prometheus uses to authenticate against k8s, so they need renewal.
When they expire prometheus is not able to get metrics from it (so any k8s related metric will just not be there).
If that's the case, you can follow this guide.
== Related information ==
* Karma UI for cloud vps (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity

The ToolforgeKubernetesCapacity alert fires when the Toolforge Kubernetes cluster is close to running out of a certain resource (CPU or memory). In general, this is caused either by:
* Natural increased usage (in which case the fix is to simply provision more capacity)
* Something misbehaving and taking up more resources than it should
== Debugging ==
=== Dashboard ===
This panel has an overview of the cluster capacity. The main thing to worry about is "CPU requests" staying below "CPU allocatable".
=== Locating new workloads ===
On a bastion run as your own user:
$ kubectl get cronjob -A --sort-by=.metadata.creationTimestamp
$ kubectl get deployment -A --sort-by=.metadata.creationTimestamp
== Fixing ==
=== Provisioning new nodes ===
On a cloudcumin:
$ sudo cookbook wmcs.toolforge.add_k8s_node --cluster-name tools --role worker
If you need to provision more than one node at once, it is safe to start a new cookbook run after the first one has created the VM.
== Related information ==
* Portal:Toolforge/Admin/Kubernetes
== Support contacts ==
== Old incidents ==
* The cluster ran out of capacity on 2023-12-01 when cron jobs could not execute fast enough during a cluster reboot after several tools had migrated from the Grid Engine to K8s.
Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown

This happens when one of the haproxy servers is down from that haproxy instance (one of the hosts haproxy redirects too, not to be confused with the haproxy server itself).
== Error / Incident ==
In the alert you should see the instance (the haproxy host), the backend (the web service being proxied) and the server (the server the requests are proxied to). That might get you started on where to look.
== Debugging ==
TBD
== Common issues ==
Add new issues here when you encounter them!
== Related information ==
* See HAProxy for generic information like commands/etc.
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyUnknown

This happens when prometheus has no data from haproy on the server status (server_up).
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
== Debugging ==
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmflabs.org/tools/classic/graph?g0.range_input=1h&g0.expr=sum(up%7Bjob%3D%22tekton-pipelines-controller%22%7D)&g0.tab=1
== Common issues ==
Add new issues here when you encounter them!
== Related information ==
* See HAProxy for generic information like commands/etc.
* Karma UI for cloud vps (use project=tools or project=toolsbeta for filtering)
* Tools prometheus
* Toolsbeta prometheus
* Alerts repository
* Toolforge admin docs
* See also any other *Unknown runbooks.
== Old incidents ==
Add any incident tasks here!
*
Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady

The ToolforgeKubernetesCapacity alert fires when a Toolforge Kubernetes node is marked as not ready. A paging alert also fires when at least 5 nodes are marked as not ready.
== Debugging ==
On a bastion run as your own user:
$ kubectl sudo get node
$ kubectl sudo describe node <node>
== Related information ==
* Portal:Toolforge/Admin/Kubernetes
== Support contacts ==
== Old incidents ==
Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerDiskFull

== Error / Incident ==
There's at least one toolforge worker node with less than 2.5 Gb of space left on their root partition, if they get under 2Gb they will start failing to allocate ephemeral storage for pods.
== Debugging ==
If you can, ssh to the node and check disk usage with du -s /* or similar, common suspects are under /var/log.
== Common issues ==
=== pacct log growing too much ===
One instance of logs getting out of control is the /var/log/account/pacct log, there's to rotate it.
== Related information ==
You can find a graph with the 20 workers with lower space left here:
https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview
== Old incidents ==
Add here any new tasks for incidents you might encounter.
* [jobs-api,jobs-cli] increased exit code 137 rate since 2024-12-14
Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses

== Error / Incident ==
There's at least one toolforge worker node with many processes in 'D' (uninterruptible sleep) pointing to an IO/NFS issue.
== Debugging ==
If you can, ssh to the node and run htop/top/ps to see the running processes and their state.
You can try checking the journal or dmesg logs for nfs-related entries, for example:
 root@tools-k8s-worker-nfs-56:~# journalctl --grep tools-nfs
 Mar 04 18:29:19 tools-k8s-worker-nfs-56 kernel: nfs: server tools-nfs.svc.tools.eqiad1.wikimedia.cloud not responding, still trying
 Mar 07 00:00:28 tools-k8s-worker-nfs-56 kernel: nfs: server tools-nfs.svc.tools.eqiad1.wikimedia.cloud OK
You can check if new processes are still getting stuck, by trying to ls the users or tools home directories from the affected worker (might get stuck, so we start it in the background):
 root@tools-k8s-worker-nfs-42:~# ls -l /data/project/ &
And you can check which processes are getting stuck by searching for processes in 'D' state:
 root@tools-k8s-worker-nfs-42:~# ps aux | grep D # will show some other stuff, but you get the gist
== Common issues ==
=== NFS server went away ===
If there was an NFS hiccup, you'll see entries like:
 root@tools-k8s-worker-nfs-56:~# journalctl --grep tools-nfs
 Mar 04 18:29:19 tools-k8s-worker-nfs-56 kernel: nfs: server tools-nfs.svc.tools.eqiad1.wikimedia.cloud not responding, still trying
 Mar 07 00:00:28 tools-k8s-worker-nfs-56 kernel: nfs: server tools-nfs.svc.tools.eqiad1.wikimedia.cloud OK
If you are unable to login or the server is having a lot of trouble, you can just restart that node with the cookbook, for example:
 dcaro@urcuchillay$ cookbook wmcs.toolforge.k8s.reboot --cluster-name tools --hostname-list tools-k8s-worker-nfs-21
=== NFS stuck processes ===
If new processes work ok (ex.ls -l of the projects dir works well), you can try restarting only the pods that are stuck, the new ones should come up without problems.
== Related information ==
You can find a graph of the number of 'D' processes per toolforge VM here:
https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview
== Old incidents ==
Add here any new tasks for incidents you might encounter.
* https://phabricator.wikimedia.org/T362690 - new processes also were getting stuck in D state, needed host reboot (if it happens again, please retake the debugging)
Portal:Toolforge/Admin/Runbooks/ToolsNFSDown

The ToolsNFSDown alert fires when the nfs-service service is not running or not being found in the stats.
== Error / Incident ==
If the value is 0, then the service is down, if the value is -1 then prometheus is not gathering the stats correctly (the service might be down, we don't know). 
As of 2024-07-30 the nfs server is tools-nfs-2.tools.eqiad1.wikimedia.cloud
== Debugging ==
=== Check the service status ===
Ssh to the server and check the service status:
dcaro@tools-nfs-2:~$ sudo systemctl status nfs-server.service 
● nfs-server.service - NFS server and services
 Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
 Active: active (exited) since Mon 2024-06-24 14:25:57 UTC; 1 months 5 days ago
 Main PID: 721 (code=exited, status=0/SUCCESS)
 Tasks: 0 (limit: 77152)
 Memory: 0B
 CPU: 0
 CGroup: /system.slice/nfs-server.service
Jun 24 14:25:55 tools-nfs-2 systemd[1]: Starting NFS server and services...
Jun 24 14:25:57 tools-nfs-2 systemd[1]: Finished NFS server and services.
=== If there's no stats ===
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
https://tools-prometheus.wmflabs.org/tools/classic/graph?g0.range_input=1h&g0.expr=sum(up%7Bjob%3D%22tekton-pipelines-controller%22%7D)&g0.tab=1
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Data_Services/Admin/Runbooks/Enable_NFS_for_a_project
* Portal:Data_Services/Admin/Shared_storage
== Old incidents ==
Add here any new tasks for incidents you might encounter.