nyuuzyou/wikis huggingface dataset:
Portal:Cloud VPS/Admin/Runbooks/CephClusterAllMonsDown

== Error / Incident ==
All the ceph monitor nodes are down in the system.
Ceph monitor nodes are the ones acting as entry points for the system, that means that the service is not working and the whole ceph cluster is unusable.
When they go down all the VMs stop having disk access.
== Debugging ==
This might be the result of an ongoing intervention gone wrong, so first try to communicate with the rest of the team and look for any ongoing actions:
* Ask on irc (#wikimedia-cloud-admin) if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
Then you can continue trying to figure out why the hosts are going down and not coming back up:
* Try to ssh to the host
* Try to ssh to the management port (see SRE/Dc-operations/Hardware_Troubleshooting_Runbook)
If you got ssh access, you can continue by checking the logs for the monitor service:
* root@cloudcephmon1001:~# systemctl status systemctl status ceph-mon@cloudcephmon1001.service
If the ceph cli is working, you can also query the cluster status:
* ceph status
* ceph health detail
See also Portal:Cloud_VPS/Admin/Runbooks/CephClusterInError in case that the problem is at the application level.
== Common issues ==
Add here any issues you encounter.
== Related information ==
* SAL (or sal.toolforge.org).
* Netbox
* Puppet merged changes
* SRE/Dc-operations/Hardware_Troubleshooting_Runbook
* Portal:Cloud_VPS/Infrastructure
* Portal:Cloud_VPS/Admin/Troubleshooting
* Portal:Cloud_VPS/Admin
* Portal:Cloud_VPS/Admin/Network#cloudgw
== Support contacts ==
== Old incidents ==
Provide a list of phabricator tasks of previous incidents to be able to more review previous instances of the issue.
Portal:Cloud VPS/Admin/Runbooks/CephClusterInUnknown

We are not getting metrics from the ceph cluster, it might or might not be running.
== Debugging ==
Check the health details of the cluster:
 root@cloudcephmon1001:~# ceph health detail
If the cluster is unhealthy yyou can see more debugging guidelines see Portal:Cloud VPS/Admin/Runbooks/CephClusterInError for other debugging and details.
Next we can check if the prometheus node can reach the ceph mgr daemon, to do so, you can login to the prometheus instance:
 dcaro@urcuchillay$ wm-ssh prometheus1005.eqiad.wmnet
 dcaro@prometheus1005:~# sudo -i
 root@prometheus1005:~# curl -v http://cloudcephmon1001:9283/metrics
If you don't get anything then it might be a network issue, or the mgr daemon might be misbehaving, you can try restarting it.
If it works, you can continue digging on the prometheus side, to find the instance that we are running you can:
 root@prometheus1005:~# ps aux | grep prometheus | grep cloud
 prometh+ 1087 3.4 0.1 30494908 678072 ? Ssl Jun06 3462:03 /usr/bin/prometheus --storage.tsdb.path /srv/prometheus/cloud/metrics --web.listen-address 127.0.0.1:9904 --web.external-url https://prometheus-eqiad.wikimedia.org/cloud --storage.tsdb.retention 4032h --config.file /srv/prometheus/cloud/prometheus.yml --storage.tsdb.max-block-duration=24h --storage.tsdb.min-block-duration=2h --query.max-samples=10000000
 prometh+ 1115 5.4 0.0 923936 73660 ? Ssl Jun06 5502:21 /usr/bin/thanos sidecar --http-address 0.0.0.0:19904 --grpc-address 0.0.0.0:29904 --tsdb.path /srv/prometheus/cloud/metrics --prometheus.url http://localhost:9904/cloud --objstore.config-file /etc/thanos-sidecar@cloud/objstore.yaml --min-time=-15d --shipper.ignore-unequal-block-size
And go from there (logs, config, etc.).
== Support contacts ==
Usually anyone in the WMCS team should be able to help/debug the issue, subject matter experts (SMEs) would be Andrew Bogott and David Caro.
== Related information ==
* Grafana dashboard: https://grafana.wikimedia.org/d/7TjJENEWz/wmcs-ceph-eqiad-cluster-overview?orgId=1
* Internal documentation: Portal:Cloud_VPS/Admin/Ceph
* Upstream documentation: https://docs.ceph.com/docs/master/rados/operations/monitoring/
== Example tasks ==
* https://phabricator.wikimedia.org/T372528 - [ceph] Metrics started not responding during the drain
Category:Runbooks
Portal:Cloud VPS/Admin/Runbooks/CephClusterInWarning

The ceph cluster is in warning status, this means that it's not highly available anymore or something might be affecting it's performance, but the cluster is still up and running.
== Debugging ==
Check the health details of the cluster:
 root@cloudcephmon1001:~# ceph health detail
For more debugging guidelines see Portal:Cloud VPS/Admin/Runbooks/CephClusterInError for other debugging and details.
=== Mons using a lot of disk space ===
We saw this once, the health details show:
 root@cloudcephmon1001:~# ceph health detail
 HEALTH_WARN mons cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 are using a lot of disk space
 [WRN] MON_DISK_BIG: mons cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 are using a lot of disk space
 mon.cloudcephmon1003 is 15 GiB >= mon_data_size_warn (15 GiB)
 mon.cloudcephmon1002 is 15 GiB >= mon_data_size_warn (15 GiB)
 mon.cloudcephmon1001 is 15 GiB >= mon_data_size_warn (15 GiB)
This usually means that the cluster is waiting for some backfilling/repair to finish but it's taking too long for whichever reason and the trimming actions on the mons datastores are not running (a regular working size right now is ~200MB).
In the last instance, that was not the case, but we did not find the root cause.
To fix you can force the mons to trigger a trim by restarting all of them, one by one to keep the cluster alive:
Note: you can use the cookbook wmcs.ceph.roll_reboot_mon_daemons once this is merged.
 root@cloudcephmon1001:~# systemctl restart ceph-mon@cloudcephmon1001
 ...
 root@cloudcephmon1001:~# ceph status # check that the mon is back online and in the cluster
 # repeat on the others
 root@cloudcephmon1002:~# systemctl restart ceph-mon@cloudcephmon1002
=== Slow ops ===
See Portal:Cloud VPS/Admin/Runbooks/CephSlowOps.
== Support contacts ==
Usually anyone in the WMCS team should be able to help/debug the issue, subject matter experts (SMEs) would be Andrew Bogott and David Caro.
== Related information ==
* Grafana dashboard: https://grafana.wikimedia.org/d/7TjJENEWz/wmcs-ceph-eqiad-cluster-overview?orgId=1
* Internal documentation: Portal:Cloud_VPS/Admin/Ceph
* Upstream documentation: https://docs.ceph.com/docs/master/rados/operations/monitoring/
== Example tasks ==
* https://phabricator.wikimedia.org/T286649 - OSD daemon crash
Category:Runbooks
Portal:Cloud VPS/Admin/Runbooks/CephSlowOps

Ceph slow ops will usually block some writes to the Ceph cluster which makes VMs unhappy.
== Debugging ==
Log in to any cloudcephmon node, and then use sudo ceph health detail to find OSDs having issues. Use sudo ceph osd find osd.NNN to find which cloudcephosd node is having issues.
On the cephosd node, use sudo ceph daemon osd.NNN ops_in_flight detail about the stuck operations.
== Common issues ==
=== Disk issues ===
== Related information ==
* Portal:Cloud VPS/Admin/Ceph
* [cloudceph] Slow operations - tracking task
== Support contacts ==
== Old incidents ==
Portal:Cloud VPS/Admin/Runbooks/Check unit status of maintain dbusers

== Error / Incident ==
The maintain-dbusers service is failing.
== Debugging ==
To gather logs, just ssh to the host having the issue and check the backups_vm.service unit status:
 root@cloudvirt1021:~# systemctl status backup_vms
 ● backup_vms.service - Backup vms assigned to this host
 Loaded: loaded (/lib/systemd/system/backup_vms.service; static)
 Active: failed (Result: exit-code) since Mon 2022-03-21 20:01:53 UTC; 17h ago
 TriggeredBy: ● backup_vms.timer
 Process: 4099425 ExecStart=/usr/local/sbin/wmcs-backup instances backup-assigned-vms (code=exited, status=1/FAILURE)
 Main PID: 4099425 (code=exited, status=1/FAILURE)
 
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: backup_entries = get_backups()
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: File "/usr/lib/python3/dist-packages/rbd2backy2.py", line 600, in get_backups
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: backup = subprocess.check_output([BACKY, "-ms", "ls"])
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: File "/usr/lib/python3.9/subprocess.py", line 528, in run
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: raise CalledProcessError(retcode, process.args,
 Mar 21 20:01:52 cloudvirt1021 wmcs-backup[4099425]: subprocess.CalledProcessError: Command '['/usr/bin/backy2', '-ms', 'ls']' returned non-zero exit status 100.
 Mar 21 20:01:53 cloudvirt1021 systemd[1]: backup_vms.service: Main process exited, code=exited, status=1/FAILURE
 Mar 21 20:01:53 cloudvirt1021 systemd[1]: backup_vms.service: Failed with result 'exit-code'.
You can also check the logs for a longer view:
 root@cloudvirt1021:~# journalctl -u backup_vms
== Common issues ==
=== Connectivity issues ===
The maintain-dbusers daemon connects to each of the NFS servers uwsgi-toolsdb-replica-cnf-web APIs to create the files in the NFS filesystem, to the toolsdb database, to each of the wikireplicas and to the accounts database.
To see the full list of databases and credentials to debug check the file /etc/dbusers.yaml
== Related information ==
* Portal:Toolforge/Admin#Regenerate_replica.my.cnf
* Portal:Data_Services/Admin/Shared_storage#maintain-dbusers
* grafana board
== Old incidents ==
* cloudvirt1021 - CRITICAL: Status of the systemd unit backup_vms
* cloudvirt1022/Check unit status of backup_vms is CRITICAL
Portal:Cloud VPS/Admin/Runbooks/GaleraClusterOutOfSync

The GaleraClusterSizeMismatch alert fires when the Galera wsrep_cluster_state_uuid status variable has multiple visible values.
== Error / Incident ==
The datastore used by all OpenStack services might be in a split-brain situation.
== Debugging ==
Find the node with a different wsrep_cluster_state_uuid, and do something about it.
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Cloud VPS/Admin/Galera
* Portal:Cloud VPS/Admin/Troubleshooting#Galera
== Support contacts ==
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/GaleraClusterSizeMismatch

The GaleraClusterSizeMismatch alert fires when the Galera cluster does not have exactly three members.
== Error / Incident ==
The datastore used by all OpenStack services might be having clustering issues.
== Debugging ==
In general, this alert is caused by either:
* Problems on a single node, which would also trigger other more specific alerts, or
* Maintenance (OS upgrades, node replacements) which hopefully have some relevant communications in -cloud-admin
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Cloud VPS/Admin/Galera
* Portal:Cloud VPS/Admin/Troubleshooting#Galera
== Support contacts ==
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/GaleraDown

The GaleraDown alert fires when mysqld exporter can't connect to the MariaDB instance on a cloudcontrol node.
== Error / Incident ==
The datastore used by all OpenStack services is unreachable.
== Debugging ==
Log in to the host in question, then:
# Check if you can log in to the MariaDB host manually:
user@cloudcontrolXXXX ~ $ sudo -i mariadb -u root
# Check the mysqld-exporter logs:
user@cloudcontrolXXXX ~ $ sudo systemctl status prometheus-mysqld-exporter.service
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Cloud VPS/Admin/Galera
* Portal:Cloud VPS/Admin/Troubleshooting#Galera
== Support contacts ==
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/GaleraNoRecentWrites

The GaleraNoRecentWrites alert fires when the Galera wsrep_last_committed status variable has not increased in the last hour.
== Error / Incident ==
Galera has applied no writes in the past hour.
== Debugging ==
Some tips:
* Look at the value of the variable on all nodes
== Common issues ==
=== Single node behind others, all nodes stuck ===
Once a single Galera node had a lower value than all others. In that case we depooled the node which moved traffic to other nodes, those two nodes then started having new writes applied.
== Related information ==
* Portal:Cloud VPS/Admin/Galera
* Portal:Cloud VPS/Admin/Troubleshooting#Galera
== Support contacts ==
== Old incidents ==
* T352539 2023-12-01 Cloud VPS network outage
Portal:Cloud VPS/Admin/Runbooks/GaleraNodeOutOfSync

The GaleraNodeOutOfSync alert fires when the Galera wsrep_cluster_state_uuid status variable does not match wsrep_local_state_uuid on a single node.
== Error / Incident ==
A single node in the Galera cluster used by all OpenStack services does not seem to be in sync with the rest of the cluster.
== Debugging ==
Look at the logs and maybe try restarting it, I guess?
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Cloud VPS/Admin/Galera
* Portal:Cloud VPS/Admin/Troubleshooting#Galera
== Support contacts ==
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/GaleraNotEnabled

The GaleraNotEnabled alert fires when the wsrep_ready status variable indicates that the MariaDB process on a host is not running in Galera mode.
== Error / Incident ==
The datastore used by all OpenStack services is not clustering properly.
== Debugging ==
Log in to the host in question, then:
# Check Galera related status variables:
user@cloudcontrolXXXX ~ $ sudo -i mariadb -u root
mysql:root@localhost [(none)]> SHOW STATUS LIKE "wsrep_%";
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Cloud VPS/Admin/Galera
* Portal:Cloud VPS/Admin/Troubleshooting#Galera
== Support contacts ==
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/Kernel panic

== Error / Incident ==
The server had a number of panic / error messages showing up in the dmesg kernel print buffer.
Depending on the error, you may want to take the server out of service immediately (depool).
== Debugging ==
This is quite a generic alert, so there's many options. In general, a kernel panic may indicate any or all of:
* a linux kernel bug
* a hardware problem
* a firmware problem
* other severe misconfiguration
Try to ssh to the host and check the status, for example:
journalctl -k --since today
Error-level logs:
journalctl -k -p err
Check all other logs:
journalctl --since today
You may also want to check idrac logs. TODO: add link.
== Common issues ==
=== x86/cpu: SGX disabled by BIOS ===
This can be ignored, it's a CPU extension that is disabled in the BIOS and we don't want to use. More details in phab:T379351.
=== x86/cpu: VMX (outside TXT) disabled by BIOS ===
Unless you want to run VMs on that machine, this message is ok.
If you want to run VMs on that machine, you'll have to enable that flag in the BIOS.
=== mpt3sas_cm0: Trace buffer memory 2048 KB allocated ===
This one is fixed in newer kernels (demoted to info):
https://lkml.org/lkml/2023/1/18/1533
=== watchdog: watchdog0: watchdog did not stop! ===
This is sometimes logged when manually rebooting a server. It can probably be ignored.
=== Add any more here when you find new issues ===
== Acknowledging known issues ==
Currently the only way is to acknowledge the alert for 24h (adding a comment with !ACK mycomment or using the tick in the ui).
Note that the alert will reappear for any message that is shown on boot.
== Related information ==
See https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation
== Support contacts ==
== Old incidents ==
Add your incident here:
Portal:Cloud VPS/Admin/Runbooks/MetricsinfraAlertmanagerDown

The MetricsinfraAlertmanagerDown alert fires when the production Prometheus AlertManager dashboard can't communicate with the alerting setup in the metricsinfra project. This is either an issue with the metricsinfrar setup, or a sign of a more general network outage in Cloud VPS.
== Debugging ==
== Common issues ==
Add here any new common issues you find.
== Related information ==
* Portal:Cloud VPS/Admin/Troubleshooting
* Nova Resource:Metricsinfra
== Support contacts ==
Taavi knows both about the network setup and the metricsinfra alerting setup.
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/NodeDownForLong

== Overview ==
== Error / Incident ==
A node has been offline (considered down) by a long period of time.
== Debugging ==
This is quite a generic alert, so there's many options, but some things to try:
* Try to ssh to the host
* Try to ssh to the management port (see SRE/Dc-operations/Hardware_Troubleshooting_Runbook)
* Ask on irc if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
== Common issues ==
=== Cloudvirt down ===
If the host being unresponsive is a cloudvirt host, it will trigger a page. You should add it to the "maintenance" aggregate to prevent new VMs being scheduled on that host, and drain the host evacuating all VMs to other hosts. 
Example:
root@cloudcontrol1005:~# openstack aggregate add host maintenance cloudvirt1063
# take note of the status of all VMs (ACTIVE, SHUTOFF, PAUSED), as you'll have to manually restore it after the VM is moved to another host
root@cloudcontrol1005:~# openstack server list --host cloudvirt1063 --all-projects
# the value for os-password can be found in /etc/novaadmin.yaml
root@cloudcontrol1005:~# nova --os-username novaadmin --os-project-name admin --os-auth-url "https://openstack.eqiad1.wikimediacloud.org:25357/v3" --os-password XXXXXX --os-user-domain-id default host-evacuate cloudvirt1063
nova CLI is deprecated and will be removed in a future release
+--------------------------------------+-------------------+---------------+
| Server UUID | Evacuate Accepted | Error Message |
+--------------------------------------+-------------------+---------------+
| 3585384e-e85f-41da-a014-448c59d76586 | True | |
| abaa359b-3468-4b56-be7c-24dc88252c9e | True | |
| 5899163e-bbfb-474e-8a9a-2dae5006de22 | True | |
| f9015696-9010-4a1c-a7cf-9bc8e8098df4 | True | |
| 15700b39-5773-4fec-9349-47f16bc457bd | True | |
| b615abe0-9cc2-4cee-a224-c1f0b90e7957 | True | |
| 20f22f9d-5734-49b9-8a19-9908e588c92f | True | |
| 1b0a01e4-e8cb-4a68-ad6c-306b9869db05 | True | |
| 3192a8d0-0fda-4d24-a1b7-2e2dc20fbe0e | True | |
+--------------------------------------+-------------------+---------------+
# restart VMs that were in ACTIVE status with:
root@cloudcontrol1005:~# openstack server start <id>
== Related information ==
* SAL (or sal.toolforge.org).
* Netbox
* Puppet merged changes
* SRE/Dc-operations/Hardware_Troubleshooting_Runbook
== Support contacts ==
== Old incidents ==
Add your incident here:
Portal:Cloud VPS/Admin/Runbooks/NovaFullstackStaleStats

== Error / Incident ==
Some of the stats used for the novafullstack alerts were not reported for some time.
== Debugging ==
Check that the prometheus series are there, search for the alert name in the alerts.git repo, and look for the expr line, would be something like:
 expr: count(cloudvps_novafullstack_instances_count) == 0 or count(cloudvps_novafullstack_instances_max) == 0
From there each of the stats inside a count function might be the one failing (or all of them!), so you can go to thanos or grafana and query them there.
This might mean:
* That the novafullstack service is misbehaving
* That the stats names did change
* That the service is down
* That the prometheus stats are not being generated (usually under /var/lib/prometheus/node.d/nofafullstack.prom in the cloudcontrol that is running the novafullstack service).
== Common issues ==
Add any new issues you find here.
== Related information ==
== Support contacts ==
== Old incidents ==
Add any tasks for incidents related to this alert here.
Portal:Cloud VPS/Admin/Runbooks/NovafullstackSustainedFailures

== Error / Incident ==
The nova-fullstack tests have been failing for some time
This happens when there's an error in the creation and/or deletion of the nova-fullstack VM.
To see the logs, you have to login to the instance that triggered the alert (if you are looking at the https://alerts.wikimedia.org, it is shown as a blue tag). For example cloudcontrol1003:
== Debugging ==
For debugging instructions follow Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedLeakedVMsEqiad, it's quite related and the debugging is the same.
== Support contacts ==
Usually anyone in the WMCS team should be able to help/debug the issue, the main subject matter expert would be Andrew Bogott.
== Related information ==
* https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Nova-fullstack
* https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring#nova-fullstack
== References ==
Category:Runbooks
Portal:Cloud VPS/Admin/Runbooks/OpenstackAPIResponse

== Error / Incident ==
This alert fires when the Openstack API performance is detected to be "poor". By the time of this writing, this value has been established as the 12h average response time to be > 1 second.
The performance problem is likely affecting the Openstack control plane and making it unrealiable, slow, and/or prone to transient failures.
== Debugging ==
This alert is based on the prometheus metrics of HAproxy. You can check this grafana dashboard:
https://grafana-rw.wikimedia.org/d/UUmLqqX4k/openstack-api-performance
As of this writing, there should be one primary HAproxy node, that you can check with:
user@laptop:~ $ host openstack.eqiad1.wikimediacloud.org
openstack.eqiad1.wikimediacloud.org is an alias for cloudcontrol1007.wikimedia.org.
[..]
You can try by hand some openstack commands and see how they behave:
user@cloudrabbit1007:~$ sudo wmcs-openstack endpoint list
[..]
user@cloudrabbit1007:~$ sudo wmcs-openstack server list --all-projects
[..]
user@cloudrabbit1007:~$ sudo wmcs-openstack zone list --all-projects
[..]
user@cloudrabbit1007:~$ sudo wmcs-openstack subnet list --all-projects
[..]
You can check basic server stats, like CPU, memory, load average, etc. Try using tools like htop and similar.
== Operations ==
Some common operations to handle this alert.
=== Services restart ===
In the past, restarting all the openstack control plane has solved the performance problem for the most part.
There is an automated cookbook to do it:
user@laptop:~ $ cookbook wmcs.openstack.restart_openstack --task-id T11111 --cluster-name eqiad1 --all
[..]
=== Increase number of worker ===
In the past, increasing the number of API workers has helped accommodate additional load.
Example ticket: T336379 - Openstack API slowdowns.
== Old incidents ==
* phab:T345084 (open task to track this recurring issue)
* In the past, we know prometheus scrapping of the openstack metrics have causes performance to suffer.
See for example phabricator ticket: T335943