nyuuzyou/wikis huggingface dataset:
Portal:Cloud VPS/Admin/Runbooks/OpenstackAPIs

Note: some of these steps might require extra access to internal infrastructure systems, we are working on improving the runbooks, until then, take this as a guideline.
== Error ==
This page is specifically for managing load, DDOS, or other malicious use of the OpenStack APIs.
There is not currently special-purpose monitoring for API responsiveness, so issues with load will probably show up as a failure of the nova-fullstack service or errors on Horizon.
== Debugging ==
All APIs are accessed via a single service address, openstack.eqiad1.wikimediacloud.org. This is mapped to a single cloudcontrol node running HA proxy. The first thing to check is whether an API outage is due to a service actually being down; this can be seen by checking the HAProxy logs:
 $ sudo tail /var/log/haproxy/haproxy.log
This will show messages about any api services being up or down. Services which show as down can be safely restarted.
If all services are up but overwhelmed by traffic, you can see traffic to a particular frontend by altering that frontend's config. For example, for nova-api, the haproxy config is /etc/haproxy/conf.d/nova_api.cfg . Enable per-request logging by uncommenting this line:
 # log /dev/log local0 debug
and then running
 # sudo systemctl restart haproxy.service
After that, you should start seeing one line per connection in haproxy.log
Note: haproxy logrotate is a bit broken. Sometimes the active logfile is haproxy.log and sometimes it's haproxy.log.1. If haproxy is empty or seems unusually quiet, check for the active log with 'ls -ltrah /var/log/haproxy/'
== Solutions ==
Haproxy supports a variety of throttling and blocking options. There are ready-made block lists for either IP address or user agent; these blocks will apply for all haproxy-backed services uniformly. Throttling and/or more nuanced blocking is left as an exercise for the reader, and best saved for after any immediate downtime is resolved.
=== Solution: block a user agent ===
The /etc/haproxy/agentblocklist.txt file contains a simple list of user agent strings. User agents matching any string in that file are blocked.
agentblocklist.txt is managed by puppet and comes from modules/openstack/files/haproxy/agentblocklist.txt. Lines preceded by # will be ignored.
=== Solution: block an IP ===
The /etc/haproxy/ipblocklist.txt file contains a simple list of ip addresses; any http request from an IP address listed there will return 403.
ipblocklist.txt is managed by puppet and comes from modules/openstack/files/haproxy/ipblocklist.txt. Don't forget to add a comment explaining why a given ip is blocked! Lines preceded by # will be ignored.
== Related tasks ==
* Open Openstack APIs to the public internet
Category:Runbooks
Portal:Cloud VPS/Admin/Runbooks/PuppetCertificateAboutToExpire

This happens when a Puppet certificate is about to expire. It might be either a normal host certificate or a CA certificate.
== Error / Incident ==
This usually comes in the form of an alert in alertmanager.
The alert will include the CN for the certificate that's expiring.
== Debugging ==
The alert will tell you also how much is left for the certificate to expire in the summary, for example:
 summary: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 3h 0m 43s
== Common issues ==
=== Host certificate about to expire ===
If a host certificate is about to expire, run the wmcs.vps.refresh_puppet_certs cookbook to renew it. (This usually means that the host is quite old, so consider replacing it with one that uses a newer OS version!)
If the cookbook fails with "The certificate retrieved from the master does not match the agent's private key", try cleaning up the cert manually from the puppet server:
# puppetserver ca clean --certname {FQDN}
Then from the puppet client delete the old cert and re-run Puppet to generate a new one:
# rm /var/lib/puppet/ssl/{FQDN}.pem
# run-puppet-agent
=== CA certificate about to expire ===
Sometimes it's the Puppet CA certificate itself that's about to expire. This is happening if the certificate CN starts with Puppet CA:.
To renew that certificate, follow Help:Standalone_puppetmaster#Renewing_puppetserver_CA_certificate.
== Related information ==
* Puppet standalone setup and maintenance docs
== Old incidents ==
* T354714 - [clouddb-service-puppetmaster-2] Renew puppet CA certificates
Portal:Cloud VPS/Admin/Runbooks/PuppetStaleCertificates

== Error / Incident ==
There's some stale certificates on the puppetmaster after the removal of some VMs.
== Common issues ==
This usually happens after manually removing a VM in a project that has it's own puppetmaster.
If the puppetserver is using the openstack_stale_puppet_certs prometheus exporter (probably yes), then you can cleanup all the obsolete certs with clean-stale-puppet-certs:
tools-puppetserver-01:~$ sudo /usr/local/sbin/clean-stale-puppet-certs --clean
Or, if you're feeling verbose:
root@tools-puppetserver-01:~# for host in $(grep -o 'cert_name="[^"]*' /var/lib/prometheus/node.d/openstack_stale_puppet_certs.prom | cut -d'"' -f2); do ping -c1 -w1 "$host" && { echo "SKIPPING: $host is alive"; continue; }; puppetserver ca clean --certname "$host"; done
Another way to try to find what to cleanup, is checking the fqdns that were removed in the following graph (search for relevant project in the stale certificates graph):
https://grafana-rw.wmcloud.org/d/SQM7MJZSz
then with the list of failed fnqdns, follow the guideline here:
https://wikitech.wikimedia.org/wiki/Puppet#node_cleanup
To avoid, you can use the dedicated cookbook to remove the instances:
$ sudo cookbook wmcs.vps.remove_instance --help
usage: cookbook [GLOBAL_ARGS] wmcs.vps.remove_instance [-h]
 [--project PROJECT]
 [--task-id TASK_ID]
 [--no-dologmsg]
 [--cluster-name {eqiad1,codfw1dev}]
 --server-name
 SERVER_NAME
WMCS Toolforge - Remove an instance from a project.
Usage example:
 cookbook wmcs.vps.remove_instance \
 --project toolsbeta \
 --server-name toolsbeta-k8s-test-etcd-08
optional arguments:
 -h, --help show this help message and exit
 --project PROJECT Relevant Cloud VPS openstack project (for operations,
 dologmsg, etc). If this cookbook is for hardware, this
 only affects dologmsg calls. Default is 'admin'.
 --task-id TASK_ID Id of the task related to this operation (ex.
 T123456). (default: None)
 --no-dologmsg To disable dologmsg calls (no SAL messages on IRC).
 (default: False)
 --cluster-name {eqiad1,codfw1dev}
 Openstack cluster_name where the VM is hosted.
 (default: eqiad1)
 --server-name SERVER_NAME
 Name of the server to remove (without domain, ex.
 toolsbeta-test-k8s-etcd-9). (default: None)
== Related information ==
* Puppet: SRE puppet docs.
* Help:Standalone_puppetmaster: Cloud VPS standalone puppetmaster docs
== Support contacts ==
== Old incidents ==
Add your incident here:
Portal:Cloud VPS/Admin/Runbooks/Refresh sssd caches

How to clear sssd caches (hosts, groups, etc).
=== Overview ===
If DNS or LDAP change needs a quick update on a Cloud VPS VM that runs sssd, sometimes waiting for the cache to expire is not something you want to do for whatever reason. This applies if the host is not running the nscd or a similar OTHER cache for this kind of information. If you are running nscd, you'll have to clear that instead and/or as well (eg. nscd -i hosts). Anyone operating a Cloud VPS instance with sudo permissions can accomplish this.
=== Fixing ===
# Make sure the Debian package sssd-tools is installed. If it is not installed on this VM, run sudo apt-get install sssd-tools 
# Run sudo -i sss_cache -E to invalidate all caches. 
The programmatic cache purge does not always work -- in extreme cases you might also need to actively delete the cache files:
$ sudo systemctl stop sssd
$ sudo rm -rf /var/lib/sss/db/*
$ sudo systemctl restart sssd
Portal:Cloud VPS/Admin/Runbooks/Runbook template

This is a template for Cloud VPS runbooks. Replace this paragraph with a general overview of what this runbook covers.
If your runbook requires admin permissions to complete, include this header:
Note: if your runbook is intended for general users, and also has some steps that require admin permissions, you should mark those steps with the following template, so folks can easily identify what they are able to do: 
== Error / Incident ==
What?
Provide a description of the error or incident.
== Debugging ==
How?
Provide a list of steps to gather information to be able to pin-point the issue
== Common issues ==
Add here any new common issues you find.
== Related information ==
Where?
Provide links to related tools, tickets of previous incidents, and documentation that may provide context or additional information, this might include links to the affected service main page (if any), diagrams, external links...
== Support contacts ==
Who?
You may choose to list support contacts, individuals who have knowledge about this topc and who may be able to confirm the accuracy of the runbook or who may can help resolve issues related to the error or incident.
Note: Including the template {{:Help:Cloud Services communication}} may be sufficient.
== Old incidents ==
Add here any new tasks for incidents you might encounter.
== Examples of runbooks ==
* https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_VPS_alert_Puppet_failure_on
* https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_for_VMs_leaked_by_the_nova-fullstack_test
Portal:Cloud VPS/Admin/Runbooks/SystemdUnitDownForLong

== Error / Incident ==
A systemd unit has been down on a host for a long time.
== Debugging ==
This is quite a generic alert, so there's many options, but some things to try:
* Try to ssh to the host and check the status of the unit:
systemctl status <unit_name>
* Check the logs for that unit:
journalctl -u <unit_name> -n 1000
From there you might have to go to the specific service details on how to debug them.
* Ask on irc if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
== Common issues ==
Add notes here when you find this issue.
== Related information ==
* SAL (or sal.toolforge.org).
* Puppet merged changes
== Support contacts ==
== Old incidents ==
Add your incident here:
Portal:Cloud VPS/Admin/Runbooks/TfInfraTestApplyFailed

== Error / Incident ==
This alert means that our tofu tests failed to apply, this is, create all the resources.
We run the tofu tests from the machine tf-bastion.tofuinfratest.eqiad1.wikimedia.cloud, that regularly try to apply (create) and then destroy a collection of resources on cloud vps.
The tests are triggered daily by a cronjob for the root user:
root@tf-bastion:~# crontab -l
[...]
0 12 * * * systemd-cat -t tf-infra-test /root/tf-infra-test/tofu-test.sh eqiad1
== Debugging ==
ssh to the tf-bastion.tofuinfratest.eqiad1.wikimedia.cloud and try running the creation manually, that will give you more logs.
NOTE: if you are not member of the project, you can try sshing as root, or using the console cookbook (wmcs.openstack.cloudvirt.vm_console).
If you know which resource failed, you can try also applying that specific resource like this:
== Common issues ==
Add here any new common issues you find.
=== Quota errors when creating the trove databases ===
There's several quotas attached to trove databases:
* Project database quotas
* Trove's regular quotas
It might be that you are hitting your own project's database quotas or torve is hitting it's own.
==== Quota usage and requests out of sync ====
It happened once that the quota that trove registers as 'requested' and the one it registers as 'used' get out of sync, it seems that the requests get lost somehow and it does not decrease the 'requested' counter.
You'll see that as a regular quota error, to debug and fix see Portal:Cloud_VPS/Admin/Trove#Reserved_quota_does_not_go_down.
=== Rabbitmq issues ===
Sometimes restarting Rabbitmq fixes an issue where tofuinfratest fails to create Trove instances:
cloudcumin1001:~$ sudo cumin 'cloudrabbit*' 'systemctl restart rabbitmq-server'
== Related information ==
* tf-infra-test repo
== Old incidents ==
Add here any new tasks for incidents you might encounter.
*
Portal:Cloud VPS/Admin/Runbooks/TfInfraTestDestroyFailed

== Error / Incident ==
This alert means that our tofu tests failed to destroy all the resources after creating them.
We run the tofu tests from the machine tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud, that regularly try to apply (create) and then destroy a collection of resources on cloud vps.
== Debugging ==
Ssh to the tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud and try running the creation manually, that will give you more logs.
NOTE: ssh might not work (see phab:T341764, in that case try using the console cookbook (wmcs.openstack.cloudvirt.vm_console).
== Common issues ==
Add here any new common issues you find.
== Related information ==
* tf-infra-test repo
== Old incidents ==
Add here any new tasks for incidents you might encounter.
Portal:Cloud VPS/Admin/Runbooks/TooManyCloudcontrolsDown

== Error / Incident ==
All the cloudcontrol nodes are down in the system.
Cloudcontrols are the nodes that host the control plane for Openstack, so this means that no new actions can be scheduled (starting VMs, creating disks, logging in, ...). This means that even though the VMs might still be running, the system is not usable.
Cloudcontrols host a lot of services, for example:
* Galera + mariadb for all openstack services
* Nova-api, nova-scheduler, nova-conductor (VM instance management)
* RabbitMQ for all openstack services
* HAProxy for all openstack services
* Trove-api, trove-conductor (DB as a Service)
* Keystone (authentication/authorization/accounting)
* Glance (VM image management service)
* Neutron API/rpc (software defined network service)
* Cinder (block device management service)
* ...
== Debugging ==
This might be the result of an ongoing intervention gone wrong, so first try to communicate with the rest of the team and look for any ongoing actions:
* Ask on irc (#wikimedia-cloud-admin) if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
Then you can continue trying to figure out why the hosts are going down and not coming back up:
* Try to ssh to the host
* Try to ssh to the management port (see SRE/Dc-operations/Hardware_Troubleshooting_Runbook)
== Common issues ==
Add here any issues you encounter.
== Related information ==
* SAL (or sal.toolforge.org).
* Netbox
* Puppet merged changes
* SRE/Dc-operations/Hardware_Troubleshooting_Runbook
* Portal:Cloud_VPS/Infrastructure
* Portal:Cloud_VPS/Admin/Troubleshooting
* Portal:Cloud_VPS/Admin
== Support contacts ==
== Old incidents ==
Provide a list of phabricator tasks of previous incidents to be able to more review previous instances of the issue.
Portal:Cloud VPS/Admin/Runbooks/TooManyCloudgwsDown

== Error / Incident ==
All the cloudgw nodes are down in the system.
Cloudgw nodes are the ones acting as routers to (and from) the external world for VMs.
When they go down all the VMs stop having network access to external networks and from external networks.
== Debugging ==
This might be the result of an ongoing intervention gone wrong, so first try to communicate with the rest of the team and look for any ongoing actions:
* Ask on irc (#wikimedia-cloud-admin) if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
Then you can continue trying to figure out why the hosts are going down and not coming back up:
* Try to ssh to the host
* Try to ssh to the management port (see SRE/Dc-operations/Hardware_Troubleshooting_Runbook)
Might be interesting also to run the network tests:
* Portal:Cloud_VPS/Admin/Network/Tests
== Common issues ==
Add here any issues you encounter.
Check also Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures
== Related information ==
* SAL (or sal.toolforge.org).
* Netbox
* Puppet merged changes
* SRE/Dc-operations/Hardware_Troubleshooting_Runbook
* Portal:Cloud_VPS/Infrastructure
* Portal:Cloud_VPS/Admin/Troubleshooting
* Portal:Cloud_VPS/Admin
* Portal:Cloud_VPS/Admin/Network#cloudgw
== Support contacts ==
== Old incidents ==
Provide a list of phabricator tasks of previous incidents to be able to more review previous instances of the issue.
Portal:Cloud VPS/Admin/Runbooks/TooManyCloudnetsDown

== Error / Incident ==
All the cloudnet nodes are down in the system.
Cloudnet nodes are the ones acting as routers and setup networks for the virtual machines.
When they go down all the VMs stop having network access and new VMs are unable to boot correctly.
== Debugging ==
This might be the result of an ongoing intervention gone wrong, so first try to communicate with the rest of the team and look for any ongoing actions:
* Ask on irc (#wikimedia-cloud-admin) if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
Then you can continue trying to figure out why the hosts are going down and not coming back up:
* Try to ssh to the host
* Try to ssh to the management port (see SRE/Dc-operations/Hardware_Troubleshooting_Runbook)
Might be interesting also to run the network tests:
* Portal:Cloud_VPS/Admin/Network/Tests
== Common issues ==
Add here any issues you encounter.
Check also Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures
== Related information ==
* SAL (or sal.toolforge.org).
* Netbox
* Puppet merged changes
* SRE/Dc-operations/Hardware_Troubleshooting_Runbook
* Portal:Cloud_VPS/Infrastructure
* Portal:Cloud_VPS/Admin/Troubleshooting
* Portal:Cloud_VPS/Admin
== Support contacts ==
== Old incidents ==
Provide a list of phabricator tasks of previous incidents to be able to more review previous instances of the issue.
Portal:Cloud VPS/Admin/Runbooks/TooManyCloudvirtsDown

== Error / Incident ==
There's a large amount of cloudvirt nodes down in the system.
This will have an effect on the capabilities of Openstack to schedule and run VMs by limiting the resources available. Possibly preventing new VMs from getting started and some running VMs to misbehave or not function.
== Debugging ==
This might be the result of an ongoing intervention gone wrong, so first try to communicate with the rest of the team and look for any ongoing actions:
* Ask on irc (#wikimedia-cloud-admin) if someone is taking any action and forgot about it
* Check SAL (or sal.toolforge.org) for host being rebooted/taken down
* Check Netbox to make sure the host is not being decommed or network changes were not applied
* Check puppet for recent changes that might have affected the node
Then you can continue trying to figure out why the hosts are going down and not coming back up:
* Try to ssh to the host
* Try to ssh to the management port (see SRE/Dc-operations/Hardware_Troubleshooting_Runbook)
== Common issues ==
Add here any issues you encounter.
== Related information ==
* SAL (or sal.toolforge.org).
* Netbox
* Puppet merged changes
* SRE/Dc-operations/Hardware_Troubleshooting_Runbook
== Support contacts ==
== Old incidents ==
Provide a list of phabricator tasks of previous incidents to be able to more review previous instances of the issue.
Portal:Cloud VPS/Admin/SSL certificate

This page contains information on how to administer and replace the star.wmflabs.org SSL certificate
For traditional certificates, they are usually valid for 1 year, and they should be renewed at least 2 weeks prior to expiration date.
For LE certificates, we will use acme-chief.
==Usage==
===*.wmflabs.org (also known as star.wmflabs.org)===
This certificate is in use by the main web proxies in WMCS, and is physically deployed in the proxy-*.project-proxy.eqiad.wmflabs instances.
The private key is hosted directly in the server (non-puppetised).
Example URL using this SSL certificate: https://tools-static.wmflabs.org/
This is a purely project-proxy thing. To move to acme-chief would probably involve project-proxy having a puppetmaster, and at least designate-admin permissions in the wmflabsdotorg tenant.
==Renewing star.wmflabs.org==
#request certificate renewal, usually with DCops and/or Traffic team. Involves purchase approval, etc. May take a couple of weeks.
#the private key is added to the /srv/private repo in a prod puppetmaster without replacing the old one. Usually with a new. prefix in the filename. This is usually done by whoever purchases the certificate.
#once the public key is received, an operations/puppet.git repo patch should be staged into gerrit. This is usually done by whoever purchases the certificate. This patch is not merged yet.
#ensure affected servers/services are working correctly previous to further operations.
#disable/stop puppet agent in all affected servers, i.e, the servers running the webservers using the certificate to be renewed.
#merge public key patch into the operations/puppet.git repo.
#replace the old private key with the new one in the /srv/private repo in production puppetmaster.
#scp private key to nova-proxy servers and put it into /etc/ssl/private.
#enable and run puppet agent in one of the server to see if all public/private keys are in place. Restart nginx to see if it can start with no issues with the new certificate.
#if all was fine in the previous test, continue to all other servers.
===Example of replacing *.wmflabs.org===
A collection of example Phabricator tasks:
*T206223 *.wmflabs.org cert needs renewing
*T174611 update *.wmflabs.org by 2017-10-16
*T237066 Push renewed *.wmflabs.org certificate and new private key to cluster (expires 2019-11-16)
TODO: add command examples
Portal:Cloud VPS/Admin/Sshd

Cloud VPS instances do some interesting things with sshd config. This page will try to explain what we do, why we do it, and offer some troubleshooting tips for when things go wrong.
== Config ==
=== ssh-key-ldap-lookup ===
The ::ldap::client::utils Puppet class provisions our custom /usr/sbin/ssh-key-ldap-lookup script. This script is used by sshd to fetch a list of authorized keys for a given user from the LDAP directory.
=== PAM ===
* /etc/pam.d/common-session - we configure pam_mkhomedir.so to create missing $HOME directories
* /etc/security/access.conf - we add config to restrict authn to a particular group ($::projectgroup)
== Troubleshooting ==
=== Check the ssh keys ===
Compare auth.log records to /usr/sbin/ssh-key-ldap-lookup <user> output to validate the key being presented. Be warned that ed25519 keys may not look the same in both outputs even when they are due to differences in the hash logging.
=== Check for project membership ===
If the auth.log output has something like "Fatal: Access denied for user $USER by PAM account configuration [preauth]", chances are pretty good that the user exists in LDAP but is not a member of the project.