Wikimedia Cloud Services team/EnhancementProposals/WMCS-SRE-book

The WMCS SRE book is a collection of agreement, best practices and engineering operations workflows that we all in the team try to follow and apply in our day to day work.
You can find here hints and tips for common tasks and situations we face. It is not intended to contain very technical information (like how to restart service X).
== Community interaction/communication ==
This section describes our practices for interacting with the community for some special operations, including but not limited to:
* deprecation plans (we are shutting down a service or are no longer supporting a given technology)
* expected downtime (we have to do a planned operation in a window)
* outage/incident communication (we suffered an unexpected issue that affected our users/community)
=== Communication channels ===
We interact with our users using different mechanisms:
* Using Phabricator tasks and a number of Phabricator tags/workboards.
* On IRC, in the channel on libera.chat.
* Via mailing lists: cloud-announce@lists.wikimedia.org, cloud@lists.wikimedia.org, and cloud-admin@lists.wikimedia.org.
* On wiki: News articles on wikitech, talk page messages on wikitech, talk page messages on "home" wikis
* Direct email
Due to the nature of each channel (real time, async, group, direct, etc) they are often used for different purposes.
TODO: introduce here general rules for picking a channel to send a particular type of communication
=== Deprecation plans ===
TODO: add information and examples here. Context: Grid migration, Ubuntu deprecation in CloudVPS/Toolforge, Jessie deprecation, etc.
=== Expected downtime ===
It is common that we need to perform operations that introduces downtime for our services or otherwise creates negative impact for our users experience.
This section contains detail on how to handle this kind of situations.
General rules:
* If the operation is going to cause downtime to users (any amount), announce it to the mailing lists at least 1 week ahead of the window.
* When communicating operation windows, be precise in which kinds of downtime users should expect (all services down? network failure? database failure?)
* When communicating operation windows, include concrete information on what services are affected (does this affect Cloud VPS but not Toolforge? or the other way around?)
* When communicating operation windows, be very explicit about time and dates. Use an internationally format for dates (YYYY-MM-DD) and the UTC timezone for times.
Recommended timeline:
* 1 week before the operation window: initial announcement email
* 1 minute before the operation window: email letting users know the operation is starting
* 1 minute after operations ended: optional email letting users know the aftermath (not always required)
==== Examples ====
Example email sent by Andrew regarding an Openstack upgrade to the cloud-announce mailing list. The email was sent 7 days in advance:
Subject: cloud-vps maintenance Monday, 2019-10-07
We'll be upgrading the cloud services OpenStack install on Monday, beginning at 14:00 UTC.
The entire upgrade process may take a couple of hours. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) 
There may also be brief network interruptions during the upgrade, although if all goes well these will not be noticeable by users.
Toolforge and existing VMs should be largely unaffected apart from possible network hiccups. 
Example email sent by Arturo regarding an operation to reboot several cloudvirt servers, sent to the cloud-announce mailing list. The email was sent 7 days in advance:
Hi there,
Next Wednesday 2019-10-09 at 09:00 UTC we will be doing a maintenance operation
on some of our cloudvirt servers (the hypervisor servers) that involves
rebooting both the physical servers and the virtual machines running on them.
The reason is that we need to update the running linux kernel version they have.
In this window we will reboot 4 hypervisors:
* cloudvirt1008
* cloudvirt1009
* cloudvirt1012
* cloudvirt1013
The procedure will be to reboot a server, wait for it to come back online (could
take up to 5 minutes) and wait for all the VMs to come back online. Then move to
the next server.
Toolforge users may see their tools and webservices briefly disrupted due to
several components of the Toolforge infrastructure being rebooted in this operation.
If nothing changes (reallocated or new virtual machine, etc) this is the list of
affected VM instances in each hypervisor:
* cloudvirt1008:
VM: tools-sgebastion-09 PROJECT: tools
VM: tools-k8s-master-01 PROJECT: tools
VM: deployment-cache-upload05 PROJECT: deployment-prep
VM: toolsbeta-paws-worker-1002 PROJECT: toolsbeta
VM: toolsbeta-puppetmaster-02 PROJECT: toolsbeta
VM: tools-mail-02 PROJECT: tools
VM: tools-prometheus-02 PROJECT: tools
VM: tools-elastic-01 PROJECT: tools
VM: tracker1 PROJECT: lta-tracker
VM: tools-clushmaster-02 PROJECT: tools
VM: tools-worker-1020 PROJECT: tools
VM: tools-k8s-etcd-01 PROJECT: tools
VM: tools-worker-1010 PROJECT: tools
VM: tools-worker-1008 PROJECT: tools
VM: tools-worker-1007 PROJECT: tools
VM: tools-worker-1003 PROJECT: tools
VM: tools-sgeexec-0937 PROJECT: tools
* cloudvirt1009:
VM: toolsbeta-paws-master-01 PROJECT: toolsbeta
VM: tools-elastic-02 PROJECT: tools
VM: tools-paws-worker-1005 PROJECT: tools
VM: tools-prometheus-01 PROJECT: tools
VM: tools-paws-worker-1002 PROJECT: tools
VM: puppet-lta PROJECT: lta-tracker
VM: tools-flannel-etcd-03 PROJECT: tools
VM: tools-worker-1017 PROJECT: tools
VM: tools-k8s-etcd-02 PROJECT: tools
VM: tools-worker-1013 PROJECT: tools
VM: tools-worker-1012 PROJECT: tools
VM: tools-worker-1009 PROJECT: tools
VM: tools-worker-1006 PROJECT: tools
VM: tools-worker-1004 PROJECT: tools
* cloudvirt1012:
VM: tools-paws-master-01 PROJECT: tools
VM: deployment-ms-be06 PROJECT: deployment-prep
VM: toolsbeta-worker-1001 PROJECT: toolsbeta
VM: deployment-cumin02 PROJECT: deployment-prep
VM: toolsbeta-k8s-master-01 PROJECT: toolsbeta
VM: toolsbeta-k8s-etcd-01 PROJECT: toolsbeta
VM: toolsbeta-puppetdb-01 PROJECT: toolsbeta
VM: tools-redis-1002 PROJECT: tools
VM: tools-paws-worker-1003 PROJECT: tools
VM: tools-paws-worker-1001 PROJECT: tools
VM: tools-elastic-03 PROJECT: tools
VM: tools-worker-1025 PROJECT: tools
VM: tools-worker-1026 PROJECT: tools
VM: tools-worker-1022 PROJECT: tools
VM: tools-worker-1019 PROJECT: tools
VM: tools-worker-1018 PROJECT: tools
VM: tools-k8s-etcd-03 PROJECT: tools
VM: tools-worker-1016 PROJECT: tools
VM: tools-flannel-etcd-01 PROJECT: tools
VM: tools-worker-1014 PROJECT: tools
VM: phlogiston-5 PROJECT: phlogiston
VM: dumps-3 PROJECT: dumps
VM: codesearch4 PROJECT: codesearch
VM: wikispeech-wiki-stretch PROJECT: wikispeech
VM: ores-worker-01 PROJECT: ores
VM: puppet-jmm-kernel-stretch2 PROJECT: puppet
VM: mcr-base PROJECT: mcr-dev
VM: rel2 PROJECT: search
VM: mc-clusterA-2 PROJECT: test-twemproxy
VM: wikibrain-embeddings-02 PROJECT: wikibrain
VM: qube-node1 PROJECT: k8splay
VM: cindy PROJECT: pluggableauth
VM: cvn-apache9 PROJECT: cvn
VM: zk1-2 PROJECT: analytics
* cloudvirt1013:
VM: tools-flannel-etcd-02 PROJECT: tools
VM: paws-ext-lb-01 PROJECT: paws
VM: abogott-puppetclient PROJECT: testlabs
VM: tools-worker-1028 PROJECT: tools
VM: tools-worker-1005 PROJECT: tools
VM: cloudstore-dev-02 PROJECT: cloudstore
VM: cloudstore-puppetmaster-01 PROJECT: cloudstore
VM: deployment-aqs03 PROJECT: deployment-prep
VM: osmit-test PROJECT: osmit
VM: tools-sgewebgrid-lighttpd-0927 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0926 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0925 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0924 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0923 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0922 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0920 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0917 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0909 PROJECT: tools
VM: tools-sgeexec-0925 PROJECT: tools
VM: tools-sgeexec-0923 PROJECT: tools
VM: tools-sgeexec-0910 PROJECT: tools
VM: cyberbot-db-01 PROJECT: cyberbot
regards.
== Changes tracking ==
TODO: Do we have internal rules for introducing patches into the several repos we use for work? ops/puppet, dns, etc.
== Tickets and work tracking ==
This section describes our practices using a ticketing system (Phabricator) to track our work and issues related to our systems and services.
See also: mw:Wikimedia Cloud Services team/Team work board practices
== Server Admin Logs ==
This section describes our practices for recording the operations we do, like changes to servers or services.
Logging an operation to a SAL creates a papertrail of what we do. This improves collaboration in a team which is distributed by nature. Also, it helps improving the transparency of our operations.
* Use !log admin ... in to log all operations related to the Cloud VPS service in general. This is the SAL for the Cloud VPS service.
* Use !log tools ... in to log all operations related to the Toolforge service. This is the SAL for the Toolforge service.
* Use !log ... in to log all operations related to physical hardware. This is the SAL for general operations tasks.
TODO: add information. Describe how we use the different !log mechanism in the different channels.
== Personal availability ==
This section includes information regarding how we try to coordinate to ensure we always have enough human availability to support our services.
TODO: information on vacations, unavailability communication, etc. Does this fit into this document?
=== On-call and paging ===
Worth mentioning Wikimedia_Cloud_Services_team/Clinic_duties which we do while on-call.
TODO: include here all we know about how we coordinate paging/on-call, etc? Does this fit into this document?
== Additional notes ==
Please, take these additional notes into consideration.
* This document is intended to be an addition to other standard and industry procedures, and it does not replace them, just adds additional information and tips.
* Changes to this document requires agreement among the affected people (i.e, the WMCS folks).
* We agreed and follow our Technical Engagement Team Social Norms.