Streamlined Service Delivery Design/research

There is an effort underway to introduce continous delivery to WMF,
which started early in 2017. The exact shape and scope of this effort
is still under discussion. User:Lars Wirzenius was hired in
October 2018 to help with this.
This document outlines Lars's understanding and thinking so far. It's
meant to aid discussion of this topic within the Release Enginering
team, and other relevant parties interested, and to show where Lars is
in the dark, to show where Lars have understood things correctly, and
reveal things that need discussion. If you disagree with something
here, please contact Lars and set him straight.
= Continuous concepts =
== Continuous integration ==
w:Continous integration (CI) is the practice of merging changes to
the main line of development frequently, preferably many times a day.
The idea is to keep everyone working on about the same version of a
code base, so that no-one's work diverges from the others for long or too much, to
reduce problems and unnecessary work due to integration issues.
CI is typically coupled with automated test suites (at least unit
tests, possibly integration tests, performance tests, or other tests),
possibly in a way that changes don't get merged until tests pass, though it
is common for tests to be run after a merge. CI also typically
involves having an automated system run tests automatically, either every
time the main line of development changes, or when a change is offered
to be integrated into the main line, periodically, or all of the
above.
== Continuous delivery ==
w:Continuous delivery is the practice of releasing software
frequently. Continous delivery typically uses CI to ensure the main
line of development is in a releable state at all times. The actual
release process if typically fully automated, but making a release is
a manual decision. The release process typically results in a number
of artifacts, which can be used to install / upgrade / deploy the software.
== Continuous deployment ==
w:Continous deployment is the practice of deploying all changes to
the main line of development to production. In contrast to continuous
delivery, what gets deployed might not be a release, and what gets
released might not get deployed.
== CD vs CD ==
Continuous delivery and continuous deployment are both commonly
abbreviated as CD. This document doesn't abbreviate either, for
clarity.
== Pipelines, gates ==
Continous integration, delivery, and deployment are often all modelled
using "pipelines", where a change enters a pipeline, goes through a
number of steps, and results in a change at the end. A step might be,
for example, "get the source code", "build source code", "run unit
tests", or "install on production server". At various steps in a
pipeline might be "gates", where progress stops, until and unless an
entity indicates that it's OK to move forward. A gate might "manual testing did not find problems".
= Current status =
WMF provides a large number of web sites. There are around a thousand
wikis, implemented with MediaWiki, plus other sites MediaWiki sites often use backing
services, which are often microservices.
* Example of microservice: Mathoid renders custom markup in wikitext to MathML and images.
* Databases and load balancers are other types of backing services.
Running a MediaWiki site counts as a service for this document, even
if MediaWiki doesn't fit into the model of a microservice with an HTTP
API. Deploying MediaWiki should ideally be similar to deploying
backing services.
Changes to MediaWiki sites are deployed via two ways: the SWAT and the
Train. SWAT is mainly for configuration changes, small bug-fixes for
user-visible problems, and other low-risk changes. The Train is for
other changes, most importantly, changes to MediaWiki core and
extensions.
SWAT happens a couple of times a day (European afternoon, North
American afternoon), and Train on Tuesday through Thursday most weeks.
The Train changes group 0 ("minor sites") on Tuesday, group 1 ("medium
sites") on Wednesday, and group 2 ("big sites") on Thursday, in the
hope of catching errors on smaller sites before they affect large
numbers of people.
* SWAT
* Train
The goal is to move services individually to a continuous delivery
model. One microservice has been moved (Mathoid). Three more are
destined to be moved by the end of 2018 (Graphoid, Zotero,
Blubberoid). This means they will be deployed and updated using the
new Delivery Pipeline for WMF in the future.
* Goals
* FIXME: Link to Deployment Pipeline
Currently deployment is triggered manually, but aided by some
scripting, and happens daily (SWAT) or weekly (Train). Many of the
mechanical steps of deployment have been scripted (scap), but the
process still requires manual work. Part of the manual work is
checking that the service still works after an upgrade. Part of it
is change review, and in some cases small fixes to the changes.
There seems to also be a lot of ad hoc communication happening
between developers and the Release Engineering team. Deployment is
labor-intensive, and as such too error-prone.
* scap
Scap uses canary servers when deploying: a small subset of the
servers are deployed to first, and then logs and other things are
checked for problems. If everything goes well, the rest of the servers
are deployed to as well.
There is some automated testing of running services.
service-checker which consumes an OpenAPI specification
for HTTP endpoints to check along with their expected responses. In
addition, the logstash log collection service is queried to see if
rates of error change after deployment.
* service-checker
* OpenAPI example
* Question: at what point are these tests run?
There is also some monitoring of services, which SRE sets up and
maintains. Monitoring alerts relevant parties, when it notices
something breaking. It should be noted that monitoring is not the same
as testing: a test suite tells you what aspect of a service doesn't
work ("front page doesn't say Wikimedia"), monitoring tells you when
measuring some aspect of a service doesn't fall into an expected range
("too many 500 status codes in the log file"). Both are needed. Test
suites are especially useful when making changes to the service code
("do the things that the test suite tests, still work?"); monitoring
is needed when something changes without deployment (server catches
fire).
= Known problems with current status =
All services (except the one already moved to the delivery pipeline)
run on bare metal, or in virtual machines. This limits how well WMF
can react to fluctuations in traffic, and thus increases hosting
costs, due to having to over-allocate hardware resources (each service
needs to have all the resources it needs for peak traffic). A more
container-based system could save on hosting costs by not allocating
hardware for the peak load time of a particular service, and instead
share hardware for different services according to load.
* Question: not sure how big a problem this is.
Software development for WMF needs is slow, because deployment slows
it down. At the same time, deployment is risky because the
deployment process doesn't keep pace with development; it can't keep
pace because it's socially and cognitively too expensive.
Software development can be modelled as loops within loops.
Development goes faster, and the software developing entity is more
productive, when loops are iterated faster (or at least that's been my
experience over the decades). Removing friction and obstacles and
automating steps within a loop helps. The innermost loop is the "edit,
build, test" cycle. Deployment is in the "build, deploy, test" loop.
Making deployment easier and faster will help the overall software
development productivity of Wikimedia and its community.
The WMF sites stay up thanks a lot of manual effort spent on review.
This is aided by some automated testing, and deployment tooling,
monitoring, and many users (both WMF staff and in the community) who
eagerly report problems. There seems to be no major issues with
quality and level of service, but it seems to require a lot more human
effort than might be necessary. The friction coefficient of the
deployment loop is high.
While WMF doesn't have a profit motive, it seems nevertheless that
making processes smoother and more automated would be beneficial for
WMF and the Wikimedia community in general, by freeing people to do
more amazing things and spend less effort on mundane, repetitive
deployment stuff. Better tools to are force multipliers for brains.
Setting up entirely new services is a medium-big project. This seems
like it should be less of an effort.
= Desired status =
All services run in containers, hosted by Kubernetes. 
This is still under discussion and possibly controversial, so Lars
proposes two phases:
Phase 1: Continuous delivery: All the mechanical steps of
deployment are automated using scap (or other tooling). Deployment is
triggered when a deployer or release engineer runs the deployment
script. There's still communication with developers, and keeping an
eye on things, as part of the process. Changes that reach the
beginning of the delivery pipeline will have been reviewed and
approved already. The Train will be replaced by a more SWAT-like
approach of changes getting deployed every workday.
Phase 2: Continuous deployment: Deployments are fully automated,
except for change review. The deployment pipeline has one or more
gates at which manual review is done. The deployment pipeline is
triggered when a developer request a change to be reviewed and merged.
There are automated tests for all services. The test have sufficient
coverage and quality that if tests pass, the release engineering team
have confidence that the sites and services work for our users.
A possible cultural change: when anything breaks, and it should
have been caught by automated tests, tests are changed to catch it in
the future.
== SRE team ==
SRE has responsibility for keeping Kubernetes running, as well as any
other infrastructure (DNS, databases, etc). SRE also sets requirements
for what runs in production: security, version traceability,
testability, monitoring, and more.
Question: Does SRE want to review the code running in
production, or its configuration? Also, changes to that. Probably not,
but check with them.
SRE handles databases and their configuration seems to be at least
partially in the MediaWiki configuration and code. They will probably
want to review any changes to that.
== Release engineering team ==
The Release Engineering team has responsibility for providing and
maintaining tooling to do deployments automatically, running
automation to make deployments happen frequently, and reviewing
changes to production (a sanity check, if nothing else).
== Service developers (community and WMF) ==
The developers of the services have responsibility for the writing
and maintaining the service code, documenting service configuration,
and writing and maintaining automated tests for the running service.
To facilitate this, developers get quick feedback from automated tests
if anything seems hinky, so that they can fix it before the Release
Engineering team gets involved. Quick here means within minutes. This
is achieved by CI (Jenkins).
Possible cultural change: If any reviewer finds anything to fix,
even if only extra whitespace, the change is made by the developer.
(But silly, simple things like whitespace can be tested for
automatically, and such tests are run before a human reviewer ever
looks at the change.)
Possible cultural change: If there is a problem, the sites can
be rolled back to a known-working version easily. When this happens,
the automated tests get improved so they'll catch that problem in the
future. The responsibility for improving the tests lies with both the
developers and the release engineering team. Canary servers are still
used, to look for problems that only happen under production
condtions.
The smaller and safer a change is, the less effort it is to get it
deployed to production. For example, a change to translations should
be possible to do within an hour after having been approved by a
reviewer. (That's a goal, not a requirement or a promise.)
= Overview of planned solution =
All services which can be run in a container, are run a container.
Persistent data, such as databases, will stay on bare metal.
Containers will communicate with them over the network. We will start
by moving microservices into containers, and move MediaWiki last,
probably not before late 2019.
All changes are built and tested by CI (Jenkins), which also builds
container images. Such images get deplooyed into test instances, and
such instances are tested.
This happens for changes to the master branch, as well as for
developer branches. Phabricator and Gerrit are used to track
changes, as before. Each change is automatically built and tested,
and once tests pass, submitted by the developer for review by a
human. If accepted by reviewer, merged into the master branch.
Changes are handled in a way that notices when they work
individually, but break together.
All configuration also gets deployed from git, and merged into the
master branch using the same process.
No human ever changes the master branch directly. (If possible, the
git server will be configured to prevent that.)
= Suggestion for a continuous delivery process =
Lars suggests the following process for a continuous delivery process.
There is a textual description, followed by a UML sequence diagram.
== Roles and systems ==
The process involves three roles:
; developer : makes the actual change
; reviewer : reviews the changes, accepts it (who this is depends on change, and may include QA or testers)
; deployer : pushes change to production (might be someone in the release team)
The process also involves a number of computer systems:
; Phabricator : ticketing system (for this)
; Gerrit : code review system
; Zuul : gating system
; Jenkins : runs automated jobs, e.g., to build Docker images
; Docker image store: stores images built by Jenkins
; staging, production: various instances of Wikimedia sites and services
An instance here means the full set of virtual machines, containers, and software running in them, configured in the same as for production. However, non-production instances may have fewer resources (less CPU, less RAM, less disk space, less everything) than production does. Also the data in the instance may be different.
== Process description ==
(This has been simplified a lot from previous versions. Important changes: 
no more ticket per change;
no more Jenkins running system tests on the Docker image)
Note that this is a description only of the delivery pipeline, not other deployments.
It also only applies to things running in containers in Kubernetes.
The process is roughly as follows:
The pipeline starts with someone (a developer) pushing a change to Gerrit.
They've already tested it locally.
Gerrit generates a change event, which Zuul listens to, and Zuul
triggers Jenkins to run a job that builds the software,
and runs any tests in the build tree. Jenkins also
builds and publishes various variants of the Docker images for the software 
to the image store.
If all of that goes OK, Jenkins informs Gerrit of this, and records in
the ticket that CI is happy with the change. This triggers a
notification to reviewers that a review is needed.
The reviewer looks at the change, and may download the image from the
image store. The reviewer decides if
they're happy or if they would like something to change in the change
before they're happy. If they're happy, they record the fact in
Gerrit. This triggers Gerrit to merge the change to the master branch,
and notify the deployers.
The deployers manually trigger a Jenkins job to deploy the changes (or
current master branch) to a staging Kubernetes cluster. The deployment
is fully automated, and uses
the exact same Docker images that will later be deployed to production.
The deployers review the staging cluster and decide if the change can
be deployed to production. If so, they record this is Gerrit as a
vote.
Then the deployers trigger a Jenkins job to deploy to production. It
is identical to the deployment to staging. This is recorded to Gerrit.
The deployer closes the ticket for the change.
If somethings fails at any point, the process resets to the beginning. If something fails
after deployment to staging or production, the instances are reverted to the state
after the change.
== Process as a UML sequence diagram ==
== Discussion ==
The process, as outlined above, allows for only one change to be in
flight at a time. This is mosly for simplicity, but it seems to Lars
that everything after change review should be linear, to avoid
confusion stemming from trying to do too many things at a time.
However, it would be catastrophic to development speed to only work on
one change at a time. Thus, it seems sensible to allow many changes to
be in flight at a time. However, it seems like a recipe for much
confusion to review or test multiple changes at a time. Thus, ideally,
each change would result in a separate "testing" instance, where each
instance has the minimal set of services running.
Given that not everything will be running in Kubernetes containers, at
least during a transition period, an "instance" should probably be a
Kubernetes cluster, plus a set of virtual machines and other systems
to run the full set of software the powers Wikimedia's sites
(MediaWiki instances, databases, load balancers, services running in
VMs, services running in Kubernetes, etc). This is not cheap in terms
of resources, which is a concern.
Production needs to be configured to handle all real traffic. Staging
needs to be identical, except it doesn't need all the resources to
handle real traffic. Staging should have a copy of real data from
production, refreshed from production at suitable intervals. Testing
can be a small instance, with minimal resources.
The idea is that production is what users actually see and interact
with, and we never break it. Staging is as similar to production as we
need it to be to be confident that if we deploy a change to staging,
and things work there, deploying the same change to production will
also work. Testing is where we can deploy any changes, confident that
if we break it, it doesn't matter. We need to be able to rebuild the
testing instance without it being a big thing; the rebuild should
ideally be fully automated. A rebuild may be necessary to undo a bad
change. Testing is where we experiment, and sometimes experiments will
fail badly: for example, changes to database schemas, or upgrades of
PHP may break everything irreversibly. It needs to be safe to deploy
changes to testing without having to be too careful.
If we have the resources, having many testing instances would be
ideal. We could then have a separate testing instance for each change.
When a change is merged into the master branch, all the testing
instances for other changes should be rebuilt, with their changes
rebased on top of the new master branch.