Heterogeneous deployment

This page documents the design and architecture of Heterogeneous deployments (abbreviated as "HET"). This document describes the current infrastructure, which allows for different wikis to be running different MediaWiki versions at the same time.
You may be looking for the following:
* Current manual on how to deploy code: Trust How to deploy code over this.
* /Train_deploys for MediaWiki upgrades on Wikimedia infrastructure.
* Rationale and status updates of the 2011 initiative that introduced HET: "Heterogeneous deployment" project page (mediawiki.org)
* Original 2010 experiment with multiple deployed versions: Heterogeneous deployment/old.
== Where things live ==
Site configuration lives in /srv/mediawiki-staging/wmf-config on the deployment host, and tools such as scap sync this directory to /srv/mediawiki on cluster machines. The live branches of the MediaWiki codebase live as subdirectories of /srv/mediawiki. Most files are maintained in the git repository .
===mediawiki-staging/===
* wikiversions.json: a JSON object mapping wikiname -> version. Versions must have a "php-" prefix, like "php-1.30.0-wmf.2". This is here for admins to edit as needed. Note that the wiki-creation script (addwiki.php) updates this file automatically.
* wikiversions.php: an optimised version of the wikiversions.json file. Never edit this directly, it is not committed to Git.
===mediawiki-staging/multiversion===
* MWMultiVersion.php: Class for determining the database name and version of a wiki. Supports loading from three entry points:
** Web access of an uploaded file through apache - setSiteInfoForUploadWiki()
** Web access of a other wiki content through apache - setSiteInfoForWiki()
** From within a maintenance script - setSiteInfoForMaintenance()
* MWScript.php: Wrapper for running the right version of a maintenance script for a wiki.
===mediawiki-staging/wmf-config===
This is where most of the site configuration lies. Main files include things like:
* CommonSettings.php: Mostly handles shared settings or settings with few exceptions. Also handles the task of getting the InitialiseSettings.php options into globals that can be used for MW, and the caching thereof. Site, language, and DB name globals are extracted from the MWMultiVersion object. A MWMultiVersion object should always be initialized by the time this file is loaded. It also sets the i18n message directory locations for extensions. Other random things are performed here.
* InitialiseSettings.php: Mostly contains MW configuration settings for individual wikis as well as wiki groups (like all "wikibooks" for example).
* extension-list: The list of extensions available for use on the site (though some may not be enabled at the moment). These extensions must be installed on cluster. This list is used to automatically generate the ExtensionsMessages-*.php files needed for localization.
* CommonSettings-labs.php, InitialiseSettings-labs.php, and other *-labs files contain overrides for the Beta Cluster.
===mediawiki-staging/php-*===
These are MediaWiki installation directories.
===/usr/local/bin===
See Wikimedia binaries for a more exhaustive list.
* scap: script to sync all hosts with the state of the deployment server.
== How to do stuff ==
=== General remarks on deploying code ===
==== Basic common sense ====
* Be careful. Breaking the site is surprisingly easy!
* If you're deploying code written by someone else, ask them to be around during deployment so they can troubleshoot if necessary.
* Make sure you know about anything hairy, such as additional prerequisites (e.g. schema changes) or potential complications when rolling back.
==== Deployment Requirements ====
* Production shell access (requires manager and/or sr dev approval).
** If you can ssh to deployment.eqiad.wmnet, and can ssh to an mw host (e.g. an mwdebug host), then you have production shell access.
* Merge access for "wmf" branches in Gerrit.
==== Don't leave town ====
Even if your deploy appears to be working, it's important to be reachable in the hours immediately following your deploy. Ideally, stay online and in IRC channels like #wikimedia-operations for an hour or two. 
If you must go offline, let people know how to reach you (and keep your mobile phone or other communications device on your person). You can use /away messages on IRC, or perhaps send a short email to the ops list.
If you are on Wikimedia Foundation staff, now might be a great time to check if your contact info is up to date.
=== Basic code deployment steps ===
Really, you should check out the How to deploy code page. It is the canonical place for documentation on how to deploy code, obviously.
===Change wiki configuration===
See Configuration files for a list of configuration files and their formats. Configuration changes are not deployed as part of the train. Instead, they are rolled out separately, often in a backport deploy. See also How to deploy code#Step 3: configuration and other prep work. If you want to merge a change that is not part of a backport window, first make sure you won't conflict with a scheduled deployment train.
==== In your own repo via gerrit ====
If you need someone else to merge and push your change, follow this path.
* for the person making the change
** in your own checkout of the operations/mediawiki-config project, make the changes you need
** push the change to gerrit for review
* for the person merging and pushing the change live
** on deployment.eqiad.wmnet, cd to /srv/mediawiki-staging/
** run git status and verify there are no pending changes. Untracked files are ok.
** in gerrit: rebase, review, and merge the change
*** The rebase is to make sure the merge commit below isn't a false positive. Old patches will create FF-only merge commits.
** on deployment.eqiad.wmnet, run git fetch origin && git diff HEAD origin. Review the diff and make sure it's only the changes that you expect
*** if more than the changes you're making show up, STOP and find out why and make sure they're ok to merge.
** if it looks good, run git merge origin/master
*** if this creates a merge commit, that means someone has committed a local change without pushing; this isn't the end of the world but it should be fixed, so either fix it yourself or ask for help
** run scap sync-file wmf-config/filename 'Summary of change including commit short hashes' for each changed file or directory separately
*** git log --abbrev-commit will give you the short hashes for commits
==== Change on deployment.eqiad.wmnet then commit ====
If you are going to make the change, review, and push all by yourself, you can follow this path in an emergency.
* Edit the file(s) you need to edit. 
* cd to /srv/mediawiki-staging/
* Run git diff wmf-config , which will display a diff of recent changes to the wmf-config directory. Verify that these changes are sane and that there are no traces of someone else messing with the same files
* If everything looks good, run scap sync-file wmf-config/filename 'Summary of change' for each changed file or directory
* Run git commit wmf-config/filename and add a decent commit summary. Then do git push origin HEAD:refs/for/master
* Do not forget the git push step above, things will get messed up if you do (and nobody can review your changes in Gerrit)
NOTE: it is possible that config changes via scap sync-file might not apply if they are not re-cached properly. If you think this happens to you, you can touch wmf-configfilename and then scap sync-file wmf=config/filename 'Summary of change' again.
===Run a maintenance script on a wiki===
This section was moved to Maintenance server.
===Run a maintenance script on a group of wikis===
See Wikimedia binaries#mwscriptwikiset.
===Run a maintenance script on all wikis===
See Wikimedia binaries#foreachwiki.
===Install a new extension on a wiki===
If you're deploying an extension or feature that can be switched off, it's usually best to leave it switched off while you deploy and carefully switch it on after that using a simple configuration change (this is called a dark launch). Even if you do this, you should build any configuration infrastructure (e.g. $wmg variable, adding entry in InitialiseSettings with default false) at this time so all you'll have to do later is flip a switch.
For specific preparations, see the sections below as well as Creating new tables, How to do a schema change, and How to do a configuration change. Best to perform schema changes before making config changes.
==== Add a configuration switch for an extension ====
In /srv/mediawiki-staging/wmf-config/CommonSettings.php, add:
if ( $wmgEnableMyExtension ) {
	wfLoadExtension( "MyExtension" );
	// Set config vars if needed
	// If you want to export config vars through InitialiseSettings.php, you need to set $wmgMyExtensionThingy there and do
	#$wgMyExtensionThingy = $wmgMyExtensionThingy;
}
In /srv/mediawiki-staging/wmf-config/InitialiseSettings.php, add something like:
'wmgEnableMyExtension' => array(
	'default' => false,
	'eswikibooks' => true,
	// etc.
),
// If needed, set $wmgMyExtensionWhatever vars here too
If your extension requires a large-ish amount of configuration, consider putting it in a separate file instead. Currently, AbuseFilter, LiquidThreads and FlaggedRevs do this.
For more documentation on these files and their formats, see Configuration files.
==== Add new extensions to extension-list ====
When adding a new extension, you need to add it to the extension list, or its i18n messages won't get picked up. For more information about this setup, see this.
# cd to /srv/mediawiki-staging/wmf-config
# Edit extension-list and add the path to the extension setup file on a line by itself
# Where X.XX is the current MediaWiki version, run mwscript maintenance/mergeMessageFileList.php --wiki=aawiki --list-file=extension-list --output=ExtensionMessages-X.XX.php and watch the output for errors. Your extension should appear in the output the same way the others do
#* If you get an error, you probably got the path wrong, or the extension setup file is wonky
====Deploy the extension====
Do what needs to be done to deploy and register changes in these config files. You will need to run scap afterwards (so the config and message cache is updated).
====Add to repos/releng/release====
To have your extension added to newly created WMF branches, please add the newly added extension to the wmf_core list in make-release/settings.yaml in the release tools repository.
Category:How-To
Category:Deployment
Category:MediaWiki production