Maintenance server

The Maintenance server is where we run scheduled and ad-hoc MediaWiki maintenance scripts from the command-line.
==Server==
As of June 2024, the servers are:
*mwmaint1002, current active maintenance server in eqiad
*mwmaint2002, this one is passive, to be used if we switch activity to codfw (switchover).
We manage the scheduling in Puppet (puppet: profile/mediawiki/maintenance.pp), where we declare which scripts to run, when and how often, and what wikis to run them on. As of 2020, the scripts are run via systemd timers. Previously, we used cron to run the maintenance jobs.
== Connecting ==
To run maintenance scripts on the Beta Cluster you need to connect to the equivalent servers in the "deployment-prep" Cloud VPS project:
deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud
== Runbook ==
=== Access recent runs ===
List scheduled scripts (what is running, which will soon run, when did they last run).
 mwmaint$ systemctl list-timers 'mediawiki*'
 NEXT LEFT LAST UNIT
 2024-07-09 20:47:00Z 5s left 2024-07-09 20:46:00Z mediawiki_job_db_lag_stats_reporter.timer
 2024-07-09 21:00:00Z 13min left 2024-07-08 21:00:00Z mediawiki_job_initsitestats.timer
 2024-07-09 21:15:00Z 28min left 2024-07-09 18:15:00Z mediawiki_job_growthexperiments-updateMenteeData-s1.timer
 …
 2024-07-09 21:39:00Z 52min left 2024-07-09 20:39:00Z mediawiki_job_wikidata_resubmit_changes_for_dispatch.timer
 2024-07-10 01:00:00Z 4h left 2024-07-09 01:00:00Z mediawiki_job_purge_parsercache_pc1.timer
 …
Display recent script output. You can use -f to follow it in real-time. Omit ".timer" from the name here (based on the output above)
 mwmaint$ sudo journalctl -u mediawiki_job_startupregistrystats -n100
=== Run a maintenance script on a wiki ===
When running a long-running maintenance script, consider using screen.
To run a maintenance script, log into the active maintenance server.
Use the mwscript command-line utility:
mwscript scriptname.php --wiki dbname 
The above command will run the script scriptname.php located in the /maintenance directory of the specified wiki's php directory.
You can also use a path relative to root of the MediaWiki installation, for example to run a script from an extension rather than core. It is recommended that in this case you first browser to one of the MediaWiki directories to allow for tab completion:
$ cd /srv/mediawiki/php-1.xx-wmf.xx/ # optional; enables tab completion
$ mwscript extensions/MyExtension/maintenance/scriptname.php --wiki dbname
If you need to pass parameters to your maintenance script, you can add them to the end. You may also want to explicitly declare the database parameter to avoid confusion:
mwscript scriptname.php --wiki=enwiki --days=1
In this case, the wiki parameter is handled by mwscript and the 'days' parameter is handled by scriptname.php. The "wiki" must be in wikiversions.json so the maintenance script machinery can figure out what MediaWiki version it is running. The wiki parameter must be exactly after the script name – otherwise, it is not recognised.
Many maintenance scripts are used to run heavy database queries; remember to check the replication lag and other possible byproducts.
=== Run a custom maintenance script ===
You can also run an arbitrary PHP file outside the MediaWiki directory as a maintenance script. Typically this is done to test changes to maintenance scripts that cannot be done locally, or to avoid having to wait for a train deployment / doing a backport when running the fixed version of a maintenance script.
mwscript /home/foo/scriptname.php --wiki=enwiki
will run /home/foo/scriptname.php with the MediaWiki root directory set to whichever directory enwiki is served from (so all other files and classes referenced in the script will be loaded from their standard location).
Use this responsibly. The privilege policy applies here as well: you should not run non-trivial code in production that wasn't reviewed by at least one other person. Make sure that, if something goes wrong, others can reconstruct exactly what code was executed (e.g. by saying !log T12345 running mwscript /home/foo/scriptname.php --wiki=enwiki on IRC).
== History ==
=== Former services ===
* noc.wikimedia.org: As of mid-2023, the mediawiki-noc web app is now served from the "mw-misc" cluster in MediaWiki On Kubernetes.
=== Hosts ===
* Eqiad data center: terbium (2013), mwmaint1001 (2018), mwmaint1002 (2018).
* Codfw data center: wasat (2017), mwmaint2001 (2018), mwmaint2002 (2021).
== See also ==
* Deployment server
* Debugging in production#Debugging databases
* Manual:Maintenance scripts on mediawiki.org
Category:MediaWiki production