Phabricator/Deployment

== Deployment windows ==
Except for emergencies, Phabricator deployments are generally handled during mw:Phabricator/Maintenance windows. As of 2024, this is done during Collaboration Services office hours. Other windows may be scheduled, depending on deployer availability.
== GitLab deployment repository ==
Phabricator deployments are handled via scap, using configuration in the phabricator/deployment repo.
For minor changes you intend to deploy immediately, you can merge changes locally to the appropriate repository, bump any affected submodules, and push directly to the wmf/stable branch.
~/code/wmf/phabricator/deployment $ git pull --recurse-submodules
# Assuming an update to the phabricator submodule:
~/code/wmf/phabricator/deployment $ git submodule update
~/code/wmf/phabricator/deployment $ git add phabricator
~/code/wmf/phabricator/deployment $ git commit
~/code/wmf/phabricator/deployment $ git push
For larger sets of changes, you should create a Merge Request in GitLab which updates the submodules (example).
~/code/wmf/phabricator/deployment $ git pull --recurse-submodules
~/code/wmf/phabricator/deployment $ git checkout -b work/2023-11-09-release
# Assuming an update to the phabricator submodule:
~/code/wmf/phabricator/deployment $ git submodule update
~/code/wmf/phabricator/deployment $ git add phabricator
~/code/wmf/phabricator/deployment $ git commit
~/code/wmf/phabricator/deployment $ git push --set-upstream origin work/2023-11-09-release
Then follow the link to create a merge request.
When you're ready to deploy, merge this Merge Request via the GitLab UI ("Merge" button) to bump submodules.
Notes:
* As of October 2023, the AVA repository and the Translations repository are still in Gerrit instead of GitLab (cf. phab:T321380)
* When deploying CSS and JS changes, dependency and packaging maps are generated by bin/celerity map, which updates resources/celerity/map.php
* When removing or adding source code files, running Arcanist's /bin/arc liberate updates the library metadata.
== Downtime host alerts ==
If you can run cookbooks:
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1004.eqiad.wmnet
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org
If you can't run cookbooks:
* Downtime Icinga
** Go to https://icinga.wikimedia.org/icinga/ and search for "phab" in the sidebar.
** Select the following hosts:
*** phab.wmfusercontent.org
*** phab1004 (production in eqiad)
*** phab2002 (fallback in codfw)
** In the "Select command" dropdown, select "Schedule Downtime for Checked Host(s)"
* Silence Alerts
** Click the bell on https://alerts.wikimedia.org/
** Click "New Silence"
** Under "Label" select "Instance"; toggle "regex" to be enabled; in the text field add a new silence for each of the following:
*** ^(phab1004)(:[0-9]+)?$
*** ^(phab\.wmfusercontent\.org)(:[0-9]+)?$
*** ^(phabricator\.wikimedia\.org)(:[0-9]+)?$
** Ensure you've put something in the comment field at the bottom, "Phab deployment T123456" or whatever you're doing.
** In the "No Alerts matched" dialog, click the "Submit" button
** If the resulting dashboard URL is a 404, check/correct the domain name
== Deploy to fallback host ==
* ssh deploy2002.codfw.wmnet
* cd /srv/deployment/phabricator/deployment/
* git pull
* git submodule update
* scap deploy -l phab2002.codfw.wmnet 'test deploy to phab2002 for Txxxxxx' (replace with task ID; also logs to )
== Deploy to production host ==
* ssh deploy1002.eqiad.wmnet
* cd /srv/deployment/phabricator/deployment/
* git pull
* git submodule update
* scap deploy -l phab1004.eqiad.wmnet 'deploy to phab1004 for Txxxxxx' (replace with task ID; also logs to )
* In parallel, check for errors on the Logstash dashboard; or via ssh phab1004.eqiad.wmnet followed by tail -F /var/log/apache2/phabricator_error.log
* In parallel, check for the next minutes for any explosions on IRC 
* Optionally,
** systemctl status phd to make sure things are running
** tail /var/log/phd/daemon.log to check for daemon problems (noisy because of repo syncing errors)
== Clean up deployed Phabricator tasks in Phabricator ==
* Go into phab project and rename "next" milestone to the date of the deploy
** i.e. "2023-08-23"
** Remove the additional hashtag "#phab-next" from the milestone
** Paste generated markup with stuff from release (past example)
** archive this old milestone
** create a new "next" milestone
*** Set description to "A future deployment of Phabricator"
** Tag and resolve open tasks that got deployed and need no further work
== See also ==
* Scap#Other software deployments
* Setting up a local Phabricator test environment