Fundraising/Cluster/Deployments

== Release tags and branches ==
Please note that many repositories include checked-in external libraries. This should only be done on the deployment branches, the master branch is rebuilt by the developer. Do check in composer.lock and similar files for version locking.
Repository 	Deployment branch
crm 	deployment
mediawiki/core 	fundraising/REL1_39
DonationInterface	deployment
CentralNotice 	wmf_deploy
SmashPig 	deployment
tools 	deploy
dash 	deployment
DjangoBannerStats	master (on fran)
all others 	master or as submodules
== Deployment process ==
Fundraising cluster deployment is always done from frpm. Software is deployed using the custom fundraising_code_update and rsync_blaster scripts, configuration is deployed with frpm as the puppetmaster. More information is available about deployment on Collab.
Our deployment scripts simply pull down specific branches from gerrit and rsync them to our production servers. We never run composer nor any minification scripts in production. This means that we need to check in all our dependencies to our repositories, generally in a /vendor submodule under the repo with the dependencies.
For payments-wiki, we deploy the fundraising/REL1_39 branch (updated every two years to track the LTS branch of Mediawiki) of mediawiki/core, as well as the fundraising/REL1_39 branch of mediawiki/core/vendor. Our branch of mediawiki/core/vendor includes dependencies for Mediawiki core and our deployed extensions but no dev-dependencies. We do not deploy additional vendor subdirectories under each extension, as we make use of the composer-merge plugin to install all dependencies to the top-level vendor directory.
We generally update mediawiki/core/vendor one library at a time. For example, if we have tagged an updated version of SmashPig, the procedure to update vendor is as follows:
 # From the root mediawiki directory:
 composer update --no-dev wikimedia/smash-pig
 cd vendor
 git add .
 git commit -m "Update SmashPig"
 # send for review and merge the vendor commit in gerrit
 cd .. # back to the mediawiki root directory
 git add vendor composer.lock
 git commit -m "Update SmashPig library and vendor directory"
== Rolling back a deployment ==
To rollback the latest deployment you can use =rollback
fundraising_code_update -p payments-wiki=rollback
To then redeploy the rollbacked code or any new code you can use =head
fundraising_code_update -p payments-wiki=head
== Tests ==
Unit tests are run on master branches.
On deployment branches just linting is run.
e.g. Smashpig, only master do unit test, but deployment not
== Configuration ==
Change paymentswiki configuration files on the frack puppetmaster, frpm. Check out the settings repo (into your homedir on frpm, not your dev machine).
 developer@frpm:~$ git clone /var/lib/git/localsettings.git
Then inside the copy in your homedir, make changes and push to the shared repo. After pushing to the shared repo, you can ask another fr-tech member to review the change if there's anything tricksy. Then deploy the change using fundraising_code_update and rsync_blaster, targeting whichever project pertains to the config file you have just changed.
fr-tech doesn't have access to configure some subsystems directly - any settings outside of the /srv directory are managed by fr-tech-ops via puppet. You can clone the puppet repo down to your frpm homedir as well but you will have to ask an fr-tech-ops member to actually apply any changes.
=== Matching gifts employers list ===
The csv containing employer names and their Civi IDs is deployed alongside the LocalSettings.php configuration file for payments-wiki. You need to copy it from the civi box to the frpm box.
On frpm you can copy a script to update it from the local scripts repo in /var/lib/git.
 git clone /var/lib/git/tools.git/ bin
When you run this script, you will need to be ssh'ed into the puppetmaster with agent forwarding (ssh -A frpm) so that your user on frpm can access files on the civi box. Run ~/bin/updateemployer.sh from your local copy of the localsettings git repo. It will
* copy the file from where it is generated on Civi to payments-wiki/employers.csv
* commit it to git
* run fundraising_code_update and rsync_blaster for payments-wiki
== Tools ==
Update things like paypal audit file locally or export silverpop, then as we did for DI or smashpig, 
1: checkout to deploy branch (git checkout --track origin/deploy, then use git log --oneline --no-merges deploy..master --reverse | cat to get the change for commit), then pull from master (git merge master), then do the git review with commit for all stuff needs to get merged to deploy branch, example (remember do not push test folder and the additional testenv for flake 8 to deploy branch)
Then ssh to frpm
fundraising_code_update -p tools
# remember the revision #
fundraising_code_update -i
# check if the revision # correct
rsync_blaster ALL:tools
# to propagate this staged code to the individual servers
== Process-Control Jobs ==
Update one of the many yaml job files in frpm101:~/localsettings/process-control/civi1001/
Once you've done that check in the update using:
git add <file>.yaml
git commit
git push
It's always good to get someone else from fr-tech to review your change before you deploy it. When done, deploy the changes using: 
fundraising_code_update -p process-control
rsync_blaster ALL:process-control
== SmashPig ==
There are three ways to deploy SmashPig: standalone, in CiviCRM, and in Payments-wiki.
=== Deploying SmashPig Standalone ===
Deploy the standalone 'SmashPig' project when you need to make code changes to the IPN listener or Pending queue consumer, or when you want to change SmashPig-level config settings on all machines on the cluster.
* For code, merge the changes you want to deploy from the master branch to the deployment branch (git checkout --track origin/deployment, then git log --oneline --no-merges --reverse deployment..master to get the change list, copy them for commit message, then git merge master to get the recent master changes)
** If you have changed composer.lock on master, update the vendor submodule with updated dependencies and update the pointer on development (similar with the civi smashpig update). 
** You can update vendor in its own commit or just amend the merge commit to include the vendor update. Be sure not to check in dev-dependencies.
* For settings changes, make your changes in the SmashPig/local-config subdirectory of your local checkout of localsettings, commit, and push, as with any other config changes.
* On frpm, use fundraising_code_update and rsync_blaster with the "SmashPig" project as a target
=== Updating the SmashPig Composer Package (needs for DI and CiviCrm) ===
Under CiviCRM and DonationInterface, SmashPig is included as a composer package.
* We distribute SmashPig as a composer package wikimedia/smash-pig, managed at https://packagist.org/packages/wikimedia/smash-pig
* Version numbers are controlled by git tags using the semantic versioning numbering scheme (breaking changes increase the 1st digit, minor changes increase the second digit, bugfixes increase the 3rd digit).
* To bump the package version from e.g. 0.5.9 to 0.5.10:
1. Checkout master in your local copy of the SmashPig repo
2. Update the git tag
 git tag v0.5.10 HEAD
 git push origin v0.5.10
3. Go to https://packagist.org/packages/wikimedia/smash-pig
4. Click the green 'Update' button
=== Deploying SmashPig inside CiviCRM ===
 Step 1: Update the SmashPig composer package
1. Follow the instructions in the updating smashpig composer package section
Step 2: Update the version in crm/master 
should update wikimedia/donation-interface first (check the deploying SmashPig inside DI) then have both do and smash-pig update for composer.json
1. in crm/master edit the composer.json to point smash-pig to the new version
2. in crm/master, composer update --no-dev wikimedia/smash-pig
* if composer doesn't see the new version, try composer clear-cache
3. commit composer.json and composer.lock change to crm/master with message as "Upgrade SmashPig to v0.x.x"
4. git review and merge
Example commit
* With Donation Interface updated: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/823694
* WIthout DI update (Starts from march 3, 2023 with this patch, we no longer need to maintain DI when update smashpig in civi: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1008526
Step 3: Updating deployment and crm/vendor 
1. Check out crm/deployment or create it with git checkout -b deployment --track origin/deployment
2. get the log by use git log --oneline --no-merges --reverse deployment..master | cat
then git merge master to merge master into deployment, if any conflict related with unit test files, remove the file as original deployment branch
3: then from deployment branch, to ensure vendor update are merged to crm/vendor rm -rf vendorgit submodule update --init vendor
then See general vendor update notes, basically just like below
cd vendorgit checkout master cd ../composer install --no-devcd vendorgit add . git commit -m "Update SmashPig"git review Example commit https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/972875/
4. go back to wmff deployment branch
cd ../ 
Add the updated submodule reference for vendor in crm/deployment
git add .
5. git commit --amend squash it for all composer update and vendor update
6.git review and merge (check the vendor hash the same with the latest commit and self Verified +2 and submit for this to merge)
Example commit https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/823698
===Deploying SmashPig in Donation Interface ===
This differs from updating in CRM, in that
Step 1: Update the SmashPig composer package
* we do not specify the smash-pig requirement in the root mediawiki composer.json but rather in extensions/DonationInterface/composer.json. The composer merge plugin pulls in requirements from the extensions and installs them to the mediawiki/vendor directory.
* We only maintain one branch of mediawiki for payments, fundraising/REL1_39 (or whichever RELX_YY represents the latest LTS). So for most updates there is no 'merge to deployment' step
* If you are drastically changing the version requirement (e.g. from 1.x to 2.x) you will need to change that in composer.json in the DonationInterface repo, merge that change to DI's deployment branch, and update the DonationInterface submodule pointer as well as all the steps below. If you only want to deploy a minor update, you can just update the composer.lock file and vendor submodule in the top-level payments-wiki source checkout.
Example commit: :
Updating the SmashPig version in DI, can be standalone or when new features are added
With a feature: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/768752
Standalone: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/777486
Step 2: Update smash-pig in vendor
# Ensure your vendor directory is consistent with the latest changes on mediawiki/vendor's fundraising/REL1_39 branch
## Delete the vendor directory from the root of your payments-wiki source (the mediawiki repo at branch fundraising/REL1_39)
## Pull down the pristine copy git submodule update --init vendorremember after init vendor, make sure the vendor folder is also on branch fundraising/REL1_39 as below
##checkout to the branch we want for vendor
cd vendor
git checkout fundraising/REL1_39
cd ../
# Update smash-pig using composer composer update --no-dev wikimedia/smash-pig
# Go into /vendor and (git remote -v should give you mediawiki/vendor) 
cd vendor
git add .
git commit -m "Update SmashPig"
git review
cd ../
 If you get an error about not being on a branch checkout fundraising/REL1_39 (or RELX_YY if a later release)
Example commit: Updating the vendor directory: https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/777492
Step 3: Update di composer.lock and vendor
# Make sure the vendor commit is merged in gerrit, then back in the root of your payments-wiki source: 
git add vendor composer.lock
// if DI version also get updated then add extra DI as the line below
git add extensions/DonationInterface
git commit -m 'Update DI, vendor and compsoer.lock for new smashpig version'
git review
Example commit: Updating the lock file and DonationInterface and vendor submodules: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/777490
Step 4: deploy
# Once above commit merged in gerrit, deploy the payments-wiki project from frpm
#* On frpm, run
fundraising_code_update -p payments-wiki
rsync_blaster ALL:payments-wiki
== Payments-wiki ==
=== Mediawiki security updates ===
You should be subscribed to the mediawiki-announce list to get pre-release announcements for Mediawiki security patches. When those are released, they are merged into the mediawiki release branches in gerrit.
To deploy them to payments-wiki:
* Merge the latest from the current LTS branch (e.g. REL1_39) into our branch (e.g. fundraising/REL1_39).
 git fetch
 git merge REL1_39
* When you merge, there will probably be some conflicts on submodules that have been updated in the remote and deleted in our local branch. Please resolve all of these in favor of the local branch where they are deleted, so we don't deploy any extra extensions.
* Remove vendor folder by run `rm -rf vendor`
* HINT: Make sure you have the latest smashpig version under your extension/DonationInterface composer.json
* Then update the vendor submodule. 
** If there are updates to composer.json in the merge, please run `composer update --no-dev <specific packages that have been updated>` to fold those in with our locally installed libraries, then commit them to our branch of the vendor submodule 
** if there are some newly added package inside composer.json, add them to composer.json, and run `rm composer.lock` and then run `composer install --no-dev`
* After composer update/install done, go inside vendor by `cd vendor`, and HINT: make sure your vendor branch is fundraising/REL1_39
* Use git to commit the merge and git-review to push it to gerrit, then C+2 your own patch.
example: https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/985012
* Then back to payment by `cd ../`
* Then git add, need to make sure composer.lock is included (after REL1_39 no vendor is going to be commit as example ) 
* Use git to commit the merge and git-review to push it to gerrit, then C+2 your own patch.
** HINT, can get the update log via 
git log --oneline --no-merges fundraising/REL1_39..origin/REL1_39 | tac | grep -v 'submodules'
* Once merged in gerrit, ssh into the deploy box (currently frpm1002) and use the following commands to deploy the fix:
 fundraising_code_update -p payments-wiki
 rsync_blaster ALL:payments-wiki
* !log the update in #wikimedia-fundraising
* As with any deploy, check that pages still load and check in the logs that donors can keep donating
We did a recording of part of this here: Recording
example: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/875409
== Donation Interface ==
=== Deploying to payments ===
Work In Progress, feel free to edit/add/rewrite
Hint: you can always mock other's most recent deploy process from Gerrit to see if everything looks smooth, and if Jenkins did not catch your commit, just send a comment "recheck". 
Workflow: submit patches to master branch, when deploying they go to deployment branch. Deployment branches don't have unit tests. 
Part 1: 
1. Make sure your master and deployment branches are up to date
2. Go onto the deployment branch (if you do not have development branch yet, just do 
git checkout -b deployment --track origin/deployment) and find the differences between deployment and master. The below command will output them in plain text: 
git log --oneline --no-merges --reverse deployment..master | cat
3. Copy the difference in commits between deployment and master as it would be used in another step.
4. Merge master into deployment git merge master
5. There may be merge conflicts in tests, they need to be removed from the deployment branch. You can remove them manually by using git rm [testname] or this could be done using the git mergetool command.
6. On the git mergetool interface, ensure the "(d)eleted file" option is selected for test files.
7. When finished, enter git commit
8. This brings you into a commit message, paste in the differences from step 2 underneath the Merge message
9. Push up the changes to gerrit git review
10. Once they are on gerrit, you can self +2 it if you feel comfortable with the code changes made. You can always ask others if you aren't sure what one of the patches is.
example: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/952977
Part 2: Submodule Update
As of mediawiki 1_39 if there are no vendor or composer.lock changes, you do not need to do Part 2 it will happen automatically. 
'''
This is needed as we deploy the entire payments project instead of just Donation Interface when changes are made.
1. In the main payments wiki folder, make sure you have the latest code at the right branch (you can always find the right branch from other's most recent deploy task). If you are running into issues and part 1 has been merged, you can do a git reset --hard
2. Add the updated Donation Interface extension git add extensions/DonationInterface This will change the pointer of the submodule.
3. git commit The commit message can be 'Update Donation Interface submodule'
4. Push up the changes to gerrit git review
5. Once they are on gerrit, you can self +2
=== Updating the Donation Interface composer package ===
Under CiviCRM Donation Interface is also included as a composer package (along with SmashPig)
We distribute Donation Interface as a composer package wikimedia/donation-interface, managed at https://packagist.org/packages/wikimedia/donation-interface
We use the same steps as in the updating SmashPig package
Version numbers are controlled by git tags using the semantic versioning numbering scheme (breaking changes increase the 1st digit, minor changes increase the second digit, bugfixes increase the 3rd digit).
To bump the package version from e.g. 0.5.9 to 0.5.10:
1. Checkout master in your local copy of the Donation Interface repo
2. Update the git tag
 git tag v0.5.10 HEAD
 git push origin v0.5.10
3. Go to https://packagist.org/packages/wikimedia/donation-interface
4. Click the green 'Update' button
== E-mail preference center ==
The email preference center is under the project donorwiki. 
Deploying is the same steps as deploying Donation Interface 
and then running fundraising_code_update with the project donorwiki
fundraising_code_update -p donorwiki
rsync_blaster ALL:donorwiki
== CentralNotice ==
Please see CentralNotice documentation on Wikitech.
== Civicrm ==
====== Disabling jobs ======
In some cases (eg. database update required) you will want to disable process control jobs when deploying. To do this
1. log onto frpm and go into localsettings repo
2. git pull to get up to date
3. go into the process-control directory
3a. (optional) you can list which jobs are enabled on which host using the 'listenabled' script, e.g.
 ./listenabled civi1002
4. run the script `disable` & commit the results, e.g. 
 ejegg@frpm1002:~/settings/process-control$ ./disable civi1002
5. deploy the change
6. once jobs are disabled you can check that they have stopped with run-job -rl
7. don't forget to reenable when finished
====== Preparing the codebase ======
1. Make sure your master and deployment branches are up to date
2. Go onto the deployment branch (if you do not have development branch yet, just do 
git checkout -b deployment --track origin/deployment) and find the differences between deployment and master. The below command will output them in plain text: 
git log --oneline --no-merges --reverse deployment..master | cat
3. Copy the difference in commits between deployment and master as it would be used in another step.
4. Merge master into deployment git merge master
5. There may be merge conflicts in tests, they need to be removed from the deployment branch. You can remove them manually by using git rm [testname] or this could be done using the git mergetool command
6. On the git mergetool interface, ensure the "(d)eleted file" option is selected for test files. (if unit test file, blank at deployment branch, accept left, no need to add to deployment branch)
7. When finished, enter git commit
8. This brings you into a commit message, paste in the differences from step 2 underneath the Merge message
9. Push up the changes to gerrit git review
10. Once they are on gerrit, you can self +2 it if you feel comfortable with the code changes made. You can always ask others if you aren't sure what one of the patches is.
11. The civi deployment branch sometimes does not automerge the code. You can check on previous deploy what happened, but you can manually merge the patch with the submit button on the top right.
12. don't forget to re-enable jobs after the deploy if you did disable them
====== Vendor updates ======
If there is an update to composer.lock you will also need to include a vendor submodule update in your commit. Vendor is the only remaining submodule in the crm codebase and
it exists in the deployment branch but not the master branch. Instructions below assume that you have not yet updated the vendor repo
1. if you already have a vendor folder in your crm root then reset your vendor repo back to upstream - ie 
cd vendor
git reset --hard origin/master
git pull
2. composer install --no-dev
3. cd vendor (run remote -v, should get wikimedia/fundraising/crm/vendor, and branch should checkout to master)
4. git add .
5. git commit
6. git review and self-merge
Example commit https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/823687
====== Updating the database ======
In some cases a CiviCRM code deploy will require a code update. Depending on the type of change you might need to issue a command on the command line
Type of change 	Command 	Notes
Drupal database update 	drush @wmff updb 	
CiviCRM version upgrade 	drush @wmff civicrm-upgrade-db 	Can be done in the UI at civicrm/upgrade
CIviCRM extension upgrade 	drush @wmff cvapi Extension.upgrade 	Can be done via the v3 api explorer or in the UI - the link will be visible from https://civicrm.wikimedia.org/civicrm/a/#/status
CiviCRM managed entity change	drush @wmff cvapi Managed.reconcile version=4 	Can be done via the apiv4 explorer. May be quicker if extension specified (see explorer)
Synchronize custom fields 	drush @wmff cvapi WMFConfig.syncCustomFields version=4	In some cases these might be stored as managed entities - but we use this command, in part for legacy reasons, but also because the field might exist on live already with different options & we are scared to do something weird
== Big Civicrm Upgrades ==
https://collab.wikimedia.org/wiki/Checklists_for_maintenance_%26_Recovery#CiviCRM_or_Drupal_outage_for_a_significant_upgrade