nyuuzyou/wikis huggingface dataset:
Fundraising/tech/crm/Upgrade CiviCRM

__TOC__
== Creating a CiviCRM upgrade patch ==
This process is for tracking upstream security patches.
This first step should be done in your development clone of wikimedia/fundraising/crm.
 # Prepare the Civi codebase by checking out our current upstream tracking head.
 git submodule update -i
 cd civicrm
 git fetch --all
 git checkout origin/contrib
 
 # Get and unpack the new release
 VERS=4.2.17 # for example
 wget http://downloads.sourceforge.net/project/civicrm/civicrm-stable/$VERS/civicrm-$VERS-drupal.tar.gz
 tar xzf civicrm-$VERS-drupal.tar.gz
 mv civicrm civicrm-$VERS
 
 # It's best to wipe out any deprecated files, as we do here,
 # but you can omit the delete/exclude stanzas if you'd like.
 # NOTE that you will lose any uncommitted development files as well.
 rsync -av \
 --delete --exclude '.git*' --exclude "civicrm-$VERS*" \
 civicrm-$VERS/ ./
 
 # Get your working stuff out of here
 mv civicrm-$VERS* ..
 # Toss pinch of salt and add everything you find.
 git add -A
 git commit -m "Upgrade to CiviCRM $VERS upstream"
 
 # Locally review the crazy
 git log -p -1 -w
* Push the drupal contrib commit for review. DO NOT SELF-MERGE, even if it seems like an emergency.
 git push gerrit HEAD:refs/for/contrib
* Merge to trunk
 git checkout origin/master
 git merge -m "Merge contrib" contrib
 # ...or you can explicitly give the treeish of your local contrib commit.
 
 # Do stupid thing to run commit hook and satisfy git-review
 git commit --amend
* Push the commit for review.
* In the top-level crm directory, update the submodule pointer.
 cd ..
 git checkout -t origin/master
 git add civicrm
 git commit -m "Update civicrm submodule to $VERS"
* Push the submodule bump for review.
* Prepare a merge to deployment and submit for review. The four review steps can be prepared in parallel, fwiw.
* If you have any doubts about code integrity, deploy to the staging box and test. It's important that WMF's custom patches still work, and determining this can be very tricky. TODO: unit tests for all Civi-core customizations.
== Deploying a CiviCRM upgrade ==
Please note that your particular upgrade may require additional precautions or other action, see the CiviCRM release notes for details.
* Shut down all jenkins jobs which depend on the CRM. Just turn 'em all off if you're feeling lazy.
* Kill the lights:
 # Make the site inaccessible
 drush vset site_offline 1; drush vset maintenance_mode 1
 
 # Clear caches
 drush cc all
* Verify database backups are current.
* Deploy new wikimedia/fundraising/crm code.
* Beyond this point, know that you cannot recover safely without doing a full database restore...
 # Update module schemas
 drush updatedb
 
 # Update CiviCRM schema
 drush civicrm-upgrade-db
 
 # Make site accessible
 drush vset site_offline 0; drush vset maintenance_mode 0
* Turn jobs on again, slowly and carefully.
== See also ==
Category:Fundraising
Fundraising/tech/crm/Upgrade Drupal

__TOC__
== Creating a Drupal upgrade patch ==
* cd to your development clone of wikimedia/fundraising/crm.
 # Get and unpack the new release
 VERS=7.27 # for example
 wget http://ftp.drupal.org/files/projects/drupal-$VERS.tar.gz
 tar xzf drupal-$VERS.tar.gz
 
 # We're first operating on the upstream tracking branch.
 cd drupal
 git fetch --all; git checkout -t origin/contrib
 # It's best to wipe out any deprecated files, as we do here,
 # but you can omit the delete/exclude stanzas if you'd like.
 rsync -av \
 --delete --exclude '.git*' --exclude 'sites' \
 ../drupal-$VERS/ ./
 
 # Toss pinch of salt and add everything you find.
 git add -A
 git commit -m "Upgrade to Drupal $VERS upstream"
* Push the drupal contrib commit for review. DO NOT SELF-MERGE, even if it seems like an emergency.
* Merge to trunk
 git checkout -t origin/master
 git merge -m "Merge contrib" contrib
 
 # Do stupid thing to satisfy git-review
 git commit --amend
* Push the drupal master commit for review.
* In the top-level crm directory, update the submodule pointer.
 cd ..
 git checkout -t origin/master
 git add drupal
 git commit -m "Update drupal submodule to $VERS"
* Push the submodule bump for review. The three review steps can be prepared in parallel, fwiw.
* If there are any doubts, deploy to the staging box and test.
== Deploying a Drupal upgrade ==
Please note that your particular upgrade may require additional precautions or other action, see the Drupal release notes for details.
* Shut down all jenkins jobs which depend on the CRM. Just turn 'em all off if you're feeling lazy.
* Kill the lights:
 # Make the site inaccessible
 drush vset site_offline 1; drush vset maintenance_mode 1
 
 # Clear caches
 drush cc all
 
 # Disable modules. Please save this list to your notepad.
 MODULES=`drush pml --type=module --status=enabled --no-core --pipe`
 echo "Disabling modules $MODULES"
 drush dis $MODULES
* Verify database backups are current.
* Deploy new wikimedia/fundraising/crm code.
* Make any necessary changes to settings.php and civicrm.settings.php — these are not under the same source control.
* Start doing naughty things you cannot recover from without a database restore...
 # Update Drupal core schema
 drush updatedb
 
 # Enable modules
 drush en $MODULES
 
 # Update module schemas
 drush updatedb
 
 # Make site accessible
 drush vset site_offline 0; drush vset maintenance_mode 0
* Turn jobs on again, slowly and carefully.
== CiviCRM upgrade ==
Read the CiviCRM release notes.
TODO:
* Port from old notes
* Howto create patches and upstream things.
== See also ==
Category: Fundraising
Fundraising/tech/deployments

== Deployments ==
This document contains links to instructions on how to deploy projects for fundraising.
=== Staged ===
These deployments have not been released.
minFraud upgrade to 1.3 - third week of May 2012 by User:Khorn
=== Released ===
These deployments have already been released.
* Upgrading CiviCRM to 4.1 - deployed March 2012 by User:Khorn
* Deploying Recurring GlobalCollect - deployed April 19th, 2012 by User:Khorn
Category:Fundraising
Category:Fundraising - Needs Updated
Fundraising/tech/git

== Fundraising and Git ==
As of April 2012, fundraising is using both Subversion and Git to manage repositories.
=== Working with Git ===
You will want to have a look at the Mediawiki [[mw:Git/Workflow|Git Workflow] to understand how to use git and [[mw:Gerrit Gerrit]
DonationInterface] is under git.
==== Checking out branches ====
One of the strengths of git is best shown in branching.
In the following example, you will see how to create a branch to work on Fundraising 2012 Mingle ticket #2
This examples assumes you have already set up git review. If you have not done so, please see Git Workflow.
cd extensions/DonationInterface
Make sure you are up to date and create a branch based on the Mingle ticket
git pull origin master
git checkout -b mingle-fr-2012-2 master
You can check to see which branches are available and which branch you are currently using with the command:
git branch -a
You should see something like:
 list
 master
* mingle-fr-2012-2
 show
 remotes/origin/HEAD -> origin/master
 remotes/origin/deploy-payments_1.17
 remotes/origin/master
This shows that we are using the branch we just created for the Mingle ticket.
If you are creating a branch to address a Bugzilla ticket, create a branch in this format:
git checkout -b bug25090 master
==== Check your status to see what you need to commit ====
git status
==== Preparing your commit ====
Add the file to the commit.
git add some_file_you_edited
==== Commit your code ====
See [[mw:Git/Workflow#Make and commit your change|make and commit your change]
git commit
If you have files that you do not want to commit, you can hide them with:
git stash
Check your status:
git status
Untracked files will stay put.
You can then see what has been stashed:
git stash list
stash@{0}: WIP on mingle-fr-2012-140: 5b11178 Adding Selenium IDE test for supplemental_address.
==== Rebase your branch ====
You can now rebase with master to push your commit to review:
git pull origin master
git rebase master
Run review
git review
Category:Fundraising
Fundraising/tech/payments1004

= Using payments1004 for integration tests =
payments1004 is a depooled production payments server. It is a good place to do a "live test".
== Access ==
To access it with a web browser you must tunnel to port 443 over ssh. The annoying part is it is a privileged port so it has to be done as root. The magic incantation is:
 sudo ssh -o ProxyCommand='ssh -i /path/to/ssh/key -W %h:%p user@frbast.wikimedia.org' -L 443:localhost:443 user@payments1004
You will have to provide the ssh key passphrase and two yubikey presses, or you can use -i in the outer command to use key based auth locally on payments1004. Now you have an interactive shell on the server and the tunnel will be open until you log out. You could also do -N (no remote shell) and -f (background ssh process) to leave the tunnel open indefinitely.
While it is possible to use an unprivileged port and change the urls, anything hard-coded breaks, e.g. payment process return urls.
== Override DNS ==
To make your laptop (on the fwd'd port) answer to the name payments.wikimedia.org, add this line to /etc/hosts:
 127.0.0.1 payments.wikimedia.org
Now visit payments and your traffic should be hitting 1004. You can confirm the right thing is happening by checking the web logs for the appropriate payment gateway on the log server.
Fundraising/techops

This page is for Fundraising Tech Operations policy and procedure documentation.
(We are actively consolidating and migrating documentation here from other places, including collab wiki.)
==Help Docs==
* Fundraising Access Requests
* Fundraising SSH Access
* Fundraising Cluster Software Deployment
* Reporting Security Bugs
* SSL Client Certificate Authentication
==Policy==
* Fundraising Server Policy
* Fundraising Server Build Standard
==Procedures==
===Server Administration===
* Server Procurement & Install
* Out-of-Band Management
* Server Roles
* Server Role - Build
* Server Decommissioning
* Server Software Vulnerability Management
===Service Administration===
* CiviCRM Database Trigger Updates
* CiviCRM Staging Database Refresh
* Datacenter Cutover Procedure
* Fundraising Backups
* Kerberos KDC Install & Setup
* Kerberos KDC New Replica Setup
* Kerberos User Management
* MariaDB Clone
* MariaDB Mariabackup
* MariaDB Primary Swap
* MariaDB Timezone Update
* Nessus Exceptions - see git repo fundraising_system_documentation/nessus/exceptions.txt
* Redis - frqueue
* Payments Cluster Settings
* System Authentication
===User Administration===
* Fundraising Tech Managed Accounts
* Fundraising Access Roles (draft)
* New User Checklist
* SSL Client Certs - Yearly Renewal
* Departing User Offboarding Checklist
Fundraising/techops/docs/fundraising access request

== Access Request ==
Before any access can be granted to Fundraising systems, three things need to be completed:
* a phabricator task to track the request
* c-level approval
* an entry on the contact list
===Phabricator task===
An access request needs to be opened in phabricator with a fundraising-tech-ops tag on it. In the request you should list the following things:
*User requiring access
*Type of access required (ssh access, database access, civi access, superset access)
*Any specific hosts that access will be needed to
Here is a template to create an access request.
Access request template
Title 	Fundraising access request for PERSON
Tags 	fundraising-tech-ops
Description	
This is a new access request for PERSON. They require the following access: (mark each box with an x)
[ ] civicrm web access
 [ ] standard access
 [ ] donor services access
[ ] ssh access - if specific hosts: `list here`
[ ] mysql - if specific hosts or databases: `list here`
[ ] superset
[ ] other: `please explain`
===C-Level approval===
Additionally, approval will need to be granted from the C-level exec. This request can be made by the manager of the user or fr-tech-ops can make the request. This request should include the services and accounts the request is for. Example:
To: CLevel Person (cperson@wikimedia.org)
CC: ${requesting_manager}
Hello CLevel,
${new_user} is a new fundraising analyst reporting to ${requesting_manager}. Can you provide authorization for ${new_user} to access ${services}?
Thanks,
===Contact List Entry===
Finally, the user will need to have an entry on the Fundraising Contact list. This entry will need a valid phone number that can receive SMS messages. Additionally, we will need a rough geographical location for the user so we can populate a client ssl certificate properly.
Fundraising/techops/docs/fundraising cluster deploy

==Fundraising Cluster Deploy==
Once your code is merged in gerrit, and any localsettings configuration is pushed to the shared repository, follow the steps below to deploy to fundraising servers.
#Shut down the thing you are updating, if this is a substantial change.
# Log in to the active fundraising deploy server
#;ssh frpm1002
#Run fundraising_code_update to refresh the deploy server's staging areaGenerally you'll want to update one project at a time as in this example, although other options are possible.
#;fundraising_code_update -p civicrm
#Confirm the reported commit changes match what you expect, you can inspect the contents of the project staging directory which is listed by
#;fundraising_code_update -i
#Run rsync_blaster to propagate this staged code to the individual serversThis example deploys to all servers where the civicrm project is installed, but it is also possible to sync specific servers i.e. for a canary deploy.
#;rsync_blaster ALL:civicrm
#Log action into #wikimedia-fundraising by typing the below (where xyz & abc are the commits) into IRC
#; !log Updating civicrm from xyz to abc
#Turn your stuff back on and make sure it works.
=== fundraising_code_update===
fundraising_code_update is a CLI tool run on the fundraising deployment server, which preps the local staging area for one or more project. It fetches the desired version of the software from version control (gerrit) including submodules, tweaks file attributes, overlays configuration files from the private localsettings repository, and keeps track of current and previous revision IDs to make it easy to roll back to the previous release. The staging area is owned by the configured deploy user and writable only by that user.
See -h help for more information on how fundraising_code_update can be used.
===rsync_blaster===
rsync_blaster is a CLI tool run on the fundraising deployment server which uses rsync to deploy whatever is in the staging area to the target hosts. It will optionally run a remote 'service-restart' command after the rsync. It allows fine-grained control over deployment allowing you to do canary- or other phased deploys. Deployment status reports are available at a cluster (-s) or target (-t <host>) level, showing the time and commits IDs last deployed.
See the -h help page for usage examples.
Fundraising/techops/procedures/servers-decommissioning

==Fundraising Server Decommissioning==
This procedure is intended to dovetail with Server_Lifecycle and Phabricator Server Decommissioning Request form, but since Fundraising servers are in a separate environment many of the steps on the Phabricator form do not apply.
===Server Decommissioning Procedure===
* create a Phabricator task using the Server Decommissioning Request form
* migrate all activity and data off the server
* remove the server from monitoring by merging an update to (production) puppet/modules/icinga/templates/nsca_frack.cfg.erb
* remove the server from (frack) puppet
* power off the server via management interface
* remove the server from DNS
* update Netbox server Status from "active" to "decommissioning"
* log into cumin and run the appropriate cookbook
sudo secure-cookbook sre.dns.netbox -t <phab task> "Remove host <fqdn> from DNS for decommissioning"
* update the Phabricator decommissioning task and reassign it according to the form instructions
Fundraising/techops/procedures/servers-out of band management access

==Server Out-of-Band Management-Access==
Every Fundraising server has an out-of-band management interface. The interface is connected to a network switch, which is connected to the PFW firewall/router and configured as an isolated management subnet. The management network is accessible via tunneled SSH connection through the Fundraising bastion servers.
===Connecting by SSH===
Make sure your ~/.ssh/config has Host config to route the ssh connection through the bastion server, and to set the correct user.
Host *.mgmt.frack.*.wmnet
 User root
 ProxyCommand /usr/bin/ssh -q -W %h:%p frbast.wikimedia.org
Then:
ssh {host}.mgmt.frack.{dc}.wmnet
Password: <first password is bastion SSH 2FA>
Password: <second password is management password>
===Dell DRAC Useful Commands/Operations===
* Display Hardware Information and Configuration
 racadm racdump
* Display System Information (including host IPs)
 racadm getsysinfo
* Control Server Power
 racadm serveraction <action>
 possible actions include powerup, powerdown, and powercycle
* Change DRAC password (one or the other, depending on DRAC version)
 racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 <thepassword>
 racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 1 <thepassword>
 racadm set iDRAC.Users.2.Password <thepassword>
* Set to PXE boot on the next restart (newer hosts):
 racadm set iDRAC.ServerBoot.FirstBootDevice PXE
* Set to PXE boot on the next restart (older hosts):
 racadm config -g cfgServerInfo -o cfgServerBootOnce 1
 racadm config -g cfgServerInfo -o cfgServerFirstBootDevice PXE
* Configure Network Interface (usually handled by dcops):
 racadm setniccfg -s 10.64.40.199 255.255.255.192 10.64.40.193
* Command and Hotkeys to Enter/Exit console or BIOS:
 enter console: console com2
 exit console: ^\
 enter bios: <esc>2
 start pxeboot: ^<esc><2>
* Fix Post-Boot Console Redirection:
 System BIOS Settings > Serial Communication > Redirection After Boot <Enabled>
Fundraising/techops/procedures/servers-role-build

== Special notes when building out a host in the build role ==
 
Sometimes things can't be fully automated with puppet. A large portion of these steps were found while setting up frpm2001 in T242269 and T247133-
Here are some of the modifications needed on this build:
=== SSH ===
# Create root ssh keys
# Update authorized keys in git for git user on build role
# Update authorized keys in git for puppet user on build role
# Verify host keys for commonly made connections on new and existing build hosts
=== Hiera ===
Requires #SSH
# Update common.yaml to specify the host as a puppetmaster
# Add host specific hieradata/site/$host.yaml file where puppet master overrides will occur
=== Puppet ===
Requires #SSH and #Hiera
# Run puppet-merge for base and private repos
# Update modules/role/files/build/etc/motd.tail to reference new host as a puppetmaster
=== GPG ===
# Import gpg keys from another build host (frpm1002/frpm2001)
# Update gpg.conf file to contain fr_tech_ops group
=== MariaDB ===
# Update mysql grants to allow new host to connect to payments db
# Add /root/.payments-my.cnf with the correct info so the host can back up the payments db
=== Backups ===
Requires #SSH and #GPG
# Update authorized keys in git for backupmover user in the default role
# Update modules/role/manifests/logger.pp for backup definition
# Update modules/fundraising/templates/archive_sync.erb for backup sync
# Update modules/fundraising/templates/archive_purge.erb for backup purging
=== Code deploy ===
# Create ssh key for mwdeploy user
# Update authorized keys in git for mwdeploy user in the default role
Fundraising/techops/procedures/servers-roles

To see current information on which hosts are assigned to which roles see (frack) puppet/hieradata/hosts.yaml
Role 	Description
analytics 	analytics application, data cube scripting - superset, jupyter notebook
auth 	central authentication - freeradius, mit kerberos
backup 	backups - log/data archive
banner_logger 	kafkatee and log storage
bastion 	ssh bastion
build 	puppetmaster, apt repo, code propagation, proxy, dhcp/tftp build environment
civicrm 	civicrm application, queue consumer job runner
frdata 	donor preferences wiki, donor data delivery
frdb_analytics 	mariadb analytics data store
frdb_staging 	mariadb dev/staging database
frmx 	output mailserver - postfix
fundraising_database	mariadb civicrm/drupal database
fundraising_qa 	application staging server, reporting
logger 	central logging, log/data archive - rsyslog
monitoring 	performance metrics - prometheus, grafana
network_security 	nessus scanner
pay_lvs 	payments load balancer - pybal/LVS
payments_listener 	smashpig payment API
payments 	payments application
queue 	contribution message and donor prefs messages queues
seim 	aide/aiderator, audit logger
Fundraising/techops/procedures/servers-server policy

==Fundraising Server Policy==
Maintainer: Fundraising Tech Operations
Created: 2023-07-21
Review Required by: 2024-07-25
===Purpose===
This policy applies to servers in the Fundraising environment, which are installed, secured, and maintained by FR Tech Ops.
===Server Role and Function===
A server should handle one primary function.
If multiple primary functions exist on the same server, one of the following must apply:
* Primary functions differing security levels are isolated from each other.
* Primary functions with differing security levels are secured to the level required by the function with the highest security need.
===Server Operating System===
Fundraising servers are required to run an industry standard operating system meeting the Infrastructure Foundations Operating System Upgrade Policy.
===Additional Software===
Only necessary software should be installed on any server.
Stock software packages provided and maintained by the OS vendor are preferred.
Non-stock vendor software packages may be used if the version available directly from the vendor is better suited for the application in our environment.
Locally-built packages may be used if no suitable off-the-shelf vendor package is available.
Custom software may be used if it is better suited for the application in our environment.
===Software Updates===
Operating system and other software updates and patches should be applied as soon as possible after they become available.
Security mitigations and patches must be applied within timeframes specified by WMF (draft) Security Policy - Vulnerability Matrix.
===Services and Protocols===
Only necessary services, protocols, daemons, and functions may be enabled, with all unnecessary functionality removed or disabled.
No insecure protocols may be enabled unless a business justification is documented, and additional security features must be documented and implemented to reduce risk.
===Users and Administrative Access===
Only necessary accounts may be enabled on servers, extraneous accounts must be removed or disabled.
Vendor default passwords must be removed or changed.
All non-console administrative access must be encrypted using strong cryptography and must employ multi-factor authentication.
===Security Services===
A host firewall must be enabled and configured appropriately for the server role.
Host intrusion detection must be enabled and configured appropriately for the server role for security-sensitive roles.
Servers in the PCI Cardholder Data Environment must employ an up-to-date industry standard virus scanner.
Audit logging must be enabled per WMF Security Logging and Alerting Policy, with off-host logging to Fundraising central logger service and audit log collectors.
Servers must be configured to remain synchronized to network time.
===Secure Decommissioning===
Server disks and other storage media must be securely erased or destroyed before leaving the secure datacenter environment.