nyuuzyou/wikis huggingface dataset:
Fundraising/techops/procedures/servers-standard software vulnerability management

==Operating System and Vendor Software Vulnerability Procedures==
===Purpose===
This procedure document applies to off-the-shelf software provided by Debian, mariadb.org, GrafanaLabs and other vendors installed on servers in the Fundraising environment. Bespoke and locally modified software is handled by FR Tech and following this procedure.
===Vulnerability Tracking===
FR Tech Ops SREs should monitor security mailing lists for vulnerability announcements for software we use. Relevant lists include:
* Debian: debian-security-announce@lists.debian.org
* MariaDB: announce@lists.mariadb.org
* Mediawiki, pybal, general: security@wikimedia.org
* CiviCRM: security@civicrm.org
* General: Bugtraq, US-CERT, fulldisclosure, oss-security
====Automated Tracking====
The pxeboot-preseed-puppet server build process configures apt repositories and modifies the package selection appropriate for the hardware type and server role. In cases where non-stock packages are used, puppet configures the relevant apt repository and uses apt-pin to ensure that only the required packages are installable from the non-Debian repository. Thus for all installed deb packages, standard package management tools show vendor updates as soon as they become available for install.
Most Fundraising servers run unattended-upgrades daily and are configured to apply patches automatically, except for cases where a package update could cause an outage.
Fundraising servers also run package_update_check daily, which uses a simulated apt dist-upgrade to check for available updates. This script notifies SREs if intervention is required. Fundraising Tech Ops SREs monitor this mailing list daily for maintenance tasks.
===Vulnerability Management===
For security updates, FR Tech Ops should evaluate the vulnerability based on the risk ranking provided the software vendor according to WMFs (draft) Security Policy - Vulnerability Matrix.
When a patch can be applied without significant service disruption, it is typically done automatically within a day by unattended-upgrades. Otherwise FR Tech Ops is notified daily by each server with outdated packages requiring intervention.
Any update that can be done without significant interruption should be performed as soon as possible, preferably the same day the notification is first received. In this case, no additional tracking is required.
If a patch requires significant service disruption, or in cases where other mitigation is required, a private Phabricator task should be opened to document the vulnerability, risk, and technical assessment, and to plan and track followup actions.
In all cases, FR Tech Ops is required to apply patches and mitigations within the time period specified in WMFs (draft) Security Policy - Vulnerability Matrix for the assessed risk level. This policy has shorter timelines than PCI DSS.
The preferred method to apply patches and updates is to run risky_package_updater on each server. This script automates apt and puppet steps, and also checks for required service restarts.
Fundraising/techops/procedures/services-backups

Fundraising Cluster Backups
role 	directory / file 	expiration	notes 	transport
frban	/srv/banner_logs/{year} 	auto/1y 	proxy banner logs 	rsync
frlog	/srv/archive/civi/audit 	n/a 	civicrm donation audit data 	archive_sync
frlog	/srv/archive/civi/process-control 	auto/1y 	queue consumer run logs 	archive_sync
frlog	/srv/archive/civi/tools.git 	n/a 	code backup 	archive_sync
frlog	/srv/archive/database_dumps 	auto/5y 	fr analytics database dumps 	archive_sync
frlog	/srv/archive/fran/backups 	auto/1y 	code/conf backup 	offhost_backup
frlog	/srv/archive/fran/process-control 	auto/90d 	analytics script run logs 	archive_sync
frlog	/srv/archive/frav/database_dumps 	auto/5y 	IDS system database dump 	archive_sync
frlog	/srv/archive/frav/logs 	auto/5y 	sitewide auditd logs 	archive_sync
frlog	/srv/archive/frav/tools.git 	n/a 	code backup 	archive_sync
frlog	/srv/archive/frlog*/logs 	auto/5y 	sitewide logs 	archive_sync
frlog	/srv/archive/frlog*/tools.git 	n/a 	code backup 	archive_sync
frlog	/srv/archive/frmon/grafana 	auto/1y 	monitoring data backup 	archive_sync
frlog	/srv/archive/frnetmon 	auto/5y 	code/config backup, paymentswiki db	offhost_backup
frlog	/srv/archive/frpm 	auto/5y 	code/config backup, paymentswiki db	offhost_backup
frlog	/srv/archive/permanent 	(tbd) 	 	manual
frlog	/srv/snapshots/frav1001.frack.eqiad.wmnet 	auto/5y 	remote code/config backup 	make_snapshot
frlog	/srv/snapshots/frdev1001.frack.eqiad.wmnet	auto/5y 	remote code/config backup 	make_snapshot
frmon	/srv/archive/frmon 	n/a 	remote monitoring data backup 	rsync
frmon	/srv/snapshots 	auto/90d 	local+remote monitoring data backup	make_snapshot
frpm 	/srv/snapshots 	auto/180d 	local code/config backup 	make_snapshot
Fundraising/techops/procedures/services-civicrm database trigger updates

= mysql Trigger uploads for CiviCRM =
CiviCRM relies heavily on mysql triggers to perform complex query functions. Occasionally these need to be updated or have new triggers added. The database trigger updates are run by the FR Tech Ops personnel in conjunction with FR Tech.
== Requirements in place before an upload ==
Before we can do the upload we need to ensure there are a few things in place.
* Approved triggers staged in gerrit
* Access to the database origin server including shell and mysql access. Make sure you have a .my.cnf file or user/password information for db access
* Access to the "failover" database host in order to stop the replication thread
== Prep work for the upload process ==
===Get the trigger scripts +2'd and pushed to production===
===Capture the state of the old triggers===
mysqldump -uroot --routines --no-create-info --no-data --no-create-db --skip-opt --add-drop-trigger civicrm > triggers.before.$(date +%Y%m%d)
===Copy generated trigger scripts from the active civicrm application server to the frdb origin server===
* Fetch the scripts from the civicrm server
rsync -var --rsh=ssh civi1002:/srv/org.wikimedia.civicrm/drupal/sites/default/civicrm/extensions/wmf-civicrm/sql/ scripts.$(date +%Y%m%d)/
* Push the trigger scripts to the frdb origin server (frdb1005):
rsync -var --rsh=ssh scripts.$(date +%Y%m%d) frdb1005:
==Running the upload process==
===Stop queues===
Depending on the update, this may not be required.
===Stop replication on one frdb replica server===
mysql "stop replica"
===Run the trigger update script on the frdb origin server===
mysql civicrm < scripts.$(date +%Y%m%d)/file_with_trigger_updates.mysql
==Verify the newly installed triggers==
====Capture the trigger state====
mysqldump -uroot --routines --no-create-info --no-data --no-create-db --skip-opt --add-drop-trigger civicrm > triggers.after.$(date +%Y%m%d)
===Verify the line count===
* In some cases the line count will be the same if it's just updating existing triggers
* If triggers are added or deleted, the line count should differ
# wc -l triggers.before.$(date +%Y%m%d) triggers.after.$(date +%Y%m%d)
===diff the triggers (choose one)===
You should see a difference in the before and after trigger information.
* Verify the diff of the mysql triggers using the holy grail for diffs
git diff -U0 --word-diff=color --no-index -- triggers.before.$(date +%Y%m%d) triggers.after.$(date +%Y%m%d) | less -R
* Use a more standard version
diff triggers.before.$(date +%Y%m%d) triggers.after.$(date +%Y%m%d)
* You may need to use vimdiff to actually see where the differences are, this may entail scrolling way to the right to find the actual update.
 # vimdiff triggers.before.$(date +%Y%m%d) triggers.after.$(date +%Y%m%d)
==Confirming the upload process==
* Verify with the FR Tech member responsible for the update that the new behavior is in place and performing as expected
==Post update actions==
* Restart the replication thread process on the frdb replica server
* Restart the queues if stopped
Fundraising/techops/procedures/services-civicrm staging db refresh

Every quarter or so, we will want to refresh the staging databases. To do so we use the following steps:
# Pull the latest database backups for the civicrm, drupal, and fredge databases from frlog1002 to the current frdb_staging host placing them in the /srv/db_restore directory.
# cd /srv/db_restore
# Run /usr/local/bin/staging_db_refresh in a screen session answering the prompts.
# Monitor the output of the log file. It will take approximately 3 days to complete the selective restore as of 2023-11-16.
Fundraising/techops/procedures/services-kerberos kdc install

==Services: Kerberos KDC Master Setup Procedure==
These are notes from the kerberos server setup circa approximately the beginning of time. It is likely the specifics are outdated.
===Create a New Kerberos Realm===
root@frauth1001:/etc/krb5kdc# krb5_newrealm
This script should be run on the master KDC/admin server to initialize
a Kerberos realm. It will ask you to type in a master key password.
This password will be used to generate a key that is stored in
/etc/krb5kdc/stash. You should try to remember this password, but it
is much more important that it be a strong password than that it be
remembered. However, if you lose the password and /etc/krb5kdc/stash,
you cannot decrypt your Kerberos database.
Loading random data
Initializing database '/var/lib/krb5kdc/principal' for realm 'WMF-FUNDRAISING',
master key name 'K/M@WMF-FUNDRAISING'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:
 * Starting Kerberos KDC krb5kdc [ OK ]
 * Starting Kerberos administrative servers kadmind [ OK ]
Now that your realm is set up you may wish to create an administrative
principal using the addprinc subcommand of the kadmin.local program.
Then, this principal can be added to /etc/krb5kdc/kadm5.acl so that
you can use the kadmin program on other computers. Kerberos admin
principals usually belong to a single user and end in /admin. For
example, if jruser is a Kerberos administrator, then in addition to
the normal jruser principal, a jruser/admin principal should be
created.
Don't forget to set up DNS information so your clients can find your
KDC and admin servers. Doing so is documented in the administration
guide.
===Add an Administrative User, Configure Realm Policies, Additional Users===
root@frauth1001:/etc/krb5kdc# kadmin -p root/admin
Authenticating as principal root/admin with password.
kadmin: Client not found in Kerberos database while initializing kadmin interface
root@frauth1001:/etc/krb5kdc# kadmin.local
Authenticating as principal root/admin@WMF-FUNDRAISING with password.
kadmin.local: add_policy -minlength 12 -minclasses 4 -maxlife "90 days" -history 10 admin
kadmin.local: add_policy -minlength 12 -minclasses 4 host
kadmin.local: add_policy -minlength 12 -minclasses 4 service
kadmin.local: add_policy -minlength 12 -minclasses 4 -maxlife "90 days" -history 10 user
kadmin.local: addprinc -policy admin jgreen/admin
Enter password for principal "jgreen/admin@WMF-FUNDRAISING": 
Re-enter password for principal "jgreen/admin@WMF-FUNDRAISING": 
Principal "jgreen/admin@WMF-FUNDRAISING" created.
kadmin.local: addprinc -policy user jgreen
Enter password for principal "jgreen@WMF-FUNDRAISING": 
Re-enter password for principal "jgreen@WMF-FUNDRAISING": 
Principal "jgreen@WMF-FUNDRAISING" created.
kadmin.local: quit
Fundraising/techops/procedures/services-kerberos new replica setup

==Services: Kerberos New KDC Replica Setup==
Here are the steps to add a new Kerberos host into the replication chain.
===First, create a host key for new Kerberos replica host===
* On Origin:
 bash# kadmin.local
 kadmin.local: addprinc -randkey host/frauth1002.frack.eqiad.wmnet@WMF-FUNDRAISING
===The next step is to extract these keys to the keytab file.===
The keytab file is a keyring which contains the cryptographic keys needed to authenticate with the KDC.
Extraction of keys to the keytab is done with the ktadd sub command:
* On Origin:
 kadmin.local: ktadd host/frauth1002.frack.eqiad.wmnet@WMF-FUNDRAISING
 kadmin.local: exit
===Next we need to copy the keytab to the new host that is being set up.===
* On Replica:
 cd /etc/
 nc -l -p 88 | tar xvBpf -
 sha256sum krb5.keytab
* On Origin:
 cd /etc/
 sha256sum krb5.keytab
 tar cvBpf - krb5.keytab | nc -q 100 frauth1002 88
===Then we can run the krb5_propagate script to do the regular propagation.===
* On Origin:
 /usr/local/bin/krb5_propagate
===Not done yet. For the service to start on the replica, there are a few more files that need to be copied across.===
The stash file:
* On Replica:
 cd /etc/krb5kdc/
 nc -l -p 88 | tar xvBpf -
 sha256sum /etc/krb5kdc/stash
* On Origin:
 cd /etc/krb5kdc/
 sha256sum /etc/krb5kdc/stash
 tar cvBpf - stash | nc -q 100 frauth1002 88
The principal file:
* On Replica:
 cd /var/lib/krb5kdc/
 nc -l -p 88 | tar xvBpf -
 sha256sum /var/lib/krb5kdc/principal
* On Origin:
 cd /var/lib/krb5kdc/
 sha256sum /var/lib/krb5kdc/principal
 tar cvBpf - principal | nc -q 100 frauth1002 88
===You should now be able to start the kdc admin processes on the replica.===
* On Replica:
 sudo systemctl restart krb5-kdc.service
 kadmin.local
 kadmin.local: list_principals
Fundraising/techops/procedures/services-kerberos user management

=Users: Kerberos User Management=
==To create a new user==
===Use kadmin.local as root===
 sudo su -
 kadmin.local
===Show existing users===
 list_principals
===Add user with username matching (frack) puppet/hieradata/user/users.yaml===
* for administrative users:
 addprinc -policy admin jgreen/admin
 {enter a temp password}
* for non-administrative users:
 addprinc -policy user jgreen
 {enter a temp password}
===Set the new password to expire after the first use===
* for administrative users:
 modify_principal +needchange jgreen/admin
* for non-administrative users:
 modify_principal +needchange jgreen
===Provide the user with their temporary password and tell them to change it via 'passwd' command on any frack host===
==To change a password for an existing user==
 cpw jgreen
 modify_principal +needchange jgreen
Fundraising/techops/procedures/services-mariadb primary swap

This is a procedure that can be used to swap a replica maria DB in as a new primary. In the future, this will hopefully be replaced by an automated procedure.
== Procedure ==
All Commands are listed in a manner to start with a comment character to protect from accidental copy/paste.
Key:
 Commands starting with # are run on the CLI
 Commands starting with -- are run in mysql
===Move site to downtime/maintenance mode===
===On current primary===
====flush tables to get consistency - in mariadb/mysql====
-- FLUSH TABLES WITH READ LOCK;
====check all replicas to make sure they have the same Relay_Master_Log_File and Exec_Master_Log_Pos====
-- SHOW SLAVE STATUS \G
====stop the db====
 # systemctl stop mariadb.service
====ensure the db doesn't start on boot====
 # systemctl disable mariadb.service
===On new primary=== 
====Stop replication - in mariadb/mysql====
 -- stop slave;
 -- reset slave all;
====Turn off read only====
 -- SET @@global.read_only=0;
====Reset 'master' - in mariadb/mysql====
 -- reset master;
====Get 'master' position====
 -- SHOW MASTER STATUS \G
===On all the other replicas ===
====Stop replication from old primary DB====
 -- stop slave;
==== Point replication at new host ====
If using GTID replication (can use replica_pos after 10.5.1):
 -- change master to MASTER_HOST = 'new_host.fqdn',
 MASTER_USE_GTID = slave_pos, 
 MASTER_USER='repl', MASTER_PASSWORD='XXXXXXXX',
 Master_SSL = 1, MASTER_SSL_CA = '/etc/mysql/cacert.pem',
 MASTER_SSL_CERT = , MASTER_SSL_KEY = , MASTER_SSL_VERIFY_SERVER_CERT = 1;
If using binlog replication using the File: and Position:
 -- change master to MASTER_HOST = 'new_host.fqdn',
 MASTER_LOG_FILE = '$FILE', MASTER_LOG_POS = $POSITION,
 MASTER_USER='repl', MASTER_PASSWORD='XXXXXXXX',
 Master_SSL = 1, MASTER_SSL_CA = '/etc/mysql/cacert.pem',
 MASTER_SSL_CERT = , MASTER_SSL_KEY = , MASTER_SSL_VERIFY_SERVER_CERT = 1;
====Start replication on new primary DB====
 -- start slave;
===Update DNS handles to point to new primary host===
===On old primary db=== 
====Turn on read only as a safety measure====
 -- SET @@global.read_only=1;
====Reset 'master'====
 -- reset master;
====Reset 'slave' knowledge====
 -- reset slave all;
==== Point replication at new host ====
If using GTID replication (can use replica_pos after 10.5.1):
 -- change master to MASTER_HOST = 'new_host.fqdn',
 MASTER_USE_GTID = slave_pos, 
 MASTER_USER='repl', MASTER_PASSWORD='XXXXXXXX',
 Master_SSL = 1, MASTER_SSL_CA = '/etc/mysql/cacert.pem',
 MASTER_SSL_CERT = , MASTER_SSL_KEY = , MASTER_SSL_VERIFY_SERVER_CERT = 1;
If using binlog replication using the File: and Position:
 -- change master to MASTER_HOST = 'new_host.fqdn',
 MASTER_LOG_FILE = '$FILE', MASTER_LOG_POS = $POSITION,
 MASTER_USER='repl', MASTER_PASSWORD='XXXXXXXX',
 Master_SSL = 1, MASTER_SSL_CA = '/etc/mysql/cacert.pem',
 MASTER_SSL_CERT = , MASTER_SSL_KEY = , MASTER_SSL_VERIFY_SERVER_CERT = 1;
====Start replication on new primary DB====
 -- start slave;
Fundraising/techops/procedures/services-payments-settings

This page describes how we manage settings across the payments cluster.
=LocalSettings Repo=
This repo contains config files for payments-wiki, CiviCRM, process-control, and SmashPig. Both fr-tech and ops can make changes here. It is kept in git to track changes.
On frpm1002, clone /var/lib/git/localsettings.git to your home directory. When you want to make changes
* git pull to make sure your copy is up to date
* edit the files
* commit the change to git
* git push
* use fundraising_code_update to copy the updated settings into your target project's rsync source dir.
** e.g. if you have pushed an edit to payments-wiki/LocalSettings.php, you must then use fundraising_code_update -p payments-wiki to copy the file to /srv/www/org/wikimedia/payments
* use rsync_blaster to sync the files from frpm1002 to all the target machines.
** e.g. rsync_blaster ALL:payments-wiki
=SmashPig=
SmashPig configuration is divided between a number of files. Global configuration, including queue and database connections, are in /etc/smashpig/main.yaml, and are managed by puppet (only ops can push changes). Provider configuration files are in the localsettings repo under SmashPig/local-config/.
SmashPig settings are shared across all projects that use the SmashPig library as well as the standalone SmashPig IPN listeners and maintenance scripts.
Fundraising/techops/procedures/services-redis-frqueue

This page documents to the 'frqueue' redis service which handles contribution tracking messages and donor preference change messages. This service runs on the 'frqueue' server role in the Fundraising server cluster.
==Overview==
* We run redis basic replication
* There are two redis instances running as separate services
** redis-frqueue (tcp/6379)
** redis-donor-prefs (tcp/6380)
* The redis-frqueue instance uses key "sequence_contribution-tracking" as a sequence generator for contribution tracking.
* When switching origin servers, shut down the old origin server before making replication changes to be sure it doesn't receive any new messages after replication is changed.
* You can invoke redis commands from a shell using redis-cli, or you can log in via redis-cli and run commands within in a client.
==Origin Swap==
===Confirm that all client services (paymentswiki, civicrm queue consumers, smashpig, etc) are shut down.===
===Obtain the redis masterauth passwords from /etc/redis/*.conf===
===If the old redis origin server is dead, verify the new origin server has the latest Contribution ID.===
* The last ID imported by queue consumers can be obtained from the drupal database.
SELECT id,contribution_id FROM drupal.contribution_tracking WHERE contribution_id IS NOT NULL ORDER BY id DESC LIMIT 10;
* Use redis-client to compare the autoincrement counter to the latest contribution ID.
# redis-cli
127.0.0.1:6379> AUTH <password>
127.0.0.1:6379> GET sequence_contribution-tracking
127.0.0.1:6379> exit
===Problems===
* If replication died in some wierd way, it's possible different replicas have different autoincrement values. In this case choose the one with the greatest value as the new origin.
* If contribution ID from the database is higher than the redis autoincrement, there is a risk of corrupting the database if you don't adjust the redis autoincrement. FR Tech should be consulted for how to proceed.
===Promote new origin server for redis-frqueue===
* On the new queue server stop replication, retaining dataset, become origin. Check if things look ok.
# redis-cli
127.0.0.1:6379> AUTH <password>
127.0.0.1:6379> REPLICAOF NO ONE
127.0.0.1:6379> INFO
127.0.0.1:6379> GET sequence_contribution-tracking
127.0.0.1:6379> exit
# redis-cli -p 6380
127.0.0.1:6380> AUTH <password>
127.0.0.1:6380> REPLICAOF NO ONE
127.0.0.1:6380> INFO
127.0.0.1:6380> exit
===On each replica, change origins (discarding dataset) and resynchronize, check status.===
# redis-cli
127.0.0.1:6380> AUTH <password>
127.0.0.1:6379> REPLICAOF <new redis origin FQDN> 6379
127.0.0.1:6379> INFO
127.0.0.1:6379> exit
# redis-cli -p 6380
127.0.0.1:6380> AUTH <password>
127.0.0.1:6380> REPLICAOF <new redis origin FQDN> 6380
127.0.0.1:6380> INFO
127.0.0.1:6380> exit
===Adjust redis-related configuration in puppet.===
This adjusts puppetized clients including civicrm, payments-listener, and sets redis replication configuration to survive a service restart.
* edit (frack) puppet/hieradata/site/common.yaml
queue_redis_primary: '<new redis origin FQDN>'
* git commit, git push, puppet-merge
Fundraising Analytics

How fundraising analytics works.
==Data Mining==
Banner and landing page impressions are recorded by udp2log. These logs are formatted according to the Wikimedia squid log format.
===Banner Impressions===
Although a lot of information is recorded by the squid logs, not all of that information is useful for fundraising analytics and is not written to a database. Banner impression stats are not intended to and should not be traceable back to individual users. They are used in aggregate to help determine the creative direction of the fundraiser.
==Data Cleaning==
==Data API==
===Normalization===
===SQL Queries===
==Confidence Testing== 
==Web Hosting and Reporting==
==Implementation==
Category:Fundraising
Category:Fundraising_Analytics
Category:Fundraising - Needs Updated
Fundraising Logging

The fundraising team utilizes the udp2log data stream to count banner impressions and click throughs.
Currently we have the following files
* bannerImpressions log 
** 1/1 recording of all banner impressions with extra data tracking added for our mining scripts
* landingpages log 
** 1/1 recording for a click throughs on banners (generally all traffic on foundation wiki, which is later mined for the appropriate information)
Because of the large size of these files, they are shipped to other storage machines (storage3) for compression and processing, rather than staying on the log machine (locke). They typically stay on the log machine for aggregation for about an hour. This process is controlled by a cron script, and is shipped over via scp (encrypted due to privacy concerns) and is shipped by the dummy user 'file_mover'.
These logs are then post processed by rfaulkner's scripts and fed into a stats portal. Post fundraiser analytics are being compiled and logged at collabwiki as well. Documentation of the mapping of these log requests to analytics are further described in Ryan Faulkner's Style Guide Notes.
Category: Fundraising
Category:Fundraising - Needs Updated