nyuuzyou/wikis huggingface dataset:
External storage/Evaluating innodb

==Notes from Asher (2012)==
I recently completed some benchmarks comparing the existing myisam based external store against an all innodb version of the same. Comparative tests were run on es1001 for myisam and es1003 for innodb.
The downside for innodb is less efficient disk space utilization. The sum of all revisions for all wikis grew from 4.3TB to 7.3TB. I tested the compact as well as the newer dynamic and compressed row formats and stuck with compact. Dynamic improves blob storage under certain schema conditions but used slightly more space for this. Compressed offered savings of just a few percent over compact, with the overhead of compressing and uncompressing pages of data the have already been gzip compressed by mediawiki. We currently have a single shard of ES servers with 10.8TB of usable disk space per node, so the additional space overhead shouldn't be an issue so long as a new shard is deployed in the coming months. A second hardware shard has been ordered which should be online in 2-3 weeks.
Benchmarks were run with read queries in the style of "SELECT blob_text from blobs_clusterNN WHERE blob_id = NNNNN" against the enwiki blobs_cluster22 and blobs_cluster23 tables (well over 1TB of data) where blob id's were generated as random integers between 1 and max(blob_id), in groups of 100k queries. I then ran variable numbers of concurrent query threads, with each thread using a different set of 100k random blob_id's. That was then repeated over 10 runs, with the first started directly after server boot. For the last run, I set long_query_time = 0 in order to capture timing data for every query. Replication was active during all tests for a steady trickle of concurrent writes.
When running with six concurrent read threads, the innodb system completed 6M random queries at an average rate of 440.92/sec while the original got 308.656/sec and finished the test around 90 minutes later. For the first run of 600k after a reboot with cold caches, the innodb system did 387.096/sec vs 286.53/sec for myisam.
Percentiles times for the last 600k queries, in seconds:
Percentile MyISAM InnoDB
50 0.014336 0.000083
95 0.036203 0.008352
99 0.053055 0.020705
99.99 0.199825 0.069474
This should dispel as myth the idea that myisam is faster in this particular use case. I think the disk space penalty is a worthy sacrifice in exchange for safer storage and crash recovery alone.
- Asher
Category:MySQL
External storage/Maintenance

How to safety perform maintenance on external storage boxes...
ES boxes are Apaches running an instance of MySQL for some simple blob storage. A couple of things you need to know:
# They come in clusters.
#: You can check /h/w/php-1.5/db.php for cluster<->machine assignments
# Each cluster has a master and one or more slaves.
#: The master is the first to appear in the cluster's list in db.php... but for some perverse reason is usually the highest-numbered server (eg, srv146 master, 145 and 144 slaves)
#: It should always be safe to take down a slave for maintenance.
#: For older, read-only clusters, taking the master down is also safe. Reads will fail-over to the slaves.
# Only the last couple clusters are active for writes.
#: These are listed in $wgDefaultExternalStore at the end of db.php
#: If you're going to shut down the master of one of these clusters, you should remove it from $wgDefaultExternalStore temporarily, otherwise some page saves will fail while it's down.
# If MySQL doesn't automatically start when you reboot the machine, punch it manually!
#:/etc/init.d/mysqld start should usually do it.
In rarer cases it may be necessary to fix replication or re-clone databases to replace a dead slave. These exercises are left to the reader.
Category:How-To
Fenari

fenari was a Dell/PowerEdge R300 in Tampa running Ubuntu 10.04. It was decommissioned after the migration to Eqiad and the splitting of Fenari's roles.
== Former services ==
* bastion: replaced by bast1001.
*deployment: replaced by tin.
*maintenance scripts: replaced by terbium.
*test.wikipedia.org: replaced by mw1017. 
* noc.wikimedia.org: replaced by tin.
*logmsgbot: replaced by neon.
FlaggedRevs setup

== General FlaggedRev installation instructions ==
# Add the extension's tables to your wiki...
#* cd /home/wikipedia/common/php-1.17/ 
#* php maintenance/sql.php --wiki=dbname extensions/FlaggedRevs/FlaggedRevs.sql
# Add the wiki to the FlaggedRevs group so it'll use it...
#* Edit /home/wikipedia/common/flaggedrevs.dblist and add the database name
# If necessary, customize the settings for it in flaggedrevs.php
#* Note: currently this sucks not as much as it did previously.
# Update live configuration:
#* touch InitialiseSettings.php to invalidate the cache
#* sync-common-all
# Note: Do not run this until the sync has completed. 
#* Run cd /home/wikipedia/common/php-1.17/ && php extensions/FlaggedRevs/maintenance/updateAutoPromote.php <dbname> to autopromote those stated in the config options.
== enwiki instructions ==
Rollout checklists:
* /enwiki 2010-06-15
* /2010-11-17 Rollout Checklist
== Open requests ==
on bugzilla
Category:How-To
FlaggedRevs setup/2010-11-17 Rollout Checklist

Here's the broad outline for the plan to roll out FlaggedRevs/Pending Changes to enwiki:
== Timeline ==
* Wednesday, 2010-11-17
** 3pm PST - Status meeting - last chance to abort
** 3:15pm PST - Roll out FlaggedRevs
*** Check in new version of extension to 1.16wmf4 branch
*** scap
** 3:45pm PST onward - Monitor the various wikis currently running FlaggedRevs for problems
==Removal steps==
Only use these instructions if something goes wrong
*Revert 1.16wmf4 branch to prior release
== See also ==
* Pending Changes enwiki trial page on mediawiki.org
FlaggedRevs setup/enwiki 2010-06-15

Here's the broad outline for the plan to roll out FlaggedRevs/Pending Changes to enwiki:
* Monday, 2010-06-14
** 4pm PDT - Roll out FlaggedRevsAlpha to existing FlaggedRevs users
** Monitor the various wikis currently running FlaggedRevs for problems
* Tuesday, 2010-06-15
** All day at least until 1pm PDT - Monitor the various wikis currently running FlaggedRevs for problems
** 1pm PDT - Go/no-go decision on enwiki rollout
** 4pm PDT - Roll out to enwiki
==Monday: Roll out FlaggedRevsAlpha to existing FlaggedRevs users==
# Remove all labs hacks, configure everything use non-alpha
# SVN move FlaggedRevs_alpha over FlaggedRevs
# Checkout /extensions on fenari. Sync.
# Check for bugs/problems on any FR wikis.
# Revert this change if needed.
==Tuesday: Roll out to enwiki ==
# Add an entry for enwiki in wmf-config/flaggedrevs.php with the same settings as the flaggedrevs_labswikimedia one.
# Add the extension's tables to enwiki...
#* cd /home/wikipedia/common/php-1.5/ 
#* php maintenance/sql.php --wiki=enwiki extensions/FlaggedRevs/FlaggedRevs.sql
# Add the wiki to the FlaggedRevs group so it'll use it...
#* Add enwiki to /home/wikipedia/common/flaggedrevs.dblist
# Update live configuration:
#* touch InitialiseSettings.php to invalidate the cache
#* sync-common-all
Post-deploy checklist:
* Ensure permissions are configured correctly
* Ensure 2000 article limit is in place
==Removal steps==
Only use these instructions if something goes wrong
#Set $wgFlaggedRevsProtectQuota = 0;. This disallows new flag-protections.
#In FlaggedRevs/maintenance, run php flagToSemiProtect.php enwiki <enwiki admin name> <reason>. This will semi-protect all flag-protected pages. The flaggedpage_config table data is *still there* so the pages are also left flag-protected.
#In FlaggedRevs/maintenance, run php purgeReviewablePages.php enwiki --makelist. This makes the list of flag-protected pages to clear for the squid cache.
#Remove enwiki from flaggedrevs.dblist
#Touch InitialiseSettings.php to make the dblist change apply. This should disable the extension and therefore no pages are flag-protected. Confirm that FlaggedRevs is disabled on enwiki.
#In FlaggedRevs/maintenance, run php purgeReviewablePages.php enwiki --purgelist. This clears the squid cache for those formerly flag-protected pages.
#SQL tables and flaggedrevs.php file dealt with only for long-term removal.
Flow

This is a How-To guide on how to deal with installation and removal of Flow on the Wikis.
== Installing Flow on a wiki ==
Do not do this. Flow is not eligible for deployment to more wikis.
== Uninstalling Flow from a wiki ==
* Ensure that all Flow boards are either transwikied or exported to wikitext (with ConvertToText) and deleted.
* Remove the wiki from flow.dblist (e.g. like this).
* Delete all pages in the Topic namespace:
** Add $wgExtraNamespaces[2600] = 'Topic'; to the config (only locally on the deployment machine, don't sync this)
** Add $wgContentHandlers["flow-board"] = $wgContentHandlers["wikitext"]; to the config (only locally on the deployment machine, don't sync this)
** Run SELECT CONCAT('Topic:', page_title) AS page_title FROM page WHERE page_namespace=2600; and put the list of pages in a file
** Run mwscript deleteBatch.php --wiki=<dbname;ie: eswiki> -u='<your_sysadmin_username>' -r='Uninstalling Flow' list-of-topic-pages.txt
** Remove the added config.
== Known problems ==
* : Summary of known problems on uninstalling Flow on a wiki where it has been used at least one time.
Flow
Fluorine

fluorine is a Dell/PowerEdge R310 and in eqiad in row A-A4. It has been installed per RT:2350 to become a deployment host.
It used to be used for collecting server logs, (api.log, exception.log, fatal.log, etc.) in /a/mw-log.
fluorine WAS a Wikimedia MediaWiki log collector (role::logging:mediawiki).
fluorine WAS a Wikimedia udp2log data collection server (udp2log::logger).
== Services ==
* Destination of MediaWiki logs.
== Deprecation ==
For the current log collector, please use mwlog1001.
Foundry Crossover Procedure

Lets figure out what steps we want to take in order to integrate the Foundry switch.
Here is a starting point:
* Connect up all access switches and routers to the Foundry.
* Move srv1-30 and db's onto the first patch.
* Do away with the netgear.
* Move any other servers we want on the Foundry from csw1 to the second patch.
* Disconnect the fiber from csw1 and route a new fiber to the foundry.
* Once the first fiber uplink is established, do the same with the other one.
* Use remaining cisco's as access switches and backup routers.
Foundry FastIron GS FGS648P

:Write this up.
== Setup ==
 hostname hostname
 ip dns domain-name mgmt.pmtpa.wmnet
 ip dns server-address resolver1 resolver2
=== Authentication ===
 enable super-user-password password
 username root password password
 aaa authentication login default local enable
Enable SSH:
 crypto key generate
Disable telnet, web management, tftp:
 no telnet server
 no web-management
 tftp disable
Configure the IP address first: 
 ip address ip 255.255.0.0
=== VLANs ===
For each:
 vlan nr name name
For the management vlan:
 vlan 3 name management
 management-vlan
 default-gateway 10.1.0.1 1
=== Logging ===
 logging host 10.0.0.1
 logging facility local6
=== SNMP ===
 snmp-server community community ro
 snmp-server contact Wikimedia Admins <noc@wikimedia.org>
 snmp-server location Rack racknr, PowerMedium, Tampa, Florida, USA
=== SNTP ===
 sntp server dobson.wikimedia.org
=== Spanning-tree ===
We don't use spanning-tree, disable it for all VLANs by default:
 no global-stp
=== Multicast ===
Access switches should be passively snooping for IGMP:
 ip multicast passive
=== RANCID ===
RANCID monitors and backs up changes of the switch configuration.
Category:Network
Fundraising/Annual PCI form

Wikimedia is at the SAQ-A level of PCI compliance, meaning we collect all card donations via third party processors and do not let any full card numbers touch our servers. Many of the PCI rules are simply good security practices, such as not using shared accounts.
Each year we have to fill out a form and submit it to Worldline (formerly Ingenico) as well as keep it on file in case any of our payment processing partners request it. TODO: Which other processors require us to upload it every year?
The form is available in various formats via the Document Library section of pcisecuritystandards.org. In the UI as of February 2022, one selects SAQs in the 'Filter by' dropdown, and the third row in the results should be the SAQ-A. It can be downloaded in PDF or DOC format. The PDF is not a fillable form, so it's generally easier to use the DOC. For inspiration answering the questions one can look to the previous year's SAQ-A, stored on the fileserver at fundraising/Tech/PCI. It's best to confirm datacenter locations and security policies with fr-tech-ops (currently Dallas Wisehaupt and Jeff Green), and to confirm the list of payment processing partners with the Payments team (currently Evelyn Martin and Pats Pena).
Once the document is complete, send it along to the Advancement C-Level (currently Lisa Gruwell) and the Administrative Specialist who works with her (currently Leticia Navarro) for a signature. Upload the signed document to the aforementioned folder on the fileserver and send it to Worldline. As of 2022, Worldline has established their own site for merchants to submit PCI forms at globalcollect.worldline-pciportal.com. Back when the part of Worldline we work with was known as GlobalCollect and then Ingenico, they had us upload it to a service called SecureTrust maintained by Trustwave, so we still may receive emails from SecureTrust.
Fundraising/Cluster/Civicrm staging server

== Overview ==
The staging server is part of fr-techs fundraising cluster. The database on it is manually updated from live every few months. The server is configured for some ticket in code review. The quick and effective to see whether the pr is working.
== Deploying to staging ==
You can pull the commit on to frdev1001. From the gerrit page for your change, click the Download dropdown in the top right, select "anonymous http" in the panel that appears, and copy the second command, labeled "Cherry Pick". 
Then on frdev1001, paste it into the staging CiviCRM directory, e.g.
 /srv/org.wikimedia.civicrm
* then pasted the command copied from gerrit, someething like below
git fetch "https://gerrit.wikimedia.org/r/a/wikimedia/fundraising/crm/civicrm" refs/changes/24/513024/1 && git cherry-pick FETCH_HEAD
*After pull the pr from gerrit, run `drush @wmff cvapi system.flush` to flush system cache.
*After that we can run any drush cmd to do the testing from recent pr for example
*if want to go to the database, run `drush @wmff cvsqlc`, this can get you go into the staging env mysql as 'dev_civicrm'
*or run `mysql` will login to basic mysql, run 'use civicrm;' to go inside civicrm;
*then you can make comparison with the civicrm and the dev_civicrm to see how staging works for the new pr you pulled in.
Fundraising/Cluster/DjangoBannerStats

A legacy python codebase that ingresses data about banner impressions and Donate Wiki landing page impressions to the pgehres database. For details, see Stats pipeline documentation.
= Repository =
DjangoBannerStats
= Testing locally =
There is a --django option in setup.sh that you can run to get everything going. It will create the dev_pgehres database.
remote banner logs are located xxx@fran1001:/srv/archive/banner_logs
= Deploying =
It's deployed to fran (currently fran1001) and has it's own project there DjangoBannerStats
Fundraising/Cluster/Payments staging server

== Overview ==
The payments-staging server is part of fr-techs fundraising cluster. This server is used to live-test changes to payments processing before final deploy to production. 
This server is only accessible with a WMF Fundraising SSL client certificate. Otherwise it has the same access to payment provider gateways, and behaves like the production payments servers in terms of wiki content, message queues, and logging. We have discussed being able to use separate queues and wiki content, but that has not been implemented so far. 
== Deploying to staging ==
The fundraising code deployment tools deploy two payments-related projects to this server, "payments-wiki" is the exact same code version as production, and "payments-wiki-staging" is the same branch but can be set to a different commit ID. Deployed project corresponds to the hostname in the web request, https://payments-staging.wikimedia.org serves the "payments-wiki-staging" project, and https://payments.wikimedia.org (accessible by changing your local /etc/hosts file) serves the regular "payments-wiki" project. 
This allows you to merge a new commit in gerrit, and do final testing before deploying to production.
To update just the staging codebase, on the deploy server you would run "fundraising_code_update -p payments-wiki-staging" and then "rsync_blaster payments::ALL".
Fundraising/Cluster/Redis

= Overview =
= Performance Tracking (Formally on main page) =
In addition to using graphana the command to monitor Redis cache misses is below - but note you can avoid putting the password on the command line by using `auth` once 'in' redis. You then type `monitor` as a separate command 
 
redis-cli -a 'password' monitor
The password is in civicrm.settings.php as CIVICRM_DB_CACHE_PASSWORD
Fundraising/Cluster/Upgrades

= Payments =
== Security Update ==
What do we do for these
== Upgrading to a new LTS version ==
This happens every 2 years as we move onto the new LTS mediawiki version
January 2025 we moved from 1.39 to 1.43
=== 1. Make a new branch ===
Branches are here
https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/DonationInterface,branches
Create new at the top right
[is there anything else specific we need to do here]
There is a standard set of changes that need to be done to start a new branch
#Delete unused extensions and skins
#Add WMF logos https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1117986
#Add Payments wiki extensions
#Update vendor for merged fundraising dependencies
=== 2. Test locally ===
=== 3. Get CI upgraded ===
[ was this actually related to the bookworm upgrade and not 143? need to confirm]
=== 4. Get tests passing on CI ===
=== 5. Deploying ===
# SRE changing fundraising_code_update to look at the new branch
# Doing the upgrade on staging
# Deploying to staging
rsync_blaster payments1008:payments-wiki-staging` but it's just as easy and clean to do ALL
# Testing on staging
# Doing the upgrade on production
# Testing on production
= Civi =
= other things =
= from mediawiki last updated june 2020=
== How to apply a security update ==
* fetch the upstream LTS release with e.g. 'git fetch origin REL1_31'
** If you are on vagrant, you may have to edit your .gitconfig to allow fetching other branches - just deleting the 'fetch' section seems to work
* look at the changes with git diff origin/REL1_31 or git log -p HEAD..origin/REL1_31
* get a clean changelog with git log --oneline --reverse --no-merges HEAD..origin/REL1_31 | cat
* do the merge: git merge origin/REL1_31
* resolve any conflicts. Note that the fundraising version of the LTS release has images added and has a different set of submodules, so be sure the merge does not delete images or add or delete submodules.
* test locally!
* review, merge to fundraising/REL1_31 and deploy.
= Upgrade scenarios =
== Schema upgrade required ==
* Confirm a recent database backup.
* Disable campaigns.
* Disable paymentswiki using LocalSettings variable.
* Update the code to the new release.
* Enable writability on the master wiki database (payments1001)
* Login to payments1001, cd to the wiki root directory, and run "php maintenance/run.php update".
* Reenable paymentswiki and test.
* Reenable campaigns.
* Watch logs for 1 hr.
== Risky, but code-only changes ==
* Disable campaigns.
* Disable paymentswiki using LocalSettings variable.
* Update the code to the new release.
* Reenable paymentswiki and test.
* Reenable campaigns.
* Watch logs for 1 hr.
== Safe, code-only changes ==
* Update the code to the new release.
* Watch logs for 1 hr.
Fundraising/ContributionTracking

ContributionTracking refers to donation metadata saved to a table which currently resides not in the fundraising civicrm database, but in the drupal database for the host drupal instance. The placement of this table in the drupal database is clearly very silly, and as such it needs to be moved elsewhere.
Payments-wiki formerly used a direct DB connection to the backend to insert new rows and get the autoincrement ID to use as the contribution_tracking_id. That DB connection was managed by an extension called ContributionTracking. The ContributionTracking extension no longer exists, and ID generation has been moved to a sequence generator in Redis, to fully decouple the payments-wiki front end from the back-end database.
UPDATE: The ContributionTracking table has been moved from drupal to the civicrm database. The table's name in the civicrm database is civicrm_contribution_tracking. The ContributionTracking entity controls the DB connection from CiviCRM and is being managed by the wmf-civicrm CiviCRM extension.
= Table Columns =
column name 	description
id 	
contribution_id 	
amount 	
currency 	
usd_amount 	
is_recurring 	
referrer 	
utm_medium 	
utm_campaign 	
utm_key 	
gateway 	
appeal 	
payments_form_variant	
banner 	
landing_page 	
payment_method_id 	
payment_submethod_id 	
language 	
country 	
tracking_date 	
os 	
os_version 	
browser_version 	
recurring_choice_id 	
device_type_id 	
banner_size_id 	
is_test_variant 	
banner_variant 	
is_pay_fee 	
mailing_identifier 	
utm_source 	
banner_history_log_id
Fundraising/Data and flow

[WIP] older docs at Fundraising#F Data and flow.
The Data and flow section of Fundraising Tech documentation explains data structures and processes involving multiple components in our stack. On this page, you'll find general overviews; see sub-pages under Fundraising/Data and flow (also linked below) for details.
The most important categories of processes are: donation, recurring charge, audit and e-mail export. There are also many processes, both automated and manually triggered, that don't fit into any of these categories.
== Donation ==
Donation involves interaction with donors via a UI. Donors provide payment details or credentials, and authorize a one-time donation and/or a monthly subscription. A successful donation flow concludes with donor information stored in CiviCRM and a thank-you e-mail sent to the donor.
We have three mutually exclusive donation flow options: one-time, recurring, and monthly convert.
Every implemented permutation across payment processor/payment method/donation flow options can follow a unique sequence of steps, in terms of interactions among donors, our servers and payment processors, generation and consumption of queue messages, and database queries and writes.
In documentation about donation flows, payment by credit card without 3DS (cc) is separated from credit card with 3DS (cc-3DS), since the flows can be different. (However, in our systems, credit card payments are represented as a single payment method, regardless of whether 3DS was used.)
=== Payment processor/payment method/donation flow permutations ===
processor	payment method	one-time	recurring	monthly convert
adyen 	cc 	Y 	Y 	Y
adyen 	cc-3DS 	Y 	Y 	Y
adyen 	ApplePay 	Y 	Y 	?
adyen 	GPay 	Y 	? 	?
adyen 	rtbt-iDEAL 	Y 	Y 	?
ingenico 	cc 	Y 	Y 	Y
ingenico 	cc-3DS 	Y 	Y 	Y
ingenico 	rtbt-iDEAL 	Y 	Y 	?
paypal_ec	paypal 	Y 	Y 	?
braintree	paypal 	Y 	? 	?
amazon 	amazon 	Y 	? 	?
dlocal 	cc 	Y 	? 	?
dlocal 	cc-3DS 	? 	? 	?
dlocal 	rtbt-? 	Y 	? 	?
== Recurring charge ==
== Audit ==
== E-mail export ==
== Other ==