nyuuzyou/wikis huggingface dataset:
Reattribute edits

Not usually done anymore, Special:Renameuser is good enough for most people.
# change directory to /home/wikipedia/common/php-new/maintenance
# delete ../attribute.log and ../attribute.sql
# run: php attribute.php en wikipedia 1.2.3.4 "User Name" (this has to be done on suda right now because ariel has no php.)
# feed ../attribute.sql to mysql ON ARIEL (or whatever the master db server is)
# append ../attribute.log to w:Wikipedia:Changing_attribution_for_an_edit/Log
You MUST switch from suda to ariel between steps 3 and 4 or you will execute the query on the slave server and break things. Don't do that.
Category:MediaWiki production
Category:How-To
Reclaim space on full db servers

== Remove old relay logs ==
MySQL servers running version 4.* have a bug (here's one, maybe there are others) which results in the relay logs not being deleted. Space will eventually fill up on /a/sqldata due to this bug. You can clear out the old logs by running the utility
reset-mysql-slave hostname-here
from fenari. This stops the slave, issues a RESET SLAVE command (which automatically tosses all old relay logs and opens a fresh one) and restarts the slave.
== Remove old bin logs ==
Old bin logs also pile up in /a/sqldata. You can check the slave hosts to see which bin log file they are using and toss all the earlier ones. Check the one in use on each slave by looking at db.php for the slaves and giving the command 'SHOW SLAVE STATUS' and looking at the column Master_Log_File. Then you can clear the old logs out with the mysql command
PURGE MASTER LOGS TO 'name-of-bin-log-file-here'
and this will remove all bin logs older than the specified file. It syncs up the index file for these logs as well. (On mysql 5.x hosts you can issue the command PURGE BINARY LOGS instead.)
Category:How-To
Category:MySQL
Recover-dump

recover-dump allows the recovery of a database dump (logical backup) as described on MariaDB/Backups#Recovering_a_logical_backup:
 usage: recover-dump [-h] [--host HOST] [--port PORT] [--threads THREADS]
 [--user USER] [--password PASSWORD] [--socket SOCKET]
 [--database DATABASE] [--replicate]
 section
 
 Recover a logical backup
 
 positional arguments:
 section Section name or absolute path of the directory to
 recover("s3", "/srv/backups/archive/dump.s3.2022-11-12
 --19-05-35") 
 
 optional arguments:
 -h, --help show this help message and exit
 --host HOST Host to recover to
 --port PORT Port to recover to
 --threads THREADS Maximum number of threads to use for recovery
 --user USER User to connect for recovery
 --password PASSWORD Password to recover
 --socket SOCKET Socket to recover to
 --database DATABASE Only recover this database
 --replicate Enable binlog on import, for imports to a master that
 have to be replicated (but makes load slower).By
 default, binlog writes are disabled.
Recover instance from NFS

== Disable per-instance ==
Note: This procedure disables NFS, which means the contents of /home, /data/project, /public/dumps and /data/scratch will disappear. We can recover them later if you would like.
# Go to your instance's Hiera page. This is at Hiera:<projectname>/host/<hostname>.
# Add the line: mount_nfs: false
# ssh to your instance, and just unmount all the NFS mounts!
# You should also check /etc/fstab for old NFS mounts that will return on reboot if you do not remove them. Puppet does not remove existing lines from fstab for you.
DONE!
== Enable per-instance ==
# Go to your instance's Hiera page. This is at Hiera:<projectname>/host/<hostname>.
# Add the line: mount_nfs: true
# Force a puppet run: sudo -i puppet agent --test --verbose
Category:Cloud VPS
Redis Bloom Filter

MediaWiki is configured to use a bloom filter located on a single Redis server (which a backup slave). Currently the servers are rbf1001 (master) and rbf1002 (slave). Both are used equally for read requests. AoF persistence is used in case of restarts or power outages.
The filter is partitioned across many Redis string values due to the 512MB object size limit in Redis. There is one large "shared" filter that many "virtual" filters can use. Some code may use it's own filter if desired however. The shared was created manually via eval.php:
BloomCache::get( 'main' )->init( 'shared', 1e9, .001 );
If the filter data is cleared, this step needs to be redone since it is not automatic.
The filters should keep themselves self up-to-date and will self-populate over time. Manual population can be done as follows (using just the "TitleHasLogs" filter as an example):
mwscript populateBloomCache.php testwiki --cache main --domain testwiki --filter TitleHasLogs
Current filters follow the interface in includes/cache/bloom/BloomFilters.php.
Regenerate cached special pages

Some special pages (like Special:AncientPages) are cached in the database and only get updated periodically. If you need to manually regenerate a cached special page (which should rarely be necessary), you can do so by sshing to mwmaint1002 and running the following command:
 mwscript maintenance/updateSpecialPages.php --wiki=<wiki-database> --only=<name-of-special-page>
For example:
 mwscript maintenance/updateSpecialPages.php --wiki=testwiki --only=AncientPages
If you need to regenerate a cached special page across all wikis:
 foreachwiki maintenance/updateSpecialPages.php --only=<name-of-special-page>
Category:How-To
Release Engineering

This page is simply a link dumping ground for Release Engineering related topics on wikitech.
See the main :mw:Wikimedia Release Engineering Team on mw.org.
== Runbooks ==
The main list and set of runbooks is maintained on MediaWiki.org, at Wikimedia Release Engineering Team/Runbooks.
There are also some secondary runbooks locally on wikitech:
== See also ==
* Release Engineering/SAL the RelEng SAL, use via !log message in 
* Nova Resource:Deployment-prep/SAL, the SAL for the deployment-prep Labs project, use via !log deployment-prep message in (but note that the RelEng SAL seems to be generally preferred)
* How to deploy code
* Heterogeneous deployment
** Heterogeneous deployment/Train deploys
* Wikimedia binaries
* Deployment-prep project (beta)
** How code is updated in beta
Release Engineering/Drafts/Deployments

User:Brennen Bearnes and others from Release Engineering are working on improvements to Wikimedia deployment documentation here. See phab:T273802 for more info.
----
This is a hub for the deployment of software to the Wikimedia Foundation production servers. MediaWiki follows a weekly deploy cadence known as the Deployment Train, with separate backport windows Monday–Thursday for bugfixes and configuration changes. For other services, teams decide their own schedule. No deploys are allowed Friday–Sunday, with the exception of emergency situations.
----
Release Engineering/Drafts/Deployments/How to become a deployer

Members of the Wikimedia technical community are welcome - and encouraged! - to become deployers of MediaWiki and its extensions. Deploying is a serious responsibility which involves risk to production, so there are some prerequisites.
== Access ==
First, you'll need access to the relevant systems.
* Production access, and in particular membership in the deployment group
* Access to merge changes in wmf deploy branches (including mediawiki-config) by being added to the wmf-deployments Gerrit group (must first have deployment membership)
** Ask an existing wmf-deployments group member to do this.
== Communications ==
Next, you need to be connected and aware of several communication streams.
* Join (and read) the operations mailing list (ops@lists.wikimedia.org)
** Announcements that could impact how and/or when to deploy things are primarily sent there.
* Be present on IRC
** Join the IRC channel
*** This is where real-time communications and alerting about the state of production happen
*** This is where you'll communicate about ongoing deploys
** Other useful channels include , , and 
== Training ==
Finally, learn how to deploy.
Release Engineering hosts regular meetings for triaging production error logs, and conducts training on how to do backport deploys. Please attend at least one of each session, and pair with an experienced deployer on your first few deploys.
(This section to be expanded into something actually useful for the seeker-of-deploy-rights.)
Release Engineering/Drafts/Deployments/How to pair on the train

= Background =
Until October of 2019, running the deployment train was done by one assigned person in two-week shifts. In an effort to better share the workload and improve continuity of knowledge between successive deployers, Release Engineering then moved to a staggered pairing rotation. Each train has a primary conductor and a backup conductor. The following week, the primary moves to backup and a new primary is assigned.
= Helpful practices =
On Monday, begin by communicating with your partner and establishing how you'll collaborate over the course of the week.
If you are going to be away from your computer during or soon after deploy windows, let your partner know and coordinate coverage for those times. For example, if you have an appointment or family commitments after the deploy on Wednesday, ask your backup conductor to watch logs and the blocker task during that time.
These are rough guidelines for sharing the work, and should be improved as we learn more.
* Updates on IRC while your partner is working and updates on the train blocker ticket if they're offline seems to be a useful pattern.
* In general, communicate decisions on the blocker ticket and make use of the Server Admin Log.
* Liberal use of video chat for pairing on hard problems is encouraged.
* It works well to have the primary do the work of cutting the branch, syncing wikis, etc., while the backup keeps an eye on logs, works on improvements to deploy tooling, and is generally an extra pair of eyes for the whole process.
* If you are in doubt about anything and it's during your partner's working hours, consult them first and get their help in resolving your questions.
* If one member of the pair is in the European window and one is in the American window, both train deployment windows should be reserved on the Deployments calendar. This gives a backup deployer a defined window for moving the train forward outside the primary's working hours, if it becomes necessary.
* If the train is blocked or there are any other issues, communicate the transfer of responsibility on the train blocker ticket by assigning it to the responsible party and leaving a note.
Release Engineering/Runbook/WikimediaDebug

This is the runbook for release management of the WikimediaDebug browser extension.
== Links ==
* WikimediaDebug, end-user manual.
* Source code
* Firefox Addons
* Chrome Web Store
== Release process ==
See #release-process in the README.
== Access control ==
=== Chrome Web Store ===
The Chrome Web Store is managed from its Developer Dashboard. It works with a concept of publishers where your Google account has access to one or more publishers, and a publisher can owner one or more browser extensions. When you register your Google account as "Chrome Web Store Developer", there is by default a personal "Publisher" associated with you based on your username.
The WikimediaDebug extension is owned by the "chrome-wikimedia-debug" publisher, which is access-controlled via the member list of chrome-wikimedia-debug@googlegroups.com (manage group members, manage group-publisher association), where each member is the email address of a Google account. (It does not appear possible to associate a wikimedia.org Google Group, hence this uses a googlegroups.com address.)
Due to Google's one-time $5 payment requirement, it is common for publishers to associate their personal Google accounts. The developer access is not limited to individual "Publisher" namespaces, so once your account has developer access, you can be part any number of Publisher namespaces (of which Wikimedia Foundation would be one).
=== Firefox Addons ===
The Firefox Addons are managed from the Developer Hub. From the "Edit Product Page" interface for WikimediaDebug, one can manage which "Authors" have access to release or manage the extension, where each author is the email address of a Firefox account.
Before an author can be invited, you have to have logged-in with your Firefox account to https://addons.mozilla.org/ at least once.
=== Contact address ===
The contact address for app store alerts, confirmations, or other notifications from Mozilla and Google is wikimediadebug@wikimedia.org (manage group members).
== Troubleshooting ==
If it appears that a request is not capturing any data, check that Tideways and Excimer are properly installed and called:
* verify that php-tideways-xhprof is installed on the target host
* ensure that the extension is loaded, i.e. look for tideways_xhprof.so under /etc/php/$version/fpm/conf.d/
* check if auto_prepend_file is set in php.ini
Category:Release Engineering
Category:Runbooks
__NOTOC__
ReleasingToMavenCentral

Maven projects based on the discovery-parent-pom should not follow the instructions below, but instead look at the Building and Release sections of this README.
== Walkthrough ==
# Shove ReleasingToMavenCentral/Paranoid.xml into ~/.m2/paranoid.xml
# cd your_project
# mvn release:prepare
# git push --tags gerrit
# mvn release:perform
# git review
# Force everything through gerrit +2/+2
# Do the sonatype stuff over at https://oss.sonatype.org/
# git checkout <tag version that mvn release:prepare made>
# rm -rf ~/.m2/repository_backup; mv ~/.m2/repository ~/.m2/repository_backup
# mvn clean install -s ~/.m2/paranoid.xml
# Make sure you have Archiva configured
## In ~/.m2/settings.xml
## In your project's pom.xml
# mvn deploy
== What can go wrong ==
=== git push --tags ===
If you don't have permission to push tags you'll need to get it granted.
=== mvn release:perform ===
If Sonatype gives you HTTP 401s, check your settings file.
~/.m2/settings.xml:
 <settings>
 <servers>
 <server>
 <id>sonatype-nexus-staging</id>
 <username>MY_USERNAME</username>
 <password>MY_PASSWORD</password>
 </server>
 </servers>
 </settings>
=== git review ===
If you never installed the Change-Id-generating commit hook, you'll need to install it, then go back and generate some Change-Ids.
 gitdir=$(git rev-parse --git-dir); scp -p -P 29418 MY_USERNAME@gerrit.wikimedia.org:hooks/commit-msg ${gitdir}/hooks/
 git rebase -i HEAD~2
Change pick to edit, and continue as usual.
=== mvn clean install -s ~/.m2/paranoid.xml ===
If one of your dependencies is missing, you'll have to upload those to Archiva's mirrored repository. Make sure that the checksums you have agree with those on http://search.maven.org/
ReleasingToMavenCentral/Paranoid.xml

<settings>
 
 <mirrors>
 
 <mirror>
 <id>wikimedia.mirrored</id>
 <url>http://archiva.wikimedia.org/repository/mirrored/</url>
 <mirrorOf>external:*</mirrorOf>
 </mirror>
 </mirrors>
 
 <profiles>
 <profile>
 <repositories>
 
 
 <repository>
 <id>wikimedia.mirrored</id>
 <name>Wikimedia Mirrored Repository</name>
 <releases>
 <enabled>true</enabled>
 <updatePolicy>always</updatePolicy>
 <checksumPolicy>fail</checksumPolicy>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 <url>http://archiva.wikimedia.org/repository/mirrored</url>
 </repository>
 
 <repository>
 <id>wikimedia.releases</id>
 <name>Wikimedia Release Repository</name>
 <releases>
 <enabled>true</enabled>
 <updatePolicy>always</updatePolicy>
 <checksumPolicy>fail</checksumPolicy>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 <url>http://archiva.wikimedia.org/repository/releases</url>
 </repository>
 
 <repository>
 <id>wikimedia.snaphots</id>
 <name>Wikimedia Snapshot Repository</name>
 <releases>
 <enabled>false</enabled>
 </releases>
 <snapshots>
 <enabled>true</enabled>
 <updatePolicy>never</updatePolicy>
 <checksumPolicy>fail</checksumPolicy>
 </snapshots>
 <url>http://archiva.wikimedia.org/repository/snapshots</url>
 </repository>
 
 </repositories>
 </profile>
 </profiles>
 </settings>
ReleasingToMavenCentral/Pom.xml

To deploy to Archiva, make sure your project pom.xml includes Archiva's repository entries in the distributionManagement section:
 <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns=...>
 
 <distributionManagement>
 <repository>
 <id>archiva.releases</id>
 <name>Internal Release Repository</name>
 <url>https://archiva.wikimedia.org/repository/releases/</url>
 </repository>
 <snapshotRepository>
 <id>archiva.snapshots</id>
 <name>Internal Snapshot Repository</name>
 <url>https://archiva.wikimedia.org/repository/snapshots/</url>
 </snapshotRepository>
 </distributionManagement>
== See also ==
* ReleasingToMavenCentral/Settings.xml
ReleasingToMavenCentral/Settings.xml

To deploy to Archiva, you'll need to get the password for the archiva-deploy user, then add a couple of server entries to ~/.m2/settings.xml:
 <settings>
 <servers>
 <server>
 <id>archiva.releases</id>
 <username>archiva-deploy</username>
 <password>ARCHIVA_DEPLOY_PASSWORD</password>
 </server>
 <server>
 <id>archiva.snapshots</id>
 <username>archiva-deploy</username>
 <password>ARCHIVA_DEPLOY_PASSWORD</password>
 </server>
 </servers>
 </settings>
== See also ==
* ReleasingToMavenCentral/Pom.xml
Requestctl/Overview

This page explains the basic concepts behind how the requestctl tool manages configuration to control access and routing of web requests.
== Overview of http request control configuration ==
The requestctl tool modifies configuration data that resides in the main Etcd cluster. The configs are:
* Written/modified by conftool: in production, Conftool2git synchronizes requestctl objects to a git repository on the puppet servers under /srv/git/conftool/auditlog.
* Read from confd;
* Translated to VCL, which gets loaded by Varnish.
The configuration stored in etcd loads the following into the Varnish configuration: 
* The list of pooled ats backends to connect to in the same datacenter 
* The list of IP ranges for every public cloud, in the form of a netmapper file 
* A list of rate-limiting/ban rules for incoming traffic
== How requestctl modifies configuration ==
requestctl modifies data that resides under /conftool/v1/request-{ipblock,action,pattern}s/. requestctl enables you to manage the configuration for:
* IP ranges for every public cloud 
* Rate-limiting/ban rules for incoming traffic
When you run the requestctl commit command, the tool automatically generates a derived data type for each type of action, called vcl and haproxy_dsl respectively. This derived object is then injected into Varnish and haproxy.
A given Varnish host has: 
* One VCL condition per enabled action defined for the cluster the host is in.
** The action is either enabled for all datacenters, or it’s enabled for the specific datacenter the host is in.
* A netmapper file containing all the ipblock entries defined under the cloud scope 
** Possibly more netmapper files for things like crawlers.
* A VCL list of ACLs, one for each ipblock entry under the abuse scope
haproxy has: 
* One or multiple ACL definitions per pattern/ipblock used in an enabled action 
* A map file for each ipblock scope. 
* One http-request condition per haproxy_action that is enabled
On all cache proxy servers, a Confd instance watches these keyspaces and generates the following files:
* /var/netmapper/public_clouds.json from the data at /conftool/v1/request-ipblocks/cloud/, using this template. This netmap is then used to add an X-Public-Cloud: <name> header to requests coming from any IP address in those ranges. This data is updated daily using a script that runs on the puppetservers.
* /etc/varnish/blocked-nets.inc.vcl from the data at /conftool/v1/request-ipblocks/abuse/, using another template. This generates a list of varnish acls that can be referenced later in the VCL. This data is currently duplicated from the private puppet hiera, and will need to be kept in sync somehow.
* /etc/varnish/requestctl-filters.inc.vcl from the data at /conftool/v1/request-vcl/cache-$CLUSTER/{global,$datacenter}, where vcl gets stored when we run requestctl commit. This code gets injected directly in the cluster_fe_ratelimit VCL subroutine, so it only applies to cache misses at the moment (see: phab:T317794).
== Related documentation ==
* requestctl user guide
* requestctl tutorials
Category:SRE Service Operations
Category:SRE Traffic
Category:Overviews
Research/Archive

==Services and applications==
* Revision scoring
** project documentation
** API: https://ores.wmflabs.org/scores/
* Article recommendations
** project documentation
** Application: https://recommend.wmflabs.org
* Wiki Labels
** project documentation
** Application: https://labels.wmflabs.org
==Productization==
We defined in coordination with Ops three different stages and the corresponding requirements to turn a service like Revision Scoring into a "productized service", i.e., a service that can be used, on top of its other use-cases, by Wikimedia Foundation's products and features.
Stage 0 (Architectural discussion)	Stage 0 (Architectural discussion)
Horizontal scalability 	allow additional load be taken up by simply adding new instances
Caching 	Decide on caching and cache invalidation strategy
SPOF spotting / planning 	Draw out general architecture, find SPOFs and think of ways to mitigate them
Stage 1 (Implementation) 	Stage 1 (Implementation)
Actually build the code! 	This is actual development, start building stuff!
Staging environment 	provide an environment with the same set up as the Production environment, for test purposes
Deployment system 	allow to deploy new changes confident that you can roll them back if they fail
Puppetized setup 	allow spinning up new instances quickly
Comprehensive logging 	identify bugs and errors more easily
Stage 2 	Stage 2
Metrics monitoring 	define metrics that should go to graphite-labs.wikimedia.org (examples of such metrics are number of revisions processed per minute, per wiki, % cached, etc.)
Scale hardening 	Things to do to reduce amount of pages - for example, control the acceptance of new web requests when the celery queue is full. (This step is not required for all services).
Stage 3 	Stage 3
API documentation 	More comprehensive document endpoints (API) and usage
== WIP ==
Guidelines around these areas are Work in Progress
* Dependencies: We should define best practices around dependencies - some considerations: installing them system wide vs isolating them, using virtual environments, debianizing packages etc.
==See also==
* mw:Wikimedia Research
Rsync alternatives

Rsync isn't the only game in town when it comes to high-performance bulk data transfers.
High Performance Bulk Data Transfer (PDF) explains why rsync's performance can be disappointing and gives tips for tuning it, as well as a survey of alternative tools (page 32 onwards):
* GridFTP
* bbcp
* lftp (parallel accelerator for HTTP & FTP)
* axel (parallel accelerator for HTTP & FTP)
See also High Performance SSH/SCP - HPN-SSH, a set of performance patches for OpenSSH.