nyuuzyou/wikis huggingface dataset:
Dumps/OtherMisc

This page documents various dumpsets that are produced daily or weekly, not part of the generation of the xml/sql dumps.
All of these dumps run on database servers designated 'vslow, dumps', on a snapshot host dedicated to 'misc' dump generation (everything other than the xml/sql dumps).
The dump scripts are in our git puppet repo.
If errors are encountered when the specific cron job runs, the output is sent to ops-dumps@wikimedia.org.
* Global block table:
** dumped weekly
** contains an sql-format dump of information in the global block table
** managed by mw:Extension:GlobalBlocking) (code)
** Issues: Unless the database server goes away during the run, or database credentials change, this job should just run
* Cirrus search dumps:
** dumped weekly
** contains text indices, the file index (for commons) and the metadata index (for the entire cirrus cluster) in json format
** run by a maintenance script in mw:Extension:CirrusSearch (code)
** Issues: it's been quite reliable so far
* Content Translation dumps:
** dumped weekly
** contains parallel corpora that can be used by developers working on machine translation. 
** run by a maintenance script in mw:Extension:ContentTranslation (code)
** Issues: it has run out of memory when the language files being dumped have too much data; these can be split apart in order to resolve the problem. Example: see this phab task.
* Media info:
** dumped weekly
** two files for each wiki, consisting of titles of media files stored locally, and those used on the project stored remotely (on Commons).
** run by a shell wrapper around the onallwikis.py script in the operations/dumps repo (code)
** Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
* Page titles:
** dumped daily
** contains a list of all page titles in the main namespace (NS 0) per project
** run by the onallwikis.py script in the operations/dumps repo (code)
** Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
* Media titles:
** dumped daily
** contains a list of all titles in the Media namespace (NS 6) per project
** run by the onallwikis.py script in the operations/dumps repo (code)
** Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
* Short url mappings:
** dumped weekly
** each line contains an entry of the form short-url|log-url 
** run by the onallwikis.py script in the operations/dumps repo (code)
** Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
Dumps/Other information sources

This is a partial list of various pages about the xml/sql dumps. It would be nice to gather all the information in a couple of places, and it would be easier to keep it up to date.
Such a list should probably live on Meta; this will likely move there in some form.
Links to the xml/sql dumps themselves:
* download.wikimedia.org
* Torrents
Docs for WMF maintainers/admins:
* Dumps (relevant pages are linked to from here)
Dumping your own wiki:
* Backing up a wiki
* Manual:dumpBackup.php
Docs for users of the dumps:
* Dumps
* Wikipedia:Database download
* Manual:Importing XML dumps
* Research Data Proposals
Tools:
* Import tools
* Download tools
* Other tools
Category:Dumps
Dumps/Phases of a dump run

Each dump of a wiki runs in several phases (jobs). If you look at any completed wiki run from the main index page (http://dumps.wikimedia.org/backup-index.html) you'll see the jobs for that wiki listed with the most recently completed job listed first and a brief explanation of what each job is.
== Backup jobs overview ==
*dumps of various database tables, both private and public (via mysqldump)
*list of page titles in namespace 0, i.e. titles of articles (via mysqldump) 
*page abstracts for Yahoo (via dumpBackup.php with filter ActiveAbstract/AbstractFilter.php)
*:additional step to recombine chunks produced in parallel -- enwiki only
*page stubs, gzipped (via dumpBackup.php)
*:additional step to recombine chunks produced in parallel -- enwiki only
*XML files with revision texts, bzipped (via dumpTextPass.php, fetchText.php)
*:additional step to recombine chunks produced in parallel -- enwiki only
*log of actions taken on pages (via dumpBackup.php)
*7z compression of the XML file with all revision texts, for full history only
*: additional step to recombine chunks produced in parallel -- enwiki only
*rewrite of XML file with revision text into file of multiple bz2 streams, 100 pages per stream, for current revisions of articles only (via /usr/local/bin/recompressxml)
== Details ==
The worker.py script will produce a list of all jobnames it knows about for the given wiki with the specified configuration if it is given a jobname it doesn't recognize. Use the --dry-run option so that it doesn't do any of the prep work (e.g. cleaning up old dumps) that it would normally do before a run.
So for example,
python3 ./worker.py --job help --configfile wikidump.conf.dumps:en --dryrun enwiki
will produce a few lines of cruft you can ignore and then:
No job of the name specified exists. Choose one of the following:
noop (runs no job but rewrites md5sums file and resets latest links)
latestlinks (runs no job but resets latest links)
tables (includes all items below that end in 'table')
usertable 
...(snipped)...
metahistorybz2dump 
metahistory7zdump 
articlesmultistreamdump 
listing the jobs that it would run in the order that it would run them.
Category:Dumps
Dumps/Snapshot hosts

==Snapshot (XML dumps generation) cluster information==
===Hardware===
These hosts generate the sql/XML dumps or the "misc cron" dumps. For information about the hosts that serve them, see
Dumps/Dump servers.
In eqiad:
*snapshot1010: primary and monitor, Dell PowerEdge R440, Debian 10 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
*snapshot1011: primary, Dell PowerEdge R440, Debian 11 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
*snapshot1012: primary, Dell PowerEdge R440, Debian 10 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
*snapshot1013: primary, Dell PowerEdge R440, Debian 10 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
*snpashot1014: spare, Dell PowerEdge R440, Debian 11 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
*snapshot1015: wikidata_fill_in, Dell PowerEdge R440, Debian 11 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
*snapshot1016: primary (other dumps), Dell PowerEdge R650xs, Debian 11 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6326 16-core CPUs, 2 480GB SSDs in software raid 1
*snapshot1017: spare, Dell PowerEdge R650xs, Debian 11 (bullseye), 64GB RAM, 2 Intel Xeon Gold 6326 16-core CPUs, 2 480GB SSDs in software raid 1
===Current setup===
The dumps monitor which cleans up stale locks and generates the index.html file, currently runs on snapshot1010.
==== Worker nodes: ====
The fulldumps.sh script is supplied with a certain wikitype parameter, which can be any one of enwiki,wikidatawiki, or regular
* snapshot1012 is configured with a wikitype of enwiki
* snapshot1011 is configured with a wikitype of wikidatawiki
* snapshot1010 and snapshot1013 are configured with a wikitype of regular
This wikitype does not mean that the host will be used exclusively for this type of dump, but it will prioritize that type.
For example when the scheduler on snapshot1012 has completed the enwiki dump, it will check to see if any of the dumps from the list of big wikis remains to be dumped. If there are no outstanding dumps, it will then check the remaining wikis and start on those. The wikidatawiki type behaves in the same way.
* snapshot1015 is used to run a special fillin_wd script, with the aim or parallelizing the wikidata dumps even more. This waits until the 10th of the month, then it starts to work in parallel with snapshot1011 to dump the page content history files.
The testbed hosts are used for catchup, emergency testing of bug fixes, at scale tests of misc cron jobs, and so on.
For info on running the dumps, see Dumps#Starting_dump_runs.
===Other tasks===
All misc cron jobs apart from the sql/xml dump runs take place on snapshot1016.
Category:Dumps
Dumps/Software dependencies

==Software dependencies for running the dumps==
(Incomplete list)
Note that on WMF production hosts, this is handled by a base install and by puppet. 
The pile of python and C code used to manage these XML dumps needs the following:
*python 3.5 or better
*python3-yaml
*php7 with bzip2 and gzip streams enabled
*php7-mysql, php-xml, php-mbstring, any php extensions MediaWiki needs
*functional base MediaWiki installation, including maintenance scripts
*ActiveAbstract MW extension
*Flow MW extension if you have Flow content in your wikis
*mwbzutils (C utilities for manipulating bz2/gz-compressed MW xml files, in wikimedia repo)
*mysqldump
*curl (for api-based dumps jobs)
*p7zip installed, 7za in path of user running the dumps
*bzip2, lbzip2, gzip and supporting libraries
Category:Dumps
Dumps/Testing

===Unit tests===
Mediawiki dump maintenance scripts may be tested to some degree via unit and/or integration tests in MediaWiki core or the appropriate extension. See e.g. https://github.com/wikimedia/mediawiki/blob/master/tests/phpunit/maintenance/backup_PageTest.php New scripts should have tests added along with the new code.
The python SQL/XML dump scripts have some unit (or perhaps really integration) tests which can be run from the dumps repo in the xmldumps-backup subdirectory. See https://github.com/wikimedia/operations-dumps/blob/master/xmldumps-backup/test/all_test.sh for those.
===Deployment-prep testing===
Beyond unit tests, dumps should be tested on a snapshot instance in the deployment-prep project. Currently the test instance is deployment-snapshot05.deployment-prep.eqiad1.wikimedia.cloud. All jobs should be run as the dumpsgen user. Configuration files for the deployment-prep setup are found in /etc/dumps/confs/ with the extension ".labs". So for example, if you are testing something other than the SQL/XML dumps, you would want the file "wikidump.conf.other.labs".
To access deployment-prep you'll need to go via the bastion, see Help:Accessing Cloud VPS instances#Accessing Cloud VPS instances
====SQL/XML dumps====
The dumps repository, if you are testing SQL/XML dumps, is /srv/deployment/dumps/dumps just like in production, and you'll want to be in the xmldumps-backup subdirectory where all of the python scripts are located.
====Other dumps====
If you are trying to test one of the other dumps, you will likely want to run the maintenance script for it manually first, from the directory /srv/mediawiki/php-master/
You'll use MWScript.php to invoke it, for example: /usr/bin/php /srv/mediawiki/multiversion/MWScript.php maintenance/exportSites.php --wiki=commonswiki --help
Once you've gotten that working happily, you will want to test the bash script that runs it from cron. These typically reside in /usr/local/bin, where you can see many examples such as dumpcontentxlation.sh.
====Output from the dumps====
Output is written to directories in /mnt/dumpsdata/xmldatadumps/public for the SQL/XML dumps, and in /mnt/dumpsdata/otherdumps for the other dumps.
Please be conscious of the space on the instance; mediawiki and all of its l10n files live on the same filesystem as the dumps. If you see that your testing is close to filling up the disk, remove some of the older dump run directories, making sure that at least one complete run for each wiki is left intact for prefetch purposes.
Category:Dumps
Dumps/WikidataDumps

This page is about the Wikidata entity dumps. For information about the xml/sql dumps, please see Dumps/XML-SQL Dumps.
These dumps currently run once a week via cron.
The dump scripts attempt to retry failures up to a maximum number of retries. After that maximum, failures are reported by email to ops-dumps@wikimedia.org.
Issues with wikidata entity dumps should be filed under the Dumps-generation project in Phabricator, as well as Wikidata.
Because wikidata grows so quickly, over a period of a few months these dumps can get slow to finish. It's a bad idea to have one week's jobs still going when the next week's starts.
Several jobs are run in parallel; this number could theoretically be increased, but the dbas should approve first, since this means more concurrent requests to the database.
These dumps run on database servers designated 'vslow, dumps', on a snapshot host dedicated to 'misc' dump generation (everything other than the xml/sql dumps).
The dump scripts are in our git puppet repo.
Category:Dumps
ES

ES may refer to:
* External storage, the abstraction layer in MediaWiki for storing wiki page content outside the main wiki database. Related host names at WMF are prefixed with es, like es1001. 
* Elasticsearch, an open-source search engine used by Wikimedia. Typical releases use a minor version like v6.3 or v7.0, but release lines can be referred to as ES6 or ES7.
*Elastic Stack, a term for the combination of Elasticsearch, Logstash, and Kibana. Also known as an ELK Stack. 
* ECMAScript, popularly known as JavaScript, of which editions are often referred to by number or year, such as ES5, ES6, or ES2017.
Category:Disambiguation pages
EchoStore

Echostore provides application servers with shared access to the Echo extension's seen-time cache. It utilizes Kask, a RESTful opaque key/value storage abstraction on top of Apache Cassandra, deployed in an active/active configuration in both the eqiad and codfw data-centers.
== Production Environment ==
=== Service endpoints ===
==== Discovery ====
https://echostore.discovery.wmnet:8082/echoseen/v1
==== eqiad ====
https://echostore.svc.eqiad.wmnet:8082/echoseen/v1
==== codfw ====
https://echostore.svc.codfw.wmnet:8082/echoseen/v1
==== staging ====
https://staging.svc.eqiad.wmnet:8082
==== Prometheus metrics ====
https://echostore.discovery.wmnet:8082/metrics
==== Readiness ====
https://echostore.discovery.wmnet:8082/healthz
=== Cassandra ===
Utilizes the RESTBase storage cluster.
== Common Tasks ==
=== Deployment ===
==== k8s ====
Follow the Kubernetes documentation for the echostore service. 
==== Deployment-prep ====
Unlike production, Docker is used to deploy the echostore service in the deployment-prep environment. 
In a web browser:
# Visit https://horizon.wikimedia.org
# Select the deployment-prep project
# Select Project → Compute → Instances (in the sidebar)
# Locate and select the deployment-echostoreNN instance
# Select the Puppet Configuration tab
# Scroll to the bottom of the page, and click edit for Hiera Config (see below)
# Click "Apply Changes"
# Run sudo run-puppet-agent from deployment-echostoreNN
===== Sample Hiera Config =====
profile::cassandra::instances:
 deployment-echostoreNN.deployment-prep.eqiad1.wikimedia.cloud: {}
profile::docker::engine::declare_service: true
profile::docker::engine::settings: {}
profile::docker::engine::version: 1.12.6-0~debian-jessie
profile::docker::runner::service_defs:
 mediawiki-services-kask:
 config:
 base_uri: /echoseen/v1
 cassandra:
 authentication:
 password: cassandra
 username: cassandra
 hosts:
 - deployment-echostoreNN.deployment-prep.eqiad1.wikimedia.cloud
 keyspace: sessions
 table: values
 default_ttl: 86400
 listen_address: 0.0.0.0
 listen_port: 8080
 openapi_spec: /go/src/gerrit.wikimedia.org/r/mediawiki/services/kask/openapi.yaml
 service_name: sessions
 namespace: wikimedia
 override_cmd: ./kask --config /etc/mediawiki-services-kask/config.yaml
 port: 8080
 version: v1.0.10
=== Pool and Depool ===
== See also ==
* Kask
Editing Port Trunked Vlans in Netbox

Netbox is used to define what Vlans are 'trunked' on datacenter switch ports on which multiple Vlans are used.
This page gives a brief tutorial on how to modify which Vlans are trunked on a given port. This task is normally carried out by Netops, and generally is not something other SREs will need to complete. The exceptions may be for the small number of hosts which have multiple Vlans trunked to them. 
WARNING: Core network changes of this type can cause site-wide problems if done in error. If such a change is required and there is any doubt about the correct actions to take please contact Netops directly for assistance.
In general it is a two-step process:
# Modify the vlan settings for the port in Netbox GUI.
# Run homer against the relevant switch to update the actual device configuration.
==Edit Port in Netbox==
1. Firstly navigate to the page for the port that needs to be modified in Netbox
2. Click on the yellow 'EDIT' button at the top of the page
3. The edit page will now be displayed
4. Scroll to the bottom to the '802.1Q Switching' section:
5. To REMOVE a Vlan simply click the 'x' beside it in the list beside 'Tagged VLANs', it will disappear from the list.
6. To ADD a Vlan click on the '+' symbol to the right of the 'Tagged VLANs' box.
7. A pop-up menu appears with a drop-down list of Vlans.
7. Search for the Vlan that needs to be added in the 'filter' box and click its name.
8. Finally click the solid blue SAVE button to finalize.
==Run Homer Against Switch==
To sync the Netbox change to the actual switch running configuration run Homer from a cumin host against it. 
It should be run as a 'commit' operation, with a description of the change, for example:
 homer "lsw1-e1-eqiad*" commit "Adding Vlan X to port Y on lsw1-e1-eqiad."
Homer will generate the full config for the switch, then show the user the diff from the current live config, and prompt to continue. Adding a Vlan to a trunk should look something like this, only modifying the 'members' list for the given port:
cmooney@cumin1001:~$ homer cloudsw1-e4-eqiad* commit "Add Vlan 1105 to et-0/0/54 on cloudsw1-e4-eqiad."
INFO:homer.devices:Initialized 54 devices
INFO:homer:Committing config for query cloudsw1-e4-eqiad* with message: Add Vlan 1105 to et-0/0/54 on cloudsw1-e4-eqiad.
INFO:homer:Gathering global Netbox data
INFO:homer.devices:Matched 1 device(s) for query 'cloudsw1-e4-eqiad*'
INFO:homer:Generating configuration for cloudsw1-e4-eqiad.mgmt.eqiad.wmnet
WARNING:homer.capirca:Netbox capirca.GetHosts script is > 3 days old.
Configuration diff for cloudsw1-e4-eqiad.mgmt.eqiad.wmnet:
[edit interfaces et-0/0/54 unit 0 family ethernet-switching vlan]
- members [ cloud-xlink4-eqiad cloud-xlink8-eqiad ];
+ members [ cloud-instances2-b-eqiad cloud-xlink4-eqiad cloud-xlink8-eqiad ];
Type "yes" to commit, "no" to abort.
> no
If there is any doubt about the changes being made type 'no' and contact Netops for assistance. Otherwise type 'yes' and hit enter to complete the change.
Please refer to the Homer documentation for more details.
Eqiad data center

eqiad (pronounced eh-key-add, or the Ashburn DC) is located at the Equinix data center in Ashburn, Virginia, United States.
== Network ==
== Media gallery ==
File:Eqiadwmf 9069.jpg
File:Eqiadwmf 9045.jpg
File:Eqiadwmf 9043.jpg
File:Eqiadwmf 9049.jpg
File:Eqiadwmf 9052.jpg
== See also ==
* Pages linking to ""
* WMF Projects/Data Center Virginia, on mediawiki.org
* Obsolete:Eqiad Migration Planning
 
Category:Data centers
Eqsin data center

eqsin (pronounced eh-cue-sin, or the Singapore DC) is a caching data center located at Equinix's SG3 data center in Singapore.
== Further reading ==
* A new data center in Singapore (April 2018) by Brandon Black, blog.wikimedia.org.
== See also ==
* :Category:Eqsin data center
* Pages linking to ""
 
Category:Data centers
Esams data center

esams (pronounced ee-sams) is a caching data center located at Digital Realty's AMS9 data center in Amsterdam, The Netherlands.
== History ==
This data center started around 2005 under the name knams ("kn" from Kennisnet, the operator at the time, and "AMS" from the code for Amsterdam Schiphol Airport). In 2008, most of the equipment was moved to the EvoSwitch AMS-1 data center. That new data center was named esams ("es" for EvoSwitch), with a small amount of equipment remaining at knams for networking. The knams data center at Science Park 121 changed owners and was renamed several times over the years (): First SARA Amsterdam, later Vancis Amsterdam, followed by Interxion Science Park and currently Digital Realty Ams9. In 2023, everything was consolidated in knams, which was renamed esams to minimize the need for code changes. 
== Usage ==
Caching.
== Layout ==
All of the layout and racking information is currently housed in our Netbox instance.
This site matches our 'standard caching site' layout of two adjacent racks with cache proxies and other misc services being hosted within roughly 25 hosts.
== Gallery ==
File:Knams-15-knsq-10.jpg|Terminal at rackserver (2006)
File:Knams-15-knsq-12.jpg|Servers (2006)
File:Knams-15-knsq-13.jpg|Servers knsq1-8 (2006)
File:Knams-15-knsq-14.jpg|Network hubs (2006)
==See also==
* Network design#Europe
* Pages linking to ""
 
Category:Data centers
Etcd/Clients

This page describes the clients that use etcd in the WMF stack.
=== PyBal ===
Pybal watches the $conftool_prefix/pools/$::site/$::cluster/service-name keyspace for each pool it's configured for, and reacts to changes there by reloading its configuration.
The client supports https but no authentication and is purely read-only; it also doesn't support SRV records, so a specific server must always be specified.
=== Confd ===
Confd is an agent that is intended to be a bridge between etcd and any application that cannot speak to it directly. It can speak HTTPS and supports use of SRV records for client connections. It has the ability to watch a set of keys, or a single one, or to just query etcd at regular intervals. It is used for the following softwares:
# Varnish: used to handle the lists of backend directors. Varnish reloads its config upon upgrades
# DNS: it is used to manage the up/down state of IPs from eqiad/codfw for dns discovery. gdnsd watches the files and reacts to the changes.
# Redis: used to handle the replication direction. Uses the same record we use for discovery.
=== Conftool ===
It's the only tool that actually is authorized to write to etcd. It supports authentication, https, SRV records.
There's also dbctl, an extension to conftool used to manage MW's view of databases.
EventLogging/Storage/PurgeData

Read the newest data retention and auto-purging criteria and implementation here:
https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/Data_retention_and_auto-purging
== Schemas that can be purged ==
These are the schemas that can be purged:
Should we move this to wikitech?
[https://docs.google.com/a/wikimedia.org/spreadsheets/d/1RAhDbppfWDQsUXXr7r_5-7GFMgdwqbk28i1Df70Q4oU/edit#gid=0]
==Purging mechanism ==
Sean Pringle scheduled the purging of data via the MariaDB using the Event Scheduler 
 CREATE TABLE purge_schedule (
 table_name varchar(100) NOT NULL,
 days tinyint(3) unsigned NOT NULL
 );
Then for each EL table you would do:
 INSERT INTO purge_schedule VALUES ('MultimediaTiming_7193302', 30);