nyuuzyou/wikis huggingface dataset:
Distribution upgrades/jessie stretch

This is a page to summarize all the things to look for when upgrading an instance/server from jessie to stretch.
Feel free to add sections by service/package name or your own more details report on a subpage.
Also see this useful version table on debian.org.
== Apache ==
=== libapache2-mod-php ===
On stretch libapache2-mod-php7.0 may have to be installed explicitly, while libapache2-mod-php5 was pulled in as a dependency automatically in the past.
example patch to fix
== MySQL ==
=== mysql-server-5.5 ===
* mysql-server-5.5 have been replaced with mariadb-server in stretch.
=== mysql-client-5.5 ===
* mysql-client-5.5 have been replaced with mariadb-client in stretch.
=== mysql_ functions in PHP ===
The MySQL extension of PHP is deprecated and does not exist in PHP7 anymore. This means if you have any mysql_connect(), mysql_query() or any other function starting with "mysql_" in your PHP code, it will stop working on stretch. You have to switch to either PDO or mysqli. Also see stackoverflow thread and tutsplus tutorial.
== PHP ==
=== php5 ===
* php5 has been replaced with php7.0 in stretch
=== php5-cli ===
php5-cli does not exist anymore and is replaced by php7.0-cli.
example patch to fix
=== php5-mysql ===
php5-mysql does not exist anymore and is replaced by php7.0-mysql.
example patch to fix
== Python ==
== instance upgrade reports ==
=== Phabricator ===
* Phabricator daemons will fail to launch and phabricator will fail to work unless php 7.1 is installed.
* php7.0 is not supported in phabricator. Only 5.6 or lower and 7.1+.
=== Wikistats ===
Wikistats uses a typical LAMP-stack. Apache with PHP and MySQL so it can serve as an example for that.
Things that had to be done:
* php5-mysql had to be replaced with php7.0-mysql 
* php5-cli had to be replaced with php7.0-cli 
* libapache2-mod-php7.0 had to be installed (while libapache2-mod-php5 was automatically pulled in as dependency before) 
* I had to remove all mysql_ functions from PHP code and replace it with PDO. The deprecated MySQL extension does not exist anymore in PHP7. 
(note: I did not have to do anything about MySQL packages because i am already using MariaDB in this project. If you use MySQL you also need to switch and see above)
== External links ==
* debian.org - What's new in Debian 9
* debian.org - Upgrades from Debian 8 (jessie)
Docker-registry/jwt-authorizer

The Docker registry also accepts authorization using JSON Web Tokens. This option was introduced to allow Trusted GitLab Runners to push images to the registry during CI builds (T308501). nginx on the registry is configured to accept jwt from a list of allowed IPs (profile::docker_registry_ha::registry::jwt_allowed_ips). 
Furthermore a dedicated daemon is executed to handle the jwt validation. This daemon is a go application hosted under https://gitlab.wikimedia.org/repos/releng/jwt-authorizer.
=== Building a new version ===
Building jwt-authorizer mostly follows the standard Debian Packaging workflow. jwt-authorizer needs golang >=1.16, so currently DIST=bullseye-backports is used and the import on apt needs a additional ignore parameter: 
* checkout the git repo on the build host as your user (see Debian Packaging for current buildhost, build2001.codfw.wmnet as of today)
* change to the jwt-authorizer folder:
# on build2001
cd jwt-authorizer
* build the package as your user:
# on build2001
DIST=bullseye-backports pdebuild
=== Publish a new version ===
The newly build version has to be synced to apt host (apt1001.wikimedia.org as of today):
# on apt1001
rsync -vaz build2001.codfw.wmnet::pbuilder-result/bullseye-amd64/*<PACKAGE VERSION>* .
Next the package has to be loaded with reprepro:
# on apt1001
sudo -i reprepro -C main --ignore=wrongdistribution include bullseye-wikimedia /path/to/<PACKAGE>.changes
# on apt1001 (for hosts still on buster)
sudo -i reprepro -C main --ignore=wrongdistribution include buster-wikimedia /path/to/<PACKAGE>.changes
Now check if the new version is available:
# on apt1001
sudo -i reprepro lsbycomponent jwt-authorizer
Log the change to SAL
 !log "Imported jwt-authorizer <PACKAGE VERSION>"
Docker/apparmor

Docker apparmor on Ubuntu Trusty:
# AppArmor profile from lxc for containers.
#include <tunables/global>
profile docker-default flags=(attach_disconnected,mediate_deleted) {
 #include <abstractions/base>
 network,
 capability,
 file,
 umount,
 # ignore DENIED message on / remount
 deny mount options=(ro, remount) -> /,
 # allow tmpfs mounts everywhere
 mount fstype=tmpfs,
 # allow mqueue mounts everywhere
 mount fstype=mqueue,
 # allow fuse mounts everywhere
 mount fstype=fuse.*,
 # allow bind mount of /lib/init/fstab for lxcguest
 mount options=(rw, bind) /lib/init/fstab.lxc/ -> /lib/init/fstab/,
 # deny writes in /proc/sys/fs but allow binfmt_misc to be mounted
 mount fstype=binfmt_misc -> /proc/sys/fs/binfmt_misc/,
 deny @{PROC}/sys/fs/** wklx,
 # allow efivars to be mounted, writing to it will be blocked though
 mount fstype=efivarfs -> /sys/firmware/efi/efivars/,
 # block some other dangerous paths
 deny @{PROC}/sysrq-trigger rwklx,
 deny @{PROC}/mem rwklx,
 deny @{PROC}/kmem rwklx,
 deny @{PROC}/sys/kernel/[^s][^h][^m]* wklx,
 deny @{PROC}/sys/kernel/*/** wklx,
 # deny writes in /sys except for /sys/fs/cgroup, also allow
 # fusectl, securityfs and debugfs to be mounted there (read-only)
 mount fstype=fusectl -> /sys/fs/fuse/connections/,
 mount fstype=securityfs -> /sys/kernel/security/,
 mount fstype=debugfs -> /sys/kernel/debug/,
 deny mount fstype=debugfs -> /var/lib/ureadahead/debugfs/,
 mount fstype=proc -> /proc/,
 mount fstype=sysfs -> /sys/,
 deny /sys/[^f]*/** wklx,
 deny /sys/f[^s]*/** wklx,
 deny /sys/fs/[^c]*/** wklx,
 deny /sys/fs/c[^g]*/** wklx,
 deny /sys/fs/cg[^r]*/** wklx,
 deny /sys/firmware/efi/efivars/** rwklx,
 deny /sys/kernel/security/** rwklx,
 mount options=(move) /sys/fs/cgroup/cgmanager/ -> /sys/fs/cgroup/cgmanager.lower/,
 # the container may never be allowed to mount devpts. If it does, it
 # will remount the host's devpts. We could allow it to do it with
 # the newinstance option (but, right now, we don't).
 deny mount fstype=devpts,
}
Domains

Wikimedia maintains many domains for a variety of reasons, including:
* Serving content (e.g. wikipedia.org)
* Redirecting users (e.g. misspellings such as wikepedia.org)
* Trademark reclamation against malicious domains that abuse e.g. typosquatting
* Internal development usage (e.g. the internal-only development domain wmnet.org)
Not all domains under the Wikimedia community are owned by WMF; some are owned and operated by trusted community members.
WMF utilizes both Markmonitor and porkbun for domain management; Markmonitor is the main domain registration entity while porkbun only exists for some savings for .wiki TLD registrations.
== Administration ==
Registration of domain name ownership for WMF is handled by the legal team. DNS management is handled by the SRE team, largely Traffic.
== Canonical domain names ==
While the Foundation may own many other domains for trademark, legal, or project/redirect reasons, there is only one small set which are considered to be the canonical set for actual projects and content, which are subjected to higher security standards:
* mediawiki.org
* w.wiki
* wikibooks.org
* wikidata.org
* wikifunctions.org
* wikimedia.org
* wikimediafoundation.org
* wikinews.org
* wikipedia.org
* wikiquote.org
* wikisource.org
* wikiversity.org
* wikivoyage.org
* wiktionary.org
* wmfusercontent.org
== Parked domains ==
WMF has two types of domain parking: Redirection parking and dead parking. Domains using either type are listed in .
# Redirect-based parking: Using ncredir, WMF redirects many unused domains to proper ones (e.g. wikepedia.org → wikipedia.org). All domains with an arrow pointing to ncredir-parking use this method of parking. This is the most common form of parking as we generally want "incorrect" domains to redirect users in helpful ways rather than error out.
# Dead parking: Domains that are registered with few (or no) records attached. This parking is usually reserved for previously-owned, malicious domains that we'd rather just not exist (e.g. expletives denigrating Wikimedia in the name): We just hold on to the name and do nothing with them. All domains with an arrow pointing to parking use this method of parking.
Redirect-based domains are automatically managed/created with ncmonitor.
== Designations ==
Domain Name 	Purpose 	Comment
.wmnet 	Internal private TLD 	
enwp.org 	Shortcut domain, e.g. enwp.org/banana → en.wikipedia.org/wiki/banana	Not to be confused as a shortener like w.wiki
toolforge.org 	 	cf. blog post
w.wiki 	Official Wikimedia URL shortener 	
wikimedia.community	 	
wikimediacloud.org 	 	cf. blog post
wmcloud.org 	 	cf. blog post
wmflabs.org 	 	Prod DNS servers do not handle this domain. OpenStack Designate.
wmfusercontent.org 	User uploaded files on Phabricator and Gerrit
Donate.wikipedia.org

donate.wikipedia.org is an alias for donate.wikimedia.org. 
== Service ==
The Apache configuration for donate.wikimedia.org also answers to donate.wikipedia.org. There is no hostname-wide redirect in place.
If a request is non-canonical or if its response was going to be a redirect anyway (like the root home page), then MediaWiki will use the canonical hostname in its redirect destination.
* Hosted on: Application servers
* Puppet/Apache configuration: sites/main.conf
== History ==
* 2013: Donation pages moved to a new wiki, donate.wikimedia.org, served by the main Application servers. donate.wikipedia.org is now an alias for that hostname.
* 2008: Donation pages moved to wiki templates on wikimediafoundation.org. The donate.wikipedia.org hosted on srv9 is now just a redirect to wikimediafoundation/wiki/Donate
* Before 2008: The donation page used to run the frontend exclusively on this server.
== External link ==
* https://donate.wikipedia.org redirects to https://donate.wikimedia.org/w/index.php?title=Special:LandingPage
* https://donate.wikipedia.org/wiki/Special:BlankPage (no redirect)
Category:Services
Category:Fundraising
Draft:AQS

The Analytics Query Service, or AQS, is a mapping of REST-style endpoints to data stores. The system is used to deliver data to consumers like Toolforge tools, custom dashboards like Wikistats, mw:Special Pages, and more. The main purpose of AQS as a piece of infrastructure is to make it easy to serve data.
==Creating a new Endpoint==
Step by step, this section contains all you need to set up a new AQS 2.0 endpoint. Step 0 is making sure you have data loaded up in a data store that is considered best to serve it. To determine this, please talk to the Data teams (Data Persistence and Data Platform Engineering).
===Step 1: Make a repository===
TODO First:
: and : clean up service scaffold golang and decide whether to add a more aqs-specific version of it or make "AQS family" one of the options when scaffolding a new service
: : document testing
: decide what to refactor out of the individual repositories
:: configuration/configuration.go
:: cassandra.go
:: MAKEFILE
:: druid-schemas.yaml
:: env.go
:: healthz.go (part of scaffold)
: decide on coding conventions (tabs standard in Go world)
:: linter
: : move repositories to reflect ownership
: : implement Hugh's idea to poll/read the latest Druid mediawiki history reduced snapshot
:: (in a nutshell: have an airflow job just run every hour and produce a config file somewhere central that the AQS services can read. Then the druid AQS services read that file at least every 24 hours and update their settings from it. This way, if a new snapshot is available, it will be automatically picked up without any deploy or restart of services needed.)
Once you settle on a name for your new repository, a scaffold will help you get started. An overview of what the scaffold creates: Draft:AQS/Repository_scaffold_overview. We suggest pushing your code to https://gitlab.wikimedia.org/repos/data-engineering/aqs/<<your-repository-name>>.
===Step 2: Unit Test and Implement===
: Section on unit tests: how to write
: Section on aqs-assist: how to implement a service and make your tests pass
:: Section on Druid and Cassandra utilities in aqs-assist
: Section on 
===Step 3: Integration Tests===
: Integration tests with Cassandra
: Integration tests with Druid
: Note on using other data stores
===Step 4: Document===
: Section on in-code documentation and the API spec
: Section on on-wiki documentation (decide on format by refactoring docs for the Existing Endpoints below)
===Step 5: Deploy===
Maybe move detailed deployment instructions to a subpage Draft:AQS/Deploy, but cover everything:
: routing and REST Gateway
: loading data in data stores and Cassandra/Druid/other
: deploying the code itself and the permissions required
==Existing Endpoints==
===Device Analytics===
===Geo Analytics===
===Media Analytics===
===Page Analytics===
===Edit Analytics===
===Editor Analytics===
Drmrs data center

drmrs (pronounced dreamers, or the Marseille DC) is a caching data center located in Marseille, France.
Its main purpose is to serve as second cache DC for EMEA (Europe, Middle East, Africa), after Esams.
== Photo gallery ==
== Further reading ==
* Building DReaMeRS: How and why we opened a datacenter in France (July 2022), techblog.wikimedia.org.
==See also==
*Pages linking to ""
 
Category:Data centers
Dsh

DSH stands for dancer's shell / distributed shell and is a way to run commands via SSH on many hosts at once.
* https://www.netfort.gr.jp/~dancer/software/dsh.html.en
----
Since agent forwarding was disabled, this is more of a pain.
However, deployers can use:
SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -o -oIdentitiesOnly=yes -o -oIdentityFile=/etc/keyholder.d/mwdeploy.pub -- [command]
From a deployment server, to run a command on all MediaWiki boxes.
If you want to run a command on a group of hosts for some other service, check Keyholder and change the name of the key file appropriately.
The reason you need to specify the identity file is that the keyholder has a lot of keys and quite likely your one is far enough down the list that ssh will offer 6 other keys first to the remote end, 6 wrong keys that is, and then will get told off for too many bad auth attempts.
Category:Deployment
Dumps.wikimedia.org

dumps.wikimedia.org hosts dumps of various kinds.
Services:
* Wiki database XML dumps (see Dumps)
** https://dumps.wikimedia.org/backup-index.html
* Pageview statistics
** https://dumps.wikimedia.org/other/pagecounts-raw/
* MediaWiki releases (only until MediaWiki v1.22.2, these are now on releases.wikimedia.org)
** https://dumps.wikimedia.org/mediawiki/
== External link ==
* https://dumps.wikimedia.org
Category:Services
Dumps/Adds-changes dumps

==Adds/changes dumps overview==
We have an experimental service available which produces dumps of added/changed content on a daily basis for all projects that have not been closed and are not private.
The code for this service is available in our git repository (master branch). It relies on the python modules used by the regular dumps, at in the regular dumps repo.
The job runs out of cron on one of the snapshot hosts (see hiera/hosts for which one), as the datasets user. Everything except initial script deployment is puppetized. Scripts are deployed via scap3 as part of the general Dumps deployment.
Directory structure:
Everything for a given run is stored in dumproot/projectname/yyyymmdd/ much as we do for regular dumps.
===How it works===
We record the largest revision id for the given project, in the file maxrevid.txt, older than a configurable cuttof (currently at least 12 hours old). All revisions between this and the previously recorded revision for the previous day will be dumped. The delay gives editors on the specific wiki some time to have weeded out vandalism, advertising spam and so on.
We generate a stubs file containing metadata in xml format for each revision added since the previous day, consulting the file maxrevid.txt for the previous day to get the start of the range. We then generate a meta-history xml file which contains the text of these revisions grouped together and sorted by page id. Md5 sums of these are available in an md5sums.txt file. A status.txt file is available to indicate whether we had a successful run ("done") or not.
After all wikis have run, we check the directories for successful runs and writes a main index.html file with links for each project to the stub and content files for the latest successful run.
==When stuff breaks==
You can rerun various jobs by hand for specified dates. Be on the snapshot host responsible (check iera/hosts for the one that runs misc cron jobs).
In a screen session, do:
# sudo -s datasets
# python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --date YYYYMMDD
If you want more information you can run the above script with --help for a usage message.
==Some numbers==
Here's a few fun numbers from the March 12 2019 run.
wiki 	revision count	stubs time	content time
wikidatawiki	756837 	2 m 	2 h 56 m
enwiki 	82370 	1 m 	27 m
commons 	75576 	1 m 	4 m
dewki 	19938 	1 m 	4 m
Explanation:
* stubs time: length of time to generate the gzipped file containing metadata for each new revision
* content time: length of time to generate the bzip2-compressed file containing content for each new revision
* h = hours, m = minutes
Category:Dumps
Dumps/CategoriesRDF

This doc is about the dumps of categories in rdf format. For information about the xml/sql dumps including the category-related tables, please see Dumps/XML-SQL Dumps.
Issues with these dumps should be reported in Phabricator under the Dumps-generation project, as well as the Wikidata-Query-Service project. WDQS ingests these dumps, which is why they are produced.
These dumps are run out of cron.
* Weekly runs: generate full lists of categories on each public Wikimedia project in rdf format
* Daily runs: generate sparql-format lists of queries to run which move, delete and insert categories that have changed since the previous day
These dumps run on database servers designated 'vslow, dumps', on a snapshot host dedicated to 'misc' dump generation (everything other than the xml/sql dumps).
The dump scripts are in our git puppet repo.
The daily runs take about 15 minutes to complete, as of early 2019. The weekly runs take about 2.5 hours to complete.
Category:Dumps
Dumps/Drafts/Using bitstream

==Notes about replacing (parts of) mwbzutils with python's bitstream module==
To look into:
* speed of python bitstream module
* proper determination of block boundaries
* flexibility of python's bz2 module
* avoiding edge cases in decompression
* building bz2 files from arbitrary blocks (object store??)
* ...?
=== Speed of bitstream module ===
Some uses don't need the world's best optimization. Need to categorize all our use cases.
=== Bzip2 block boundary determination ===
Just searching for the bit-aligned start of block marker is not enough. Decompressing the first 8 or 16k is not enough. What is enough to be sure we have a good block?
Cases in which this is not an issue:
* decompressing first block in a file
* decompressing last block in a file
* decompressing streams that are known to have a maximum (small) size
* decompressing in serial and keeping track of block boundaries as we go along
=== Avoiding edge cases in decompression ===
We don't use lbzip2 for decompression anywhere, because it is possible, when it does not read blocks serially, that it can get the block boundaries wrong.
It uses two methods to find block boundaries:
* scan - naively find block markers, possibly bit-shifted
* parse - TBD
(More notes soon)
=== Flexibility of python's bz2 module ===
The python routines, even those that incrementally decompress data from a buffer, expect a bzip2 stream header. We can construct a stream of bytes with that header at the front. we can calculate a stream crc in cases where we need one, and stuff it on the end, if we are clever. This should cover our uses.
=== CRC calculation of bz2 files written from arbitrary blocks ===
If we can get good block boundaries, this can be done; a proof of concept exists in C this very minute. The if is the hard part.
=== See also ===
* Brion's notes on dbzip2 (repo: https://github.com/sowawa/mediawiki-svn/tree/trunk/dbzip2)
* Python's bz2 module (sources: bz2.py _bzmodule.c
* Python's bitstream module (source: https://github.com/scott-griffiths/bitstring)
* bzip2 file and block format (source: https://gitlab.com/federicomenaquintero/bzip2/commits/master)
* lbzip2 docs and source
Dumps/Mirror status

:See also: Mirrors
=== Current mirrors ===
We keep track of current mirrors for dumps in the operations/puppet repo (code) in hiera; check common/profile/dumps/distribution.yaml for the entry profile::dumps::distribution::mirrors. Adding the host information here (ipv4, ipv6) will add the mirror to the list of hosts which may rsync from our dumps servers.
Mirrors that are active should be marked as such; mirrors that are not yet up or which seem to have lapsed are marked as not active. The rest of the fields should be self-explanatory.
=== Adding a mirror ===
Requirements: Before you add a mirror, discuss bandwidth and storage with the contact person. Current storage requirements are listed at mirroring. Note that dumps are run twice a month and so bandwidth includes the mirror pulling the latest copy of all dumps on all projects of the types of dumps they choose to mirror. 
Phab: As soon as an organization has agreed to host some dumps or datasets, add a Phabricator task for it tagged with the Dumps-generation project.
Config: Once a decision has been reached on what to be mirrored, you can add the relevant information as described under 'Current Mirrors', above.
Mirroring: The host itself can rsync either from another mirror, if they are set up for rsync, or from us. We have strict bandwidth caps but we provide rsync stanzas for pulling copies of just the number of xml/sql dumps desired; see the puppet template for the rsync config.
If an organization wishes to pull from another mirror using this convenient setup, they can pick up the appropriate rsync-inc-last-n.txt file (example) from the mirror at the top level, and put together an appropriate rsync command on their end.
Announcing: Once this is done and the mirror is up to date, you should add it to the list of current mirrors on Meta, email an announcement to xmldatadumps-l, and add it to the mirrors web page in our puppet repo.
Boooooo: Note that at this time there is no automatic redirection of dumps downloaders to our mirror sites.
=== In progress mirrors ===
See Phabricator for mirrors being set up/worked on.
Category:Dumps