nyuuzyou/wikis huggingface dataset:
Vlan migration

= Migration to per rack vlans =
As you might have heard before, we're migrating our network infrastructure to a per rack vlan design (also known as L3 to the Top Of Rack).
And we need your help!
== Why ? ==
Our "legacy" design consists on vlans shared across all the racks of a given row (usually 8 in core DCs). This had the upside of easier management, but now that we have proper automation, it is not that much of an upside.
However the downsides are significant:
* Shared fate (a network flood or platform bug would impact all the host on the rows)
* Higher cost (requires special licenses)
* Higher complexity to troubleshot
== Where ? ==
Now:
* codfw
** Main tracking task : https://phabricator.wikimedia.org/T354869
No migration needed, 100% on a per rack vlan :
* eqiad rows E and F
* drmrs
* magru
* esams
In the future :
* eqiad rows A-D
* eqsin
* ulsfo
== How ? ==
If you're provisioning a new server, you have nothing to do. Hurray.
For now let's focus on the 557 baremetal codfw servers that are still in the legacy "private" vlans. You can see them grouped on this Netbox report : https://netbox.wikimedia.org/extras/scripts/19/
You can also query them by team-ownership with (adapt the first alias for your own team):
sudo cumin 'A:owner-collaboration-services and A:codfw and P{F:fqdn ~ ".wmnet$"} and not A:vms and not P{F:netmask = "255.255.255.0"}'
To not have to wait for a regular 5 years refresh cycle, we're kindly asking SREs to re-image their servers using the --move-vlan argument.
For a lots of servers this doesn't matter an will work out of the box, for example if it's referenced by its hostname in other parts of the infra. But for others (like database) where IPs are hard-coded or cached in exotic places, it can be more complex.
The cookbook will run some validations before doing the re-numbering, as well as check for the potential presence of the server's "old" IP in various repositories.
If the host have BGP configured, the migration should take care of creating the new sessions automatically, but won't clean up the old ones. Some edge cases might exist as well when the BGP peers IPs are hard-coded.
Don't hesitate to contact Netops or the Infrastructure Foundation team for any help to prepare this migration.
Kuddos to the Service Operations team who has been massively and successfully migrating its mw (wikikube-worker) servers.
== What's next ? ==
The above only applies to baremetal servers in the private vlans.
Ganeti is being tackled with Ganeti#Routed Ganeti
Baremetal public vlans hosts have nothing to do for now.
Vopsbot

Vopsbot (source) is an IRC bot that allows interacting with the splunk oncall api from IRC.
At the moment it's running in the #wikimedia-sre and #wikimedia-sre-private IRC channels on libera.chat with nickname sirenbot
It is designed to interact with the victorops api.
=== Installation and configuration ===
The bot can be installed in the wikimedia environment using the profile::vopsbot class.
It uses a local database using sqlite for basic functions like ACL and topic storage, located in /srv/vopsbot/vopsbot.db. It also uses two configuration files:
* /etc/vopsbot/ircbot-config.json, a json file containing the configuration for the IRC bot framework vopsbot is based upon
* /etc/vopsbot/users.yaml which contains a list of irc nick -> victorops usernames overrides. When adding a new user to victorops, also update this configuration (found in private hiera as profile::vopsbot::vo_users)
In addition, the victorops credentials are passed to the bot via environment variables.
=== Bot commands ===
====== See who is oncall now ======
Command: !oncall-now [team]
If you omit the team, vopsbot will try to figure out which team you're on and report who's oncall for your team; otherwise it will default to reporting who's oncall for SRE.
====== List paging incidents for the last 24 hours ======
Command: !incidents [team]
If you omit the team, vopsbot will try to figure out which team you're on and report incidents for your team; otherwise it will default to reporting paging incidents for SRE
====== Acknowledge an open incident ======
Command: !ack <incident-number> [victorops username]
If you omit the username, vopsbot will try to figure out what your victorops username is; if none is found, an error will be returned. This command can only be given in public.
====== Resolve an open incident ======
Command: !resolve <incident-number> [victorops username]
If you omit the username, vopsbot will try to figure out what your victorops username is; if none is found, an error will be returned. This command can only be given in public.
=== Access control ===
All members of the SRE team are granted admin status automatically, meaning you can issue commands freely and anywhere. Specific ACLs (using ircbot's !acl_add, !acl_remove and !acl_get commands) can be managed
for every command. We can grant the right to execute a command to an individual nickname or to a whole channel. I would argue that if other teams want to use it, we'll add an ACL specifically for their "reserved/trusted" channel for ease of maintenance.
=== Other capabilities ===
The bot can sing, if you ask. You'll figure out how, if you never give up looking...
=== Restart sirenbot ===
Log on the active alert node and sudo systemctl restart vopsbot.service.
W.wiki

w.wiki is the official Wikimedia URL shortener, powered by Extension:UrlShortener. It was designed and approved in an RfC. URL shorteners are evil and problematic in general because they obfuscate the link target and if/when the shortener disappears, you have no way of accessing the destination. But people will use them anyways, so we created one that was less evil.
== Approved domains ==
URL shorteners are banned on most Wikimedia projects because they obfuscate the link target, making anti-spam work much harder. w.wiki is explicitly allowed because it has a list of approved domains, all of which go to Wikimedia sites.
To add a new domain to the list, it must meet the following criteria:
* No open redirects (allows bypassing the allowed domains list)
* No dangerous actions on GET requests
* No reflective XSS
* Should fall under the Wikimedia privacy policy or anyone who has access to private information (any PII such as IPs, User-agents, emails, etc.) should have signed an NDA.
*Running software that goes through the code review process or comes from a trusted upstream (e.g. Debian)
Perennial requests that have been declined include *.toolforge.org and *.wmflabs.org (T231518), translatewiki.net (T238413) and qrwp.org (T249416).
== Dumps ==
Weekly dumps are available with a mapping of short codes to targets: https://dumps.wikimedia.org/other/shorturls/
These are regularly uploaded to the Internet Archive.
WMDE

Wikimedia Deutschland (WMDE) is a chapter organization in the Wikimedia movement. In addition to a wide variety of services and support offered to German language projects they employ an engineering staff who are the primary maintainers of Wikibase and Wikidata.
This page is meant as a 'cheat sheet' page for WMDE related technical projects and related infrastructure.
Tidy documentation about products etc. should probably live on other wikis and just be linked to from here.
It is currently a work ...
== About Us ==
=== Contact ===
We have various public IRC channels
* 
* (mirrored to the WMDE mattermost)
=== People organization ===
* gerrit wmde group & sub groups https://gerrit.wikimedia.org/r/#/admin/groups/1450
* List of people in the WMDE ldap group https://tools.wmflabs.org/wmde-access/
=== Work Organization ===
WMDE mainly uses Phabricator to track it's projects.
TBA
TODO Links to teams? Boards, What is the campsite? How does the journey model work?
* (activity) #WMDE-Analytics-Engineering
* (activity) #WMDE-Design
* (activity) #WMDE-FUN-Team
* (activity) #WMDE-Fundraising-CN
* (activity) #WMDE-Fundraising-Frontend-Release
* (activity) #WMDE-Fundraising-Tech
* (activity) #WMDE-New-Editors-Banner-Campaigns
* (activity) #WMDE-Tech-Communication
* (activity) #WMDE-Tech-Communication-Data-Partnerships
* (activity) #WMDE-Tech-Communication-Mentoring-And-Events
* (activity) #Wikidata
** many sub-projects
== Product Umbrellas ==
=== Wikidata (tech docs) ===
* Wikidata.org
** Wikibase (repo (extension) & client (extension) & lib (extension)) (project)
*** Bridge
*** Tainted references
** Termbox
** WikibaseLexeme (extension)
** WikibaseMediaInfo (extension) - Maintained by the WMF Multimedia team
** ArticlePlaceholder (extension)
** PropertySuggester (extension)
** WikibaseQuality (project)
** WikimediaBadges (extension)
** WikibaseSchema (extension)
** Wikidata.org
* Wikidata.org clients
** Wikibase (repo & client & lib)
** WikimediaBadges
* Cognate (extension page) (tech docs)
* ORES (tech docs) - Maintained by WMF
* Wikidata query service - Maintained by the WMF Search Platform team
* Wikiba.se
===Technical Wishes (meta page)===
* RevisionSlider (project, extension, dashboard)
* FileImporter (extension) / FileExporter (extension) (project)
* TwoColConflict (project, extension)
* CatWatch (project)
* Wikidiff2 improvements (project)
* Deep cat
* Mention Notifications (project)
* Attribution Generator
* Watchlist refactoring (project)
=== Fundraising ===
* WMDE Fundraising application (github)
=== Misc ===
* Wikidata Concepts Monitor
* Analytics/WMDE
*Phragile
* https://tools.wmflabs.org/wmde-access/ - Maintained by Addshore
== Infrastructure ==
==Table of Content==
Go to the WMDE/TOC page to have a list of all pages we have under WMDE.
WMDE/Cognate

The Cognate DB runs on the extension1 cluster.
Currently the only Cognate db is called cognate_wiktionary.
== Use sql to get to the cognate tables ==
From a deployment host, connect to the extension1 cluster:
sql metawiki --cluster=extension1
Switch to the cognate DB:
use cognate_wiktionary;
Full example:
addshore@deploy1001:~$ sql metawiki --cluster=extension1
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 821580245
Server version: 10.1.36-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [metawiki]> use cognate_wiktionary;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [cognate_wiktionary]> show tables;
+------------------------------+
| Tables_in_cognate_wiktionary |
+------------------------------+
| cognate_pages |
| cognate_sites |
| cognate_titles |
+------------------------------+
3 rows in set (0.00 sec)
== Check if a site has been added to Cognate system ==
Example where a site has not been added:
MariaDB [cognate_wiktionary]> select * from cognate_sites where cgsi_dbname = 'xxxwiktionary';
Empty set (0.01 sec)
Example of an existing site:
MariaDB [cognate_wiktionary]> select * from cognate_sites where cgsi_dbname = 'zawiktionary';
+---------------------+--------------+----------------+
| cgsi_key | cgsi_dbname | cgsi_interwiki |
+---------------------+--------------+----------------+
| 9214146034733842265 | zawiktionary | za |
+---------------------+--------------+----------------+
1 row in set (0.00 sec)
== Adding a site to the Cognate system manually ==
The site should automatically be added when created as the script should be run as part of the addWiki.php process in WikimediaMaintenance https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/blob/master/addWiki.php#L277-L285
If it was not for some reason you can follow the below steps...
Run the maint script to add all sites for the group to the table from the maintenance server
addshore@mwmaint1002:~$ mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki yuewiktionary --site-group wiktionary
You can then check that the site exists in the cognate sites table (using instructions above)
If the site already has page creations (usually the case when you are doing this manually) you also need to populate the pages table for the site:
mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki yuewiktionary
== Catching up cognate after a period of x1 read only time (when writes were still happening on wiki) ==
Loop through all wiktionaries and populate the table (adding rows that don't exist but should.)
addshore@mwmaint1002:~$ foreachwikiindblist wiktionary extensions/Cognate/maintenance/populateCognatePages.php --batch-size 1000
WMDE/Wikidata

This page is intended to serve as a central hub for documentation related to the production deployment, infrastructure and maintenance of Wikidata.
* /Alerts, covering Icinga and Grafana alerts
* /Runbooks, including detailed step-by-step instruction in case of alerts or other tasks
* /Caching, covering Varnish, ParserCache & Memcached
* /Deployment, deployment related tasks
* /Dispatching, the "dispatching" system that runs on a `mwmaint` host pushing changes out to sister projects & triggering page purges
* Dumps/WikidataDumps, about the Wikidata entity dumps
* /Growth, predicted growth in various areas
* /Maintenance, covering various tasks related to maintaining the content infrastructure and production deployment of Wikidata
* /Scaling, current and future scaling issues
* /SQL, what tables we have in production, and some details
* /SSR_Service, the SSR service for Wikidata
Cloud VPS Projects:
* Nova_Resource:Wikidata-dev, a general use project for Wikidata / Wikibase usage
==Table of Contents==
Go to the WMDE/Wikidata/TOC page to have a list of all pages we have under WMDE/Wikidata.
==See also==
- Wikidata/Wikibase team site
WMDE/Wikidata/Alerts

==Alertmanager==
Wikidata contact information for Alertmanager is set in https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/alertmanager/templates/alertmanager.yml.erb
The status of alerts can be seen at https://alerts.wikimedia.org/?q=team%3Dwikidata
Internally in WMDE there is a wikidata-monitoring mailing list you can subscribe to.
All alerts coming from grafana with tag of team: "wikidata-team" and receiver of Alertmanager will end up contacting the wikidata-monitoring internal mailing list. (See Alertmanager for more information on how to setup alerts that would contact the team).
==Grafana==
Alertmanager handle wikidata alerts dashboard on Grafana.
The dashboard can be found here: https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts
=== Maxlag: Above 10 for 1 hour ===
* More specific information about the lag can be queried from the API: https://www.wikidata.org/w/api.php?action=query&maxlag=-1
* More data on the query service: https://grafana.wikimedia.org/d/000000489/wikidata-query-service
* SAL can be used to track down changes to the problematic servers
* dispatch lag is no longer part of max-lag and now tracked/alerted with its own metric
* contains people that know more about running/administrating the wdqs servers
In the past this has been caused by:
* lag on queryservice servers (see: )
* wdqs host being restarted, but no depooled (see: )
=== Edits: Wikidata edit rate ===
The edit rate on Wikidata can be a good indicator that something somewhere is wrong, although it will not always indicate exactly what that is.
You can view the edits dashboard at https://grafana.wikimedia.org/d/000000170/wikidata-edits
If MAXLAG is high, that might be a reason for low edit rate.
You may want to investigate what is going on with the API (as all edits go via the API) https://grafana.wikimedia.org/d/000000559/api-requests-breakdown?refresh=5m&orgId=1&var-metric=p50&var-module=wb*
=== API: Max p95 execute time for write modules ===
Investigate the wb api @ https://grafana.wikimedia.org/d/000000559/api-requests-breakdown?refresh=5m&orgId=1&var-metric=p50&var-module=wb*
In the past this has been caused by:
* s8 db being overloaded, often for a fixable reason
* Memcached being overloaded, in the past indicating UBNs
=== Termbox Request Errors ===
This kind of error occurs when Wikibase is unable to reach the Termbox Service, i.e. the HTTP request itself fails and is unlikely to have reached its destination. This error does *not* get triggered by erroneous responses, so it means there is a problem on the MediaWiki/Wikibase side or network issues.
=== Change Dispatching ===
See WMDE/Wikidata/Dispatching and Wikibase: Change propagation. More metrics can be found on https://grafana-rw.wikimedia.org/d/hGFN2TH7z/edit-dispatching-via-jobs
See WMDE/Wikidata/Runbooks/Change dispatching/Alert for a details on the alerts.
==Oozie Job==
Sometimes these jobs will fail for random reasons.
They will be restarted, so no need to worry on a first failure.
If things continue to fail, contact WMF analytics to investigate on IRC in #wikimedia-analytics.
WMDE/Wikidata/Convert Properties

= Convert Properties to External IDs =
== Preparation ==
* Make sure you belong to the restricted or deployment user group on production machines.
== Converting Data Types ==
Any string based data type can be converted into an External ID. To convert a property's data type to External ID:
1. Log in to a maintenance host
 $ ssh maintenance.eqiad.wmnet
2. For each property to convert, run the following script with the property ID (double check that the ID is correct) and Phabricator task:
 $ mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki=wikidatawiki --property-id P12345 --new-data-type external-id --summary '[[phabricator:T1234567|T1234567]]'
3. Go to Wikidata to confirm that the provided Property ID's data type has been converted to External ID
WMDE/Wikidata/PropertySuggester update

Occasionally, the data for the property suggester needs to be updated from the latest JSON dumps.
This process requires access to a Wikimedia deploy host (deployment.eqiad.wmnet).
Please note that occasionally this property suggester will be referred to as the 'legacy property suggester'. However as of 2024 December, the new property suggester has not yet been deployed.
== One-time setup ==
* git clone https://github.com/mariushoch/wbs_propertypairs-refine.git
== Each update ==
* Find the latest wbs_propertypairs on https://analytics.wikimedia.org/published/datasets/wmde-analytics-engineering/Wikidata/wbs_propertypairs/ (generated on stat1005 by a cron from ladsgroup). We’ll use yyyymmdd as a placeholder for its name below.
* Pull analyzed-out.gz to your local machine, apply wbs_propertypairs-refine refine.py (README) and commit to the wbs_propertypairs repo with the commit message Add propertypairs from the yyyymmdd dump.
* Load it down to the deployment host with https_proxy=http://webproxy.eqiad.wmnet:8080 wget 'https://github.com/wmde/wbs_propertypairs/raw/master/yyyymmdd/wbs_propertypairs.csv.gz'.
* Unpack it: gzip -d wbs_propertypairs.csv
* Update the actual table: mwscript-k8s --attach -- extensions/PropertySuggester/maintenance/UpdateTable.php --wiki wikidatawiki --file php://stdin < wbs_propertypairs.csv.
** This will take ca. 10 minutes.
** It will first log (to your terminal) a bunch of “Deleting a batch” lines, then “X rows inserted” up to the total number of lines in the CSV file (which you can count with wc -l wbs_propertypairs.csv beforehand).
* Log your changes: !log Updated the Wikidata property suggester with data from yyyymmdd's JSON dump
WMDE/Wikidata/SQL

==DB locations==
* wikidatawiki database is stored on the s8 WMF shard.
* testwikidatawiki is on the s3 WMF shard.
* commonswiki is on the s4 WMF shard.
* Full text is stored on the same MariaDB#External_storage as other wikis.
== Wikidata specific tables ==
You can read more details in the docs https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_storage.html
=== Repo tables===
====Change dispatch related tables:====
* wb_changes
* wb_changes_dispatch
* wb_changes_subscription
====Secondary terms index====
*wbt_item_terms
*wbt_property_terms
*wbt_term_in_lang
*wbt_text_in_lang
*wbt_text
*wbt_type
====Other:====
*wb_id_counters
**Simple tables storing counters for wikibase entity ids
*wb_items_per_site
**Links client site + title pairs to item ids.
**NOTE: This reflects the sitelinks found on items
*wb_property_info
**Secondary storage for property info (datatype, formatter url etc)
*wbqc_constraints (WikibaseQualityConstraints)
*wbs_propertypairs (PropertySuggester)
</pre>
=== Client tables ===
* wbc_entity_usage
**Data usage tracking system for client sites.
WMDE/Wikidata/SSR Service/sequence

Compatible with:
* https://sequencediagram.org/
title Wikibase Termbox Server Side Rendering
actor Browser
participant MediaWiki PHP
participantgroup 
 participant SSR
 participant SSR Proc Cache
end
participant MW / WB APIs
Browser->MediaWiki PHP:/wiki/Q123
MediaWiki PHP->SSR:Q123, en, ...
SSR->MW / WB APIs:/wiki/Special:EntityData/Q123.json
MW / WB APIs-->>SSR:Entity data (JSON)
SSR->SSR Proc Cache:Get Content Languages
SSR Proc Cache->MW / WB APIs:<align:center>/w/api.php?action=query&\nmeta=wbcontentlanguages</align>
SSR Proc Cache-->>SSR:language data
note over SSR Proc Cache,MW / WB APIs:requests can be combined
SSR->SSR Proc Cache:Get Messages
SSR Proc Cache->MW / WB APIs:<align:center>/w/api.php?action=query&\nmeta=allmessages\n(specific langs & messages)</align>
SSR Proc Cache-->>SSR:messages
SSR-->>MediaWiki PHP:HTML of termbox
MediaWiki PHP-->>Browser:page HTML
WebPageReplay

== Background ==
In the path to have more stable metrics in our synthetic testing we have been trying out mahimahi, mitmproxy and WebPageReplay to record and replay Wikipedia. For mahimahi we have used patched version fixed by Gilles over Benedikt Wolters HTTP2 version of https://github.com/worenga/mahimahi-h2o. With mitmproxy and WebPageReplay we use the default version. The work has been done in T176361.
We have put mahimahi on ice because it is too much of hack to get HTTP/2 to work at the moment and WebPageReplay works out of the box with HTTP/2. mitmproxy worked fine but offered no clear benefit over WebPageReplay. 
=== Replaying vs non replaying ===
Let us compare what the metrics looks like comparing WebPageTest vs WebPageReplay (Chrome).
== WebPageReplay setup ==
The current version run that collects the data for https://grafana.wikimedia.org/d/IvAfnmLMk/page-drilldow is a Docker container with this setup:
We run all tests on a couple of bare metal servers. We have tried running the same code on AWS, WMCS and Google Cloud and in all those cases the metrics stability over time was at least 2 to 4 times worse than AWS and over time the metrics are most stable on bare metal.
=== Servers ===
We run tests from four server at the moment:
* hetzner1 (195.201.12.112) - Run Chrome tests for desktop against enwiki.
* hetzner2 (138.201.207.48) - Run Chrome tests for desktop against group 0 and group 1.
* hetzner3 (136.243.175.100) - Run Firefox tests for desktop against group 0, group 1 and enwiki.
* hetzner4 (88.99.26.157) - Run emulated mobile tests for group 0, group 1 and enwiki.
=== Upgrade to a new version ===
Checkout WebPageReplay/Runbook#Update to new version.
=== Alerts ===
We also run alerts on the metrics we collect from WebPageReplay. Checkout Performance/Guides/WebPageReplay alert.
== Maintenance ==
=== Debug missing metrics ===
Checkout WebPageReplay/Runbook#Debug missing metrics
== See also ==
* WebPageReplay/Runbook
* Performance/Guides/WebPageReplay alert
Category:Wikimedia Performance