nyuuzyou/wikis huggingface dataset:
Ms3

Linux setup with 14 RAID-1 volumes of 3 disks each (spread over controllers), striped together in an LVM volume group.
== Disk setup ==
# The first disks of the first and second controller,
# /dev/sda and /dev/sdi respectively, are used as host OS drives.
# The first disks of the remaining controllers will be used as spares.
#
# (md7) | h | p | x || af | an | av | (md15)
# (md6) | g | o | w || ae | am | au | (md14)
# (md5) | f | n | v || ad | al | at | (md13)
# (md4) | e | m | u || ac | ak | as | (md12)
# (md3) | d | l | t || ab | aj | ar | (md11)
# (md2) | c | k | s || aa | ai | aq | (md10)
# (md1) | b | j | r || z | ah | ap | (md9) 
# (md0) | a | i || q | y | ag | ao | (spares, md8 missing)
#
# c0 c1 c2 c3 c4 c5
Output of /home/mark/thumper-es-raid-lvs-setup.py:
mdadm --create --level 1 --raid-devices 3 /dev/md1 /dev/sdb /dev/sdj /dev/sdr
mdadm --create --level 1 --raid-devices 3 /dev/md9 /dev/sdz /dev/sdah /dev/sdap
mdadm --create --level 1 --raid-devices 3 /dev/md2 /dev/sdc /dev/sdk /dev/sds
mdadm --create --level 1 --raid-devices 3 /dev/md10 /dev/sdaa /dev/sdai /dev/sdaq
mdadm --create --level 1 --raid-devices 3 /dev/md3 /dev/sdd /dev/sdl /dev/sdt
mdadm --create --level 1 --raid-devices 3 /dev/md11 /dev/sdab /dev/sdaj /dev/sdar
mdadm --create --level 1 --raid-devices 3 /dev/md4 /dev/sde /dev/sdm /dev/sdu
mdadm --create --level 1 --raid-devices 3 /dev/md12 /dev/sdac /dev/sdak /dev/sdas
mdadm --create --level 1 --raid-devices 3 /dev/md5 /dev/sdf /dev/sdn /dev/sdv
mdadm --create --level 1 --raid-devices 3 /dev/md13 /dev/sdad /dev/sdal /dev/sdat
mdadm --create --level 1 --raid-devices 3 /dev/md6 /dev/sdg /dev/sdo /dev/sdw
mdadm --create --level 1 --raid-devices 3 /dev/md14 /dev/sdae /dev/sdam /dev/sdau
mdadm --create --level 1 --raid-devices 3 /dev/md7 /dev/sdh /dev/sdp /dev/sdx
mdadm --create --level 1 --raid-devices 3 /dev/md15 /dev/sdaf /dev/sdan /dev/sdav
spares:
mdadm --add /dev/md1 /dev/sdq /dev/sdy /dev/sdag /dev/sdao
pvcreate:
pvcreate /dev/md1
pvcreate /dev/md2
pvcreate /dev/md3
pvcreate /dev/md4
pvcreate /dev/md5
pvcreate /dev/md6
pvcreate /dev/md7
pvcreate /dev/md9
pvcreate /dev/md10
pvcreate /dev/md11
pvcreate /dev/md12
pvcreate /dev/md13
pvcreate /dev/md14
pvcreate /dev/md15
vgcreate:
vgcreate data /dev/md1 /dev/md2 /dev/md3 /dev/md4 /dev/md5 /dev/md6 /dev/md7 /dev/md9 /dev/md10 /dev/md11 /dev/md12 /dev/md13 /dev/md14 /dev/md15
lvcreate:
lvcreate -i 14 -I 64 -n ES data
== bonnie++ ==
Version 1.03b ------Sequential Output------ --Sequential Input- --Random-
 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ms3 64608M 414517 45 147501 24 364129 33 539.1 1
 ------Sequential Create------ --------Random Create--------
 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
 16 2085 2 +++++ +++ 9414 18 7345 33 +++++ +++ 5624 19
ms3,64608M,,,414517,45,147501,24,,,364129,33,539.1,1,16,2085,2,+++++,+++,9414,18,7345,33,+++++,+++,5624,19
Category:Servers
Mw1017

mw1017 was one of the many application server. While it received code during deployments as any other app server, this server was different in that the load balancer only served traffic with a X-Wikimedia-Debug header to this server.
See X-Wikimedia-Debug#Available backends for current debug servers.
== See also ==
* Debugging in production
* X-Wikimedia-Debug
* mw1099
Mwdebug

== MediaWiki on Kubernetes ==
As part of MediaWiki On Kubernetes, two mw-debug deployments exist (one in each datacenter). These deployments can be reached via the browser through the k8s-mwdebug option for WikimediaDebug (for the current datacenter), or k8s-mwdebug-eqiad or k8s-mwdebug-codfw in the rare case that you want a particular one.
=== Connecting ===
* REPL
* Shell (production root only)
== Bare metal ==
mwdebug hosts are a small cluster of application servers set aside for use with WikimediaDebug.
The servers run on shared Ganeti hardware rather than baremetal and have a few additional packages installed to aid in debugging (such as php-tideways).
We currently have:
* mwdebug1001.eqiad: Ad-hoc testing
* mwdebug1002.eqiad: Stage during deployment for Eqiad.
* mwdebug2001.codfw: Ad-hoc testing.
* mwdebug2002.codfw: Stage during deployment for Codfw.
=== Connecting ===
== Services ==
* WikimediaDebug
== History ==
* mwdebug deployments of Mediawiki on Kubernetes were made available in 2021
* Debug hosts were reimaged with buster in Feb 2021.
* mwmdebug1003 was a WikimediaDebug host in Eqiad, used for testing Debian Buster for a while when most servers ran Stretch. It was later used for testing Stretch when most servers were Buster. It was deleted in 2020 per T267248.
Category:Servers
Category:Eqiad data center
Category:Codfw data center
Mwlog

mwlog are a class of logging hosts which serve multiple functions.
== Hosts ==
* mwlog1002 (Eqiad)
* mwlog2002 (Codfw)
== Services ==
=== udp2log/mwlog ===
Ingest and storage of "udp2log" logs, in flat file format in /srv/mw-log.
These are logs transported over udp/8420 to an rsyslog listener on the mwlog hosts which mirrors the logs to the udp2log listeners on all mwlog hosts (udp/8421)
udp2log/mwlog producers are configured in various different ways: 
* Mediawiki uses hard coded IPv4 addresses (intentionally, for performance reasons) for the mwlog destinations, configured in mediawiki-config/wmf-config/ProductionServices.php
* Some services (including rsyslog which relays on behalf of php-fpm and apache2 ) use the udplog.(eqiad|codfw).wmnet DNS records. Updating the udplog DNS record will require an rsyslog restart across the fleet.
* Some services are defined via puppet hiera (can be found with e.g. git grep mwlog1002 in operations/puppet)
=== Redis server for Arc Lamp ===
The primary (eqiad) mwlog host recieves sampling profiles from MediaWiki app servers (collected by Excimer), which are then subscribed to by Arc Lamp (formerly known as "webperf" or "xenon").
=== Long-term security storage ===
Currently /srv/security on the mwlog primary (eqiad) is used as a long-term store for security log archives.
== Utilities ==
* logspam-watch displays and filters a summary of recent logs from disk - see Wikimedia binaries#mwlog-server
Category:Services
Mwmaint1001

mwmaint1001 was the maintenance host for Eqiad. It replaced terbium in July 2018. It has been replaced by mwmaint1002 in October 2018.
== Connecting ==
== Services ==
* noc.wikimedia.org
* maintenance scripts: People log in here to run shell scripts that take a lot of time or CPU, or to run various MySQL queries.
== See also ==
* Maintenance server
* deployment.eqiad.wmnet, for deployments.
Mwmaint2001

mwmaint2001 was the maintenance script host for Codfw. It was previously called wasat. It was replaced by mwmaint2002 (T275905).
== Connecting ==
== Services ==
* noc.wikimedia.org
* maintenance scripts: People log in here to run shell scripts that take a lot of time or CPU, or to run various MySQL queries.
== See also ==
*mwmaint1002, maintenance script host in Eqiad.
Mysql.py

mysql.py is a wrapper around the mysql command line client from mariadb, which is tuned for WMF production admin convenience. It is intended primarily for interactive usage, so it defaults to WMF configuration and saves typing. It requires sudo or root user privileges, to read /root's config. It is backwards compatible with the mysql command line client, but has some additional configuration changes and shortcuts:
* It will try to force the use of TLS for any connections outside of localhost (for security), using the right CA (independently of the /etc/ configuration.
* It will use current user's .my.cnf configuration, so typically you want to run it with sudo meaning it will try to connect as root and with the appropriate root password, depending on the role (mediawiki db vs wikireplicas, etc.)
* It will try to complete the fully qualified name of the host, with just the hostname: So mysql.py -h db1215 will be interpreted as if one had written mysql -h db1215.eqiad.wmnet
* It will make the port easier to write, so if one writes mysql.py -h db1215:3315 it will be as if one writes mysql -h db1215.eqiad.wmnet -P 3315
* It will translate sections names into ports, based on the /etc/wmfmariadb/section_ports.csv file, generated from puppet's hiera: So if one writes mysql.py -h db1215:s5 it will be as if one writes mysql -h db1215.eqiad.wmnet -P 3315
mysql.py is available on cumin hosts (mysql management profile) as it requires private account information access to execute database server admin account queries: .
== Example usage ==
Connect quickly to a server:
 mysql.py -h db1215
Connect quickly to a multinstance server:
 mysql.py -h db1215:3315
Alternative syntax:
 mysql.py -h db1215:s5
All other mysql parameters can still be used, they are just passed on:
 mysql.py -A -h labsdb1009 enwiki_p
Faster way to query 1 time things:
 mysql.py -h db1215 -u dump -e "SELECT @@read_only" -p
Combined with lists of host:section or host:port, it can do batch requests:
 ./section s5 | while read host; do mysql.py -h $host -e "SELECT @@hostname, @@port"; done
== Development ==
Its code can be found on the WMFMariaDBpy repo.
Category: MySQL
NFS-Ganesha

NFS-Ganesha is an NFS fileserver that runs in user mode on most UNIX/Linux systems and supported NFS v3, v4, and v4.1 protocols.
== Releases ==
Users are encouraged to use the latest version due to resource constraints in supporting older releases.
The release history seems to suggest that there are no backports to older releases (i.e., once v2.6.0 is released there won't be a v2.5.x release).
Date 	Release	Comments
2018-09-14	2.7.0 	
2018-08-22	2.6.3 	
2018-08-22	2.6.3 	
2018-04-09	2.6.2 	
2018-03-20	2.6.1 	
2018-02-20	2.6.0 	
2018-01-27	2.5.5 	
2017-11-10	2.5.4 	
2017-10-09	2.5.3 	
2017-08-23	2.5.2 	
2017-07-21	2.5.1 	
2017-06-09	2.5.0 	
2017-04-05	2.4.5 	
2017-03-21	2.4.4 	
2017-02-07	2.4.3 	
2017-01-23	2.4.2 	
2016-10-28	2.4.1 	
2016-09-22	2.4.0 	
== Packages ==
Debian packages are published by the upstream project at https://download.nfs-ganesha.org and built from the nfs-ganesha-debian repository.
Available packages:
* Buster: up to 2.7.0
* Stretch: up to 2.7.0
* Jessie: up to 2.4.5
Packages are currently not available from Wikimedia's APT repository.
NIS

:This is now outdated. We no longer use NIS, or LDAP (for production shell), and manage user accounts and /home access with Puppet.
NIS (formerly YP) is a directory service for Unix databases (passwd, group, hosts, etc). it's like LDAP but more simpler and more limited. we are replacing LDAP with NIS.
we run the NIS server, ypserv, on srv1 (the master) and srv2. the NIS databases are in /var/yp. to make changes, edit ypfiles/* on srv1, and run "make" to rebuild the databases and update slaves.
== Adding a user ==
For LDAP, adding the user on srv1 and running ldapsync seems sufficient. This doesn't appear to affect NIS.
* Edit /var/yp/ypfiles/passwd, group, and shadow and manually add the entries...
* sacrifice goat
* run 'make' in /var/yp
* profit!
== Client setup ==
the NIS client is called ypbind, and runs on each server.
ubuntu: 
* apt-get install nis
* edit /etc/yp.conf and add:
 ypserver srv1.pmtpa.wmnet
 ypserver srv2.pmtpa.wmnet
* edit /etc/nsswitch.conf and change:
 passwd: files nis
 group: files nis
 shadow: files nis
* make sure /etc/defaultdomain contains "pmtpa.wmnet"
* /etc/init.d/nis start
fedora:
* edit /etc/sysconfig/network:
 NISDOMAIN=pmtpa.wmnet
* edit /etc/yp.conf and add:
 ypserver srv1.pmtpa.wmnet
 ypserver srv2.pmtpa.wmnet
* edit /etc/nsswitch.conf and change:
 passwd: files nis
 group: files nis
* chkconfig --level 2345 ypbind on
* /etc/init.d/ypbind start
NRPE

NRPE stands for Nagios Remote Plugin Executor.
It's a service running on monitored hosts. An Icinga server can connect to it and tell it to locally execute
pre-defined commands.
This way Icinga can monitor things that can't be checked from external but need to run something on hosts.
Not to be confused with NSCA / passive checks.
It is important for security that sending command arguments from remote isn't allowed. All commands are predefined.
They can be found in /etc/nagios/nrpe.d/ on the monitored hosts. On the icinga server side there will be a reference
to them using check_nrpe.
The service is called nagios-nrpe-server, so status check is: systemctl status nagios-nrpe-server.
Sometimes a nagios-nrpe-server process gets killed due to OOM for some other reason. This manifests as ALL the checks executed
via NRPE failing with CRIT but with an extra "could not connect to NRPE"-like message. Restarting the service
should make them all recover.
Restart is systemctl status nagios-nrpe-server. 
Category:Monitoring
Nagios-wm

nagios-wm is an IRC bot that outputs Nagios changes on channels.
nagios-wm [~nagios-wm@spence.wikimedia.org]
ircname : IRC echo bot
channels : +#wikimedia-operations +#wikimedia-tech 
It lives on spence and runs from user "nobody". Use the init script /etc/init.d/ircecho to stop, start and restart it.
Category:Bots
Ncredir

Ncredir is our Non-Canonical Redirect service. It handles traffic from the outside world to a list of domain names which we own but are not primary canonical domain names for our projects. A key example would be wikipedia.com.
It is separate from our primary edge traffic clusters for the canonical domains, running on independent instances and public IPs. Under normal conditions, it gets little traffic.
== Components ==
ncredir is implemented using a combination of acme-chief managed certificates and nginx. nginx redirects are created in the Puppet repository according to the rules laid out in the nc_redirects.dat file. nc_redirects.dat is used as input for the the custom compile_redirects() function in Puppet which outputs redirects as nginx configuration.
=== Redirection logic ===
Nginx is fed with two maps containing the redirection logic. The first map populates a variable called $override, and the second one a variable called $rewrite.
The first map populating $override is generated with the override stanzas contained in the redirects definition file, while the $rewrite map is populated with the funnel and rewrite stanzas from the definition file.
This mapping between the nc_redirects.dat file and nginx happens on puppet compilation time. So in the ncredir servers only nginx + the acme-chief managed certs are needed to run the service.
=== TLS ===
ncredir does not utilize the CDN clusters; It handles its own TLS termination. ncredir exposes itself to live traffic using the high-traffic1 class in LVS. The service is geographically balanced via GeoDNS with the ncredir-lb.wikimedia.org record which balances the traffic across:
* ncredir-lb.codfw.wikimedia.org
* ncredir-lb.eqiad.wikimedia.org
=== Logging ===
ncredir uses fifo-log-demux to provide logging through pipes. Providing pipes allows for real-time debugging without needing to store logs locally on the server and compromise user privacy.
To view the real-time logs, run:
# fifo-log-tailer -socket /var/log/nginx/ncredir.access_log.socket
An alias is provided for convenience:
# ncredirlog-access_log
== See also ==
* fifo-log-demux repository: Used for nginx log reading
Category:Services
Category:SRE Traffic
Neon

neon is a kubernetes::staging::master
== Services ==
role(kubernetes::staging::master)
== Former Services on old host neon ==
The host name "neon" has been reused for different purposes. In the past it was a monitoring server hosting the following services:
* Icinga
* logmsgbot: tcpircbot IRC relay for syncs / scaps
* tendril.wikimedia.org
Category:Kubernetes
Netconsole

== What is netconsole? What is it used for? ==
Quoting the manual: 
This module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical.
It can be used either built-in or as a module. As a built-in, netconsole initializes immediately after NIC cards and will bring up the specified interface as soon as possible. While this doesn't allow capture of early kernel panics, it does capture most of the boot process.
== Production setup ==
The syslog::centralserver hosts run a rudimentary but fully working netconsole server to receive kernel logs. Clients send logs through syslog anycast IP address (syslog.anycast.wmnet) on port 6666, therefore the logs might land in eqiad or codfw syslog hosts.
To display the netconsole log into centrallog hosts and issue journalctl -u netconsole
As of Dec 2021 netconsole is opt-in for clients, see also the documentation on profile::netconsole::client on how to enable it for your role.
== Manual setup ==
This is a quick guide to set up a rudimentary netconsole logger over netcat just for a punctual requirement. 
There is no need to install extra tools for this setup.
For this testing we will use the following two servers
 db2034 - server that needs troubleshooting
 db2048 - server that will receive the messages
On db2048 we need to set up a netcat listening on any port, let's use 6666 and let's redirect that to a logfile so we can check it later:
 db2048: nc -l -u -p 6666 | tee /var/log/netconsole.log
From db2034 we can test if the port is able to receive UDP connections:
Now let's configure netconsole on db2034 to send the messages to db2048.
If possible, use a host on the same subnet, to simplify things
Data we need before we set up the kernel module:
 db2034 IP: 10.192.32.5
 db2034 MAC: a0:d3:c1:fa:2d:80
 db2048 IP: 10.192.32.108
Let's go ahead and install the kernel module:
 modprobe netconsole netconsole=6666@10.192.32.5/eth0,6666@10.192.32.108/40:a8:f0:35:22:8c
Now we can check that it was enabled
 db2034: dmesg | tail -n5
 <snip>
 [ 5205.486828] console [netcon0] enabled
 [ 5205.486829] netconsole: network logging started
We are using the same local and remote port (6666 to simplify things).
After that we are saying that db2034's ip will send stuff from eth0 via 6666 to db2048's IP and it should expect db2034's mac
Before we are ready to test if it works, let's change the logging level on db2034 (so it sends everything to console)
 db2034:dmesg -n 8
== How can I test if it is working: ==
Let's try to log a test message to see if we can see it on db2048:
 root@db2034:/srv# modprobe floppy
 modprobe: ERROR: could not insert 'floppy': No such device
 root@db2034:/srv#
And we can see it on the other side:
 root@db2048:~# nc -l -u -p6666 | tee /var/log/netconsole.log
 [ 8644.321281] Floppy drive(s): fd0 is 1.44M
 [ 8647.363068] floppy0: no floppy controllers found
Network Error Logging

== Intro ==
There are many classes of reliability issues (e.g. failures/misconfigurations in intermediate networks) that we only find out about via direct, manual reports from users, or, for very widespread cases, we notice because traffic is 'missing' and below expected rates.
Many modern browsershttps://caniuse.com/?search=nel support a feature called Network Error Logging, or NEL. On successful requests, we ask browsers to remember "if you later encounter an error talking to us, let an error reporting endpoint know".
Asking browsers to enable NEL is implemented by serving HTTP response headers Report-To and NEL, which together define a set of endpoints that can receive reports, sampling fractions for each of failures and successes, and a TTL for this entire definition to be stored in the user's browser. See also Sample Policy Definitions. 
Our policy sets a 5% failure_fraction -- a tradeoff between fidelity and load placed on Logstash.
We use a GeoDNS trick to receive reports more promptly: gdnsd is configured to send the "next-best" edge site for a given user for the intake-logging.wikimedia.org domain where reports are received. This is because if a user needs to send a NEL report, it's very likely that from their perspective something is wrong with their usual 'primary'/closest site.
That being said, browsers will buffer reports and retry submission later if they are unable to send them immediately. No absolute time is included, due to both clock skew and privacy fingerprinting concerns, but the age field of the report indicates the number of milliseconds elapsed between generation and successful submission. EventGate timestamps each report in the _dt field.
== Dashboards ==
Logstash data
When diagnosing network connectivity issues, especially focus on sudden increases in tcp.timed_out reports, as well as tcp.address_unreachable.
== Serving the necessary response headers ==
For our implementation, sub https_deliver_networkerrorlogging in wikimedia-frontend.vcl.erb.
== Our report receiver implementation ==
TODO EventGate, Kafka, logstash
Category:SRE Infrastructure Foundations
Network design

The Wikimedia Foundation operates a network made of 2 main datacenters (Ashburn, Dallas), 5 caching sites (San Francisco, Amsterdam, Singapore, Marseille, São Paulo) and 2 networking sites (Dallas, Chicago).
Since 2017, this network is unified under AS14907.
Those sites are connected in a redundant way to each other using dedicated fibers, wavelengths and VPLS circuits.
Connectivity to the world is achieved via various transit providers and exchange points.
== USA ==
== Europe ==
== Asia-Pacific ==
==See also==
*Network cheat sheet
*Network design (historic) - 2015 and earlier
*Network design - Eqiad Expansion EVPN
*Network design - Eqiad WMCS Network Infra
Category:Network
Category:SRE Infrastructure Foundations
Network design (historic)

== AS 14907 ==
The US network
 
=== Subnets ===
==== eqiad ====
subnet 	vlan ID	IPv4 	IPv6
public1-a-eqiad 	1001 	208.80.154.0/26 	2620:0:861:1::/64
public1-b-eqiad 	1002 	208.80.154.128/26	2620:0:861:2::/64
public1-c-eqiad 	1003 	208.80.154.64/26 	2620:0:861:3::/64
public1-d-eqiad 	1004 	208.80.155.96/27 	2620:0:861:4::/64
private1-a-eqiad	1017 	10.64.0.0/22 	2620:0:861:101::/64
private1-b-eqiad	1018 	10.64.16.0/22 	2620:0:861:102::/64
private1-c-eqiad	1019 	10.64.32.0/22 	2620:0:861:103::/64
private1-d-eqiad	1020 	10.64.48.0/22 	2620:0:861:104::/64
* cr1-eqiad
* cr2-eqiad
* cr1-ulsfo
== AS 43821 ==
The European network.
=== 2010 ===
The purchase of several Juniper EX4200s in a stack, for extra access ports for servers, also brings some opportunities w.r.t. the network topology. Since the EX4200s have excellent L3 support they can help create redundancy.
The 2nd dark fiber is linked between br1-knams and csw2-esams to create a ring. csw1-esams and csw2-esams can then share responsibility as core switches, for inter-vlan routing and switching, using VRRP. Since an EX4200 can not install a full BGP routing table in FIB, it defaults to either of the two Foundry routers using OSPF.
Toolserver can be connected redundantly as well, using (R)STP to both core switches and VRRP, or alternatively a LAG to the EX4200 stack.
=== 2009 ===
Temporary situation after the move from knams to esams. The network is split, with a new Foundry BigIron RX-4 as a pure router at knams for external connectivity, with Telia, DataHop, Init7 (partial) transit, and 2x 1 Gbps AMS-IX for peering. Connectivity between the two sites is supplied by a 10GBase-ER link over dark fiber, and a 3 Gbps MPLS backup link. A second dark fiber is being installed to form a ring.
=== 2008 ===
BGP default transit from AS1145 (Kennisnet), with some partial transit and peering over a 1 Gbps AMS-IX link. Everything on one core router/switch, csw1-knams (Foundry BigIron RX-8).
== See also ==
* Network cheat sheet
Category:Network
Network design - Eqiad Expansion EVPN

This page will provide an overview of the WMF switch fabric networking as deployed in the 'new cage' (rows E/F) in Eqiad (Equinix Ashburn).
==Detailed Design==
While this page provides covers the main points of relevance to outside teams, more detail is available in the Google doc.
https://docs.google.com/document/d/1n_0H91eO4oM8F85XEin04QUGiOaki1CfxR2xZhgA6D8/edit#heading=h.zievftjj9d40
==Testing==
Network functionality and failover testing is documented on the VXLAN-EVPN Network Testing - Eqiad Expansion page.