nyuuzyou/wikis huggingface dataset:
SRE/Service Operations/Ownership

Service/Procedure 	Description 	Phabricator tag	Notes
WikiKube Kubernetes Cluster	Kubernetes (often abbreviated k8s) is an open-source system for automating deployment, and management of applications running in containers. 	 	
Mediawiki servers 	The Application servers (or app servers) are the several hundred Apache servers that run the MediaWiki backend software (written in PHP). 	 	
Memcached for MediaWiki 	There are two logical pools of memcached servers for MediaWiki. There are critical for performance of all sites and used extensively 	 	
Redis Misc 	Redis is used in Wikimedia production for:
changeprop (role::redis::misc)
As a cache and queue backend in ORES
Receiver of sampled profile data from PHP, as part as the sampling/profiling pipeline (Arc Lamp). 	 	
Shellbox 	Shellbox is a library for remote command execution, and a server for secure command execution. It was primarily implemented to sandbox lilypond (used by the Score extension) and provide a way for MediaWiki to utilize external binaries without needing them to be in the same container. Shellbox relies on Kubernetes (and Linux containers/namespaces) to provide isolation and resource limits for external commands. 	 	
Datacenter Switchover 	A datacenter switchover (from eqiad to codfw, or vice-versa) comprises switching over multiple different components, some of which can happen independently and many of which need to happen in lockstep. This page documents all the steps needed to switch over from a master datacenter to another one, broken up by component. SRE Service Operations maintains the process and software necessary to run the switchover.	 	
Service Level Objectives 	Service Level Objective (SLO) and Service Level Indicators (SLI) 	 	
Kafka-main 	kafka-main is the low-volume, critical production services cluster. Talk to us before starting to send events there. kafka-main is currently used directly by Event_Platform/EventGate and change-propagation.
SRE/Traffic

__NOTOC__
Documentation	Team members	Currently working on
We are a remote-first, globally distributed team responsible for designing and operating a private and privacy-protecting global CDN for Wikipedia and related sister projects. Our Traffic stack handles public DNS, global request routing, TLS termination, HTTP content caching, and internal service routing for the public services of the Foundation, and is a critical layer in bringing our free content to our global audiences.
You can reach us at:
*IRC: 
*Email: SRE-Traffic Mailing List
*Bug reporting: traffic Phabricator tag
== What we own ==
Service 	Description
Acme-chief 	Centrally request configured TLS certificates from ACME servers, then make the public and private parts available to authorized API users.
Apache Traffic Server	aka ATS, is a caching HTTP proxy used as the backend (on-disk) component of Wikimedia's CDN. In-memory, ephemeral caching is done by cache frontends running Varnish..
DNS 	Wikimedia uses two separate kinds of the DNS servers, authoritative nameservers (that respond to queries from third party nameservers for our domains) and recursive resolvers (that resolve DNS queries when any of our servers need to look up a name)
HAProxy 	TLS and HTTP/2 termination at the CDN
LVS 	A high-traffic Layer 4 load balancer.
Ncmonitor 	Monitor Wikimedia domains and sync with downstream services
Ncredir 	Non-Canonical Redirect service (e.g. wikipedia.com to wikipedia.org)
Purged 	Daemon that reads Kafka purge messages, parses them, and turns them into HTTP purge requests for ATS and Varnish
PyBal 	Automated manager for LVS
Varnish 	A caching HTTP proxy used as the frontend (in-memory) component of Wikimedia's CDN.
SRE/Working Groups

This page lists all currently active SRE working groups, with their name and description, membership and expectations around time commitments.
== ONFIRE ==
The ONFIRE (ONgoing reFormulation of Incident Response Efforts) is a cross-team working group dedicated to the definition and evolution of Wikimedia incident response processes and tooling. The ONFIRE group has informally existed since about 2018 or 2019, but was reformed in September 2021.
ONFIRE Working Group Charter 
== Cache Busting working group ==
The Cache Busting working group investigates, designs and implements response measures, infrastructure and tooling to defend our infrastructure and services from DDoS attacks (specifically those passing through the CDN layer), including measures to better withstand attacks, as well as to improve our response during attacks.
The Cache Busting working group was formed in early 2022.
=== Composition ===
Lead: Kwaku Addo Ofori
Current members: Cole White, Giuseppe Lavagetto, Valentin Gutierrez, Reuven Lazarus, Chris Danis, John Bond, Fabrizio Furnari
== Training working group ==
The Training working group goal is to efficiently and effectively onboard any new team member (internal/external) as part of the Site Reliability Engineering (SRE) team within a short period. To accomplish this, the working group focuses on coming up with a structured, formalized training plan, defining standardized training documentation, videos and/or any other material and maintaining them up to date.
The Training working group kick-off meeting happened in December 2023. The work is intended to start from January 2024.
=== Composition ===
Lead: Kavitha Appakayala
Current members: Cathal Mooney, Keith Herron, Clément Goubert, Jaime Crespo
Expected time commitment: 10% time.
== Mentorship working group (completed/ended) ==
The group will explore and propose effective strategies to incentivize mentorship within the SRE team. This includes identifying rewards, recognition methods, and other motivational tools to encourage potential mentors and mentees to participate actively in the mentorship program.
Mentorship Working Group Charter: how to create incentives for mentorship within the organization.
Deliverable: https://office.wikimedia.org/wiki/SRE/Mentorship
SRE/business case

A business case is a document that provides a justification for making a change or undertaking a project. It defines an observed problem or opportunity, evaluates benefit, cost and risk both for proposal as well as alternative solutions and provides a rationale of the undertaking.
== Create a new business case ==
type = create
prefix = SRE/business case/
preload = SRE/business case/template
buttonlabel = Create a business case
placeholder = You're making a case for…
== ==
SRE/business case/template

== 1. Executive Summary ==
== 2. Business Problem ==
== 3. Problem Analysis ==
== 4. Current Technology/Solution ==
== 5. Available Options ==
=== 5.1 Option 1 - Name ===
==== 5.1.1 Description ====
==== 5.1.2 Benefits, Goals and Measurement Criteria ====
==== 5.1.3 Costs and Funding Plan ====
==== 5.1.4 Risks ====
==== 5.1.5 Issues ====
==== 5.1.6 Assumptions ====
== 6. Recommended Option ==
== 7. Implementation Approach ==
=== 7.1 Dependencies ===
=== 7.2 Rough estimates ===
SSH/monitoring

=== Runbook for 'SSH unreachable' alerts ===
#Try logging in via the mgmt interface. Can you get a console from there? What is the output?
#sshd can be overwhelmed by rapid attempts to connect; if you can get on the host and things seem normal, check /var/log/auth.log for errors
#If you can get on the host, run 'ip link show' to check if there is physical network connection
#If 3) fails, escalate the issue to the dcops team.
Category:Runbooks
Scap/Release

This page is documentation for releasing a new version of Scap to Wikimedia Production.
=== Building ===
To prepare a new release for Scap, the Release Engineering Team needs to follow the steps in RELEASE.md (in the Scap git repository)
===Roll out to beta===
Follow the steps in the beta section of RELEASE.md . Please note you will require sudo privileges in the mentioned hosts.
===Roll out to production===
Performed by a member of the Release Engineering Team: 
*Make sure you select a time window where no deployments are happening
*On the main deployment server (deploy1002.eqiad.wmnet at the time of writing), run scap install-world. This will select for installation the latest available version tag from the git repo and prompt you for confirmation.
*Log into mwdebug* servers and check there if a scap pull still works
* Check if scap still works for the scap3 code path, and on stretch. You will need to run a test deployment to test environment. Run the following from the main deployment server:
cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster
...
canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups: n
Rollback all deployed groups? [Y/n]: Y
===Rollback===
In case a rollback is necessary, it is possible to pass a previous version tag to the install command. For example: scap install-world --version 4.9.3
===Target host bootstrapping===
Hosts added to the list of targets should be automatically primed with the latest deployed scap version. But if the automated process failed for a host for whatever reason, it is possible to trigger it manually from the main deployment server with: scap install-world --batch This will ensure all hosts are in sync with the version staged in the deployment server (i.e. the last deployed scap version)
Scap/Scap3 check scripts

Scap3 can run check scripts for each phase of deployment.
=== NRPE checks ===
 
nrpe check type for reusing existing Icinga/NRPE checks in deployments.
 
By default, checks are loaded and registered from all NRPE definitions in /etc/nagios/nrpe.d and can be referenced in checks.yaml using type: nrpe.
checks:
 service_endpoints:
 type: nrpe
 command: check_service_endpoints
 stage: promote
Scap/Scap3 config deployment

Scap3 can be used to deploy a service's configuration files.
=== What happens on the deployment server ===
 
* Looks for environment-specific ./scap/config-files.yaml with target files in the format:
/path/to/target:
 template: env-specific-template.yaml.j2
 remote_vars: /optional/remote/variable/file.yaml
* Looks for environment-specific ./scap/vars.yaml that includes variables used to render the template. These variables will be overridden by any conflicting variables in the file specified by remote_vars
* Variables from any environment-specific vars.yaml file are combined with variables from the root vars.yaml file.
* A json file is created at [repo]/.git/config-files/[revision].json that contains the final path to any environment-specific templates as well as a final list of combined variables.
=== What happens on targets ===
* Download the file from [deployment-server]/[repo]/.git/config-files/[revision].json
* Loop through the config files, and render each template using the variables from the downloaded json file and the variables from the (now) local remote_vars file
* Links rendered file (in [repo]/.git/config-files/[path]) to final location
Scap/Scap3 deployment groups

This feature was introduced in differential revision D16
In addition to the dsh_targets config variable, scap looks for multiple [anything]_dsh_targets config variables. This enables canary_dsh_targets.
All additional deployment groups will be executed before the primary deployment group (defined by the dsh_targets variable).
Additionally, checks now can be scoped to a specific deployment group using:
check_name:
 stage: promote
 group: dsh-group-name
 command: touch /tmp/hi-there
The group name is optional in a check. If not group name is specified, check runs for all deploy groups.
Scap/Scap3 limit

=== Filter deploy hosts with --limit-hosts ===
--limit-hosts or -l accepts a pattern in one of the following formats:
* ~[regex] - if the pattern starts with ~ it is interpreted as a regular expression, this is not combined with any other option
* ![pattern] - can be combined with any other pattern to negate that pattern. May be combined with other pattern matching options.
* host[01:10] - range matching, works for ascii chars and numbers, including numbers with a leading 0, may be combined with other pattern-matching options.
* host* - Matches 0 or more characters in the set A-z, '_', '.', or '-'. May be combined with other pattern matching options. This pattern is applied to the dsh_targets file to return a sub-set of hosts to use as a deployment target.
Scap/Scap3 structured logging

The structured logging feature of scap3 was introduced in differential revision D18
The main deploy application now sends all structured log output to a
file under scap/log/{git-tag}.log which the new deploy-log utility
can tail and filter using a given free-form expression. By default the
latter utility will periodically scan the scap/log directory for new
files and immediately begin tailing them. It can also be given an
explicit log file to parse via the --file option or the latest log
file by using --latest; in this case, it will simply filter the entire
file for matching records and exit.
As an alternative to the default behavior, you can either specify the log file
to parse via the scap deploy-log --file option or choose the newest log file by
using scap deploy-log --latest; in this case, it will simply filter the entire
file for matching records and exit, rather than watching for more log files to
be created.
The default behavior is convenient for monitoring an ongoing deployment from a
separate terminal. Simply start scap deploy-log in a separate terminal prior to
running scap deploy. Once your deployment starts, scap deploy-log will discover
the new log file and immediately begin displaying relevant log messages
== Examples ==
=== Tail behavior ===
* Run deploy-log {expr}
* Run deploy in a separate terminal
* Verify that **deploy-log** in the first terminal starts reading the new log file. It should say -- Opening log file: {file}.
* Verify that only log messages matching the given expression are output.
=== Latest log file behavior ===
* Run deploy.
* Run deploy-log -l {expr}
* Verify that only log messages from the latest log file matching the given expression are output.
=== Single log file behavior ===
* Run deploy a couple of times.
* Run deploy-log -f {log-file} {expr}
* Verify that only log messages from the given log file matching the given expression are output.
Schema changes/old

== What? ==
How do you know if you need to request a schema change? If you are changing anything in how the database is setup (columns, tables, column types, etc), you need to use this process.
==Guidelines==
* All schema changes must be optional for your code; schema changes can take a while to propagate across the entire WMF cluster so your code should not depend on them.
* All schema changes must be reviewed by DB Ops and/or Platform Engineering.
* To get your schema change reviewed, open a ticket in Phabricator with all pertinent information.
* Please create your ticket as soon as possible as it can take time to review and deploy the change, working around other Deployments.
* We use a tracking ticket in Phabricator to list all schema change tickets.
== See also ==
* How to do a schema change
* 2015 proposal: Schema changes/Coordination proposal
Category:MySQL
Screen

screen is a terminal multiplexer.
Once logged in on a server, use screen bash and you will have another "bash" in a virtual window. From there you can add other windows with different programs and move from a window to another. Ultimately, you can send "screen" in the background & logout - "screen" will detach and run in the background.
Example:
A new "bash" is invoked, my "xterm" title show:
This means I am in the "screen" window #0 and running "bash".
and so on ...
=== Multiple windows inside one "screen" ===
I am monitoring the server. Now let us create another window to edit a file. This is done by using CTRL + A then C (for create).
A new "bash" is invoked, my "xterm" title show:
I can do whatever I want there. To move back to the window with "vmstat" use CTRL + A then P (for previous), to switch again CTRL + A then N (for next).
=== Splitting windows ===
You can also split a "screen" to see more than one window at once. Use CTRL + A then S (yes, that is a capital "S", so SHIFT + s, for Split), then use CTRL + A then <Tab> to switch between them. Now you can create a new window, using CTRL + A then C as described above and keep switching between them using Tab.
=== Detaching and Re-attaching ===
Finally it is dinner time or you need to reboot your computer, or whatever. You can send "screen" in the background and logout, that will keep all your program running. This is done with CTRL + A then D (for detach).
"Screen" is in the background. Now you can logout. Once logged back, you want to resume your backgrounded "screen":
You are back to the "vmstat" stuff which is still spamming values :)
Finally log out of all windows, when there is none left, "screen" exit:
==== Attaching a shared "screen" ====
If you want to share a "screen" with somebody else who is logged in as the same user, or you have forgotten to detach from somewhere else, and screen -r does not appear to work use:
(Attach to a not detached "screen" session.)
== Troubleshooting ==
From inside "become", "screen" currently (cf. phab:T52248) outputs an error:
You can work around this by running script /dev/null (you can first read man script to figure out what "script" does). A second workaround is running "screen" first and then running "become" from inside "screen". However, per the cloud listserv, "screen" should not be used by tool accounts and instead jobs should be submitted via the Toolforge Jobs Framework.
== External link ==
* http://www.guckes.net/screen/ Sven Guckes "screen" pages
* "I like to know which system I am currently on inside of a "screen" window, which shell server I am on, and the date and time in UTC" / handling the ssh agent -> Ryan's blog with really nice additions to .screenrc/.bashrc 
Category:How-To
Scripts

* Sync scripts
* Squid checker
* Misc scripts
* check-yum-updates.sh - run from zwinger (needs root?) to check all machines for available updates via yum - results in /tmp/check-updates.all
* sync-mirror-fedora.sh - run from (TODO) crontab on albert to update FC1 and FC2 mirror
* backup-all - make database dumps - log in as root to albert and run with /home/wikipedia/bin/backup-all . If errors, check the db name and path in /home/wikipedia/bin/backup-site .
Search

The linked pages contain technical documentation on how search works on
Wikipedia and its sister projects (Wikimedia Commons, Wikidata, etc). This software and infrastructure are owned
and supported by the Wikimedia Foundation’s Search Platform team.
== In case of emergency ==
Here is the (sorted) emergency contact list for Search issues on the WMF production clusters:
Name 	IRC nick
David Causse 	dcausse
Erik Bernhardson 	ebernhardson
Brian King 	inflatador
Ryan Kemper 	ryankemper
Guillaume Lederrey	gehel
You can get contact information from officewiki.
== Overview ==
The WMF’s production system has four components: Elastica, CirrusSearch, Cirrus Streaming Updater, and Elasticsearch.
=== Elastica ===
Elastica is a MediaWiki extension that provides the library to interface with Elasticsearch. It wraps the Elastica library. It has no configuration.
=== CirrusSearch ===
CirrusSearch is a MediaWiki extension that provides search support backed by Elasticsearch. For more details about how the CirrusSearch Mediawiki plugin is deployed in WMFs production environment, see this page.
=== Cirrus Streaming Updater ===
The Cirrus Streaming Updater updates the Elasticsearch indexes for each and every mediawiki edit. In other words, when someone edits a Wikipedia page, this is the process that finds the edit and adds it to the Elasticsearch index, making it possible for others to find it via a search from a Wikipedia page.
=== Elasticsearch ===
Elasticsearch is the application that stores our search data. It can be queried via its rest API. For more details on how Elasticsearch is deployed in WMF’s production environment, see this page.
Search.wikimedia.org

search.wikimedia.org holds a web service for running title prefix searches for Apple's Dictionary application in Mac OS X 10.5. It is a simple ~100 line script hosted on Kubernetes.
There is a contract with Apple (restricted) requiring this service to be maintained.
== History ==
Apple requested that we implement this service in October 2007. Mike Godwin negotiated the contract, and Brion Vibber (then CTO) wrote the code and set up the service. It used the TitleKey extension for the first two months, and then Brion migrated it to use the new OpenSearch endpoint. Originally the service ran on separate hardware: first yongle and then ekrem. When it came time to decommission ekrem in 2013, there was some talk of discontinuing the service (phab:T81982). Log analysis showed that it was still actively used. So in 2013 Tim Starling moved it to the main application cluster, and updated the source (commit), with a view to allowing another 5-10 years of low-maintenance service.
In 2021, Taavi Väänänen moved the service from the main appserver cluster to a separate Kubernetes service in preparation for moving MediaWiki itself into Kubernetes (phab:T289224). A dashboard showing request rate is now available.
== External link ==
* https://search.wikimedia.org
Category:Services
Search/Cloudvirt-wdqs

== Cloudvirt-wdqs hosts ==
Search Platform Team has their own set of hypervisors, originally used for testing WDQS . These hosts are currently configured as WMCS Openstack nova hosts. You can see these hosts in Puppet's site.pp or on in the WMCS horizon project "wikidata-query."
=== Operations Ownership ===
The WMCS team is responsible for day-to-day operations of this environment.
=== Current State ===
As of this writing, there are two VMs running, dating back to 2020.