nyuuzyou/wikis huggingface dataset:
How-To

For a complete list of How-To guides, check :Category:How-To.
== Deployment ==
* Writing an extension for deployment
* How to deploy code
* How to deploy Wikidata code
* How to perform security fixes
* How to do a schema change
* How to deploy current master branch of an extension
* Heterogeneous deployment
* Mobile frontend deployment
* Train deploys
* Deploying ORES
* Deploying a new service
* Regular service deployments
* Backport deployments
== Database ==
* Searchindex update
* adding a file in innodb
* Setting up a MySQL replica
* Checking MySQL replication
* Switch master : Switching Master Database
* Make a database dump
* Compress old revisions
* Reclaim space on full db servers
* How to run queries on live data
* How to do a schema change
== Application servers ==
* Memcached
== Wikifarm ==
* Add a wiki : Instructions on how to add a new language for a project, chapter, committee, so on...
* FlaggedRevs setup
* AbuseFilter setup
* GrowthExperiments setup
* Adding Namespaces
* Add a server
==Other operations==
* Operations_requests : How to get help from operations.
* Decommissioning Servers : What to do to remove a server from our datacenter.
* Proxy access to cluster : Using FoxyProxy for web access to cluster/labs.
* Screen : how to keep stuff running when you log out
* Make an email alias
* Alter mailman archives
* Redirect traffic to tampa
* Build a new server
* Reset a password or 2FA for a wiki account
* Regenerate cached special pages
How to deploy code/Adding a new submodule

Normally, new extension submodules are added to weekly core deploy branches by the make-release release tool. This is the preferred method for deploying a new extension to production. Your extension can be added by adding it to the list in the tool's configuration file. It will begin rolling out to the WMF production wikis on the train beginning the week after it is configured.
Submodules which use a non-standard setup, such as non-conventionally named deployment branches, must be added to the core deployment branches manually.
NOTE: When adding a new extension to one branch, you also need to add the extension to any other branches in use on the cluster (typically the wmf.{N-1} branch), even if the extension will not be enabled on any wikis running that branch. Otherwise the localization cache builder will fail. It is best to add to the N-1 branch even if when is not used on the cluster anymore, to avoid catastrophic error in case of an unexpected train rollback.
$ cd mediawiki/core # Go to your checkout of core
# Set up a local wmf/1.27.0-wmf.1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.27.0-wmf.1 branch, you can skip this step
# If you get an error, try 'git remote update' or 'git fetch' first
$ git checkout --track origin/wmf/1.27.0-wmf.1
Branch wmf/1.27.0-wmf.1 set up to track remote branch wmf/1.27.0-wmf.1 from origin.
# Make sure you have an up-to-date copy of the deployment branch. The submodule update might take a while.
$ git pull
$ git submodule update --init --recursive
# Add a submodule for your extension (this will check out the extension's master branch if the directory did not exist, otherwise will just use what's there)
$ git submodule add https://gerrit.wikimedia.org/r/mediawiki/extensions/MyCoolExtension.git extensions/MyCoolExtension
Cloning into extensions/MyCoolExtension...
# Check the diff. Make sure the .gitmodules entry is in line with the others (you might need to edit it and set the branch),
# and that the subproject commit hash points to the version you want deployed (which is usually the head of master)
$ git diff --cached
diff --git a/.gitmodules b/.gitmodules
index 3ab3d48..9a4cc66 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -346,3 +346,6 @@
 [submodule "extensions/PrefSwitch"]
 path = extensions/PrefSwitch
 url = https://gerrit.wikimedia.org/r/mediawiki/extensions/PrefSwitch.git
+[submodule "extensions/MyCoolExtension"]
+ path = extensions/MyCoolExtension
+ url = https://gerrit.wikimedia.org/r/mediawiki/extensions/MyCoolExtension.git
diff --git a/extensions/MyCoolExtension b/extensions/MyCoolExtension
new file mode 160000
index 0000000..46727ad
--- /dev/null
+++ b/extensions/MyCoolExtension
@@ -0,0 +1 @@
+Subproject commit 46727ad74adda33621323deb2bebdc2527cb4917
# Commit the submodule addition and submit it for review
$ git commit -a -m "Add MyCoolExtension submodule"
$ git review
# If you don't want or need this to be reviewed, you can +2 your own commit if you are in the wmf-deployment group
Category:Deployment
How to deploy code/Core submodule update

While Gerrit performs automatic MW core submodule updates for most extensions when you merge a commit to a wmf/ branch in them, it's confused by VisualEditor core's repo (VisualEditor/VisualEditor) and doesn't do it for the VE extension.
Additionally, our deployment branches are not set up to create automatic submodule updates for extensions using custom deployment branches. This process applies to those as well as VisualEditor. 
Here's how to do it manually, starting from MW core:
 git fetch
 git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
 git checkout wmf/1.20wmf1
 git pull
 git submodule update --init --recursive extensions/VisualEditor
 cd extensions/VisualEditor
 git fetch
 git checkout origin/wmf/1.20wmf1
 cd ../..
 git diff extensions/VisualEditor
 git commit -m "Update VisualEditor for I{CHANGE_ID_HERE}" extensions/VisualEditor
 git review
Category:Deployment
How to run queries on live data

If you need to query the live data (rather than a dump or tools mirror), follow the following steps:
# ssh to mwmaint1002
# Connect to the slave database server with the syntax sql <DBNAME>, for example sql enwiki
# Run your query
You should avoid running queries on the master databases whenever possible. Otherwise you will bog down performance of the live websites. If you absolutely must run a query on a master database, follow these instructions:
# ssh to mwmaint1002
# Connect to the master database server with the syntax sql --write <DBNAME>, for example sql --write enwiki
# Run your query
To check the current status of a database, you can click on its server name in the dbtree chart or log into the server from mwmaint1002 and run show engine innodb status\G.
== Querying a specific server ==
If you need to query a specific replica server, follow the following steps:
# ssh to mwmaint1002
# Connect to the slave database server with the syntax sql <DBNAME> --host <HOSTNAME>, for example sql enwiki --host db1001
# Run your query
Category:How-To
Category:MySQL
Hume

== Hardware ==
Dell PowerEdge 1950/2950
== Ran ==
* spence and hume were maintenance/batch servers, not Apache servers; hume is being replaced by its eqiad equivalent terbium
* Apache (serves tarballs of static html dumps of the various wikipedia projects from June 2008, as well as live browsable copies of them)
* Gearman (what is this piece doing? No docs or logs anywhere, can't find out what clients are talking to it...)
=== Puppet Roles ===
hume is a Wikimedia Misc - Maintenance Server: foundationwiki (misc::maintenance::foundationwiki).
hume is a Wikimedia Misc - Maintenance Server: pagetriage extension (misc::maintenance::pagetriage).
hume is a Wikimedia Misc - Maintenance Server: parser cache purging (misc::maintenance::parsercachepurging).
== Storage ==
* has a large partition /archive which has fundraising backups and banner impression logs, see Backup_status_chart
== History ==
*2009-08-04: was being used for hadoop testing, that's been turned off, now doing its normal jobs only
*2008-02-29: Server racked, DRAC online. Will not netboot due to DHCP requiring new configuration for this subnet.
Category:Fundraising
IDM/Account Blocking

Trusted administrators (select stewards and SREs) are allowed to block user account via the Bitu IDM.
Navigate to https://idm.wikimedia.org and select: "Block/unblock accounts" for the navigation menu.
Using the search field, locate the user who are to be blocked or unblocked. You can search for Wikimedia Developer Account, shell account or email.
Once you have located the correct user, click block or unblock. Fill in a comment, stating why the user have been blocked/unblocked, and click Block or Unblock. 
You will now be returned to the search page, with the relevant user already listed in the results. Use the "View Logs" to verify that all systems have been blocked, or unblocked.
Only select users can utilize the "Block/unblock accounts" feature. These are either users defined in ACCOUNT_MANAGERS settings, see: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/idm/templates/idm-django-settings.erb#320 or users in the LDAP group: "cn=bitu-account-managers,ou=groups,dc=wikimedia,dc=org". To get added, please reach out to the Infrastructure Foundations SRE team.
IP and AS allocations

This page lists IP addresses and Autonomous system numbers (ASN) allocated by Wikimedia Foundation.
== Public IPs ==
RIR 	Subnet
RIPE 	185.15.56.0/22
RIPE 	2a02:ec80::/29
RIPE 	91.198.174.0/24
RIPE 	195.200.68.0/24
RIPE 	193.46.90.0/24
ARIN 	2620:0:860::/46
ARIN 	198.35.26.0/23
ARIN 	208.80.152.0/22
APNIC	103.102.166.0/24
APNIC	2001:df2:e500::/48
=== IPV4 ===
Moved to Netbox.
=== IPv6 ===
Moved to Netbox.
== RFC1918 IPs ==
Moved to Netbox.
== Multicast ==
As of July 16th 2020, there is no more multicast/PIM in the infra.
== Autonomous System Numbers ==
Moved to Netbox
Private ranges: 64512-65534 and 4200000000-4294967294 (RfC 6996)
== BGP communities ==
Community	Name 	Allocation 	Action
14907:0 	AVOIDED_PATH 	Manual as-path-group 	local-pref 50
14907:1 	BACKUP_PATH 	Manual as-path-group 	local-pref 70
14907:2 	PEERING_ROUTE_PRIMARY	Prefixes learned from an IX we want to prefer (when multiple IXs are present at a site)	local-pref 260
14907:3 	PEERING_ROUTE 	Prefixes learned from IX peers 	local-pref 250
14907:4 	TRANSIT_ROUTE 	Prefixes learned from transit peers 	none, info only
14907:5 	PARTIAL_TRANSIT_ROUTE	Unused 	none
14907:6 	PEER_INTERNAL 	Unused 	local-pref 290
14907:7 	PEER_CUSTOMER 	WMF customers prefixes (SF office, WMCS) 	local-pref 250
14907:8 	PEER_PRIVATE_PEER 	Prefixes learned from private peers (similar to IX peers) 	local-pref 270
14907:9 	PEER_PUBLIC_PEER 	Unused 	none
14907:10 	PREFERRED_TRANSIT 	Manual as-path-group 	local-pref 180
14907:11 	SELECTED_PATH 	Manual as-path-group 	local-pref 350
14907:12 	DIRECT_PEER 	Prioritize directly connected peers - T280054 	local-pref 280
14907:13 	ANYCAST 	Public anycast prefixes - T288843 	none
14907:14 	KUBERNETES 	Prefixes from k8s/Calico nodes - T328523 	export to other k8s nodes
14907:15 	DIRECT_PEER_PRIMARY 	Prioritize directly connected peers on primary IXPs - T338201 	local-pref 285
Category:SRE Infrastructure Foundations
Category:Network
IPv6 deployment

IPv6, the successor to IPv4, features a greatly-enlarged address space. Despite its age and feature maturity, deployment of IPv6 on the public Internet is still ongoing.
The success of IPv6 is important for the preservation of the any-to-any internet, a technical characteristic of the Internet which is important to Wikimedia's mission and other decentralized or peer-to-peer information services. 
For Wikimedia IPv6 deployment was relatively straight-forward as the majority of the software infrastructure already had solid IPv6 support. A small number of technical limitations, or lack of time for engineers to work on things, means that in some small corners of our infra we still rely on IPv4 but we are continually moving closer to having everything fully working over IPv6. All the main public-facing projects are available over IPv6.
Wikimedia participated in the World IPv6 Launch on June 6, 2012.
==Current IPv6 deployment status==
===IPv6 network status at Wikimedia===
* Wikipedia and most internet-facing services are available over IPv6 from all our core and caching POP sites.
* The entire Wikimedia network infrastructure runs dual-stack IPv4 and IPv6, meaning most internal traffic uses IPv6.
====DNS servers====
The authoritative nameservers ns(0-2).wikimedia.org all return AAAA records correctly, but due to concerns about the accuracy of IPv6 geo-ip maps versus those for IPv4 we have not made the DNS records available over IPv6 yet (see ).
===IPv6 production testing at Wikimedia===
* Ongoing testing for clients with broken IPv6 support using client side scripting on English Wikipedia. ipv6and4.labs.wikimedia.org (Decommissioned)
===Fully IPv6 enabled services at Wikimedia===
*lists.wikimedia.org (as well as the Wikimedia MXes). SSL does not work, however (:bugzilla:22028)
*download.wikimedia.org
*svn.wikimedia.org
*bugzilla.wikimedia.org
*Gerrit code review
*toolserver.org
*Main production wikis - enabled as part of Wikimedia's participation in the World IPv6 Launch on 6 June 2012 (list of participants, announcement):
**wikipedia.org
**wikisource.org
**wikibooks.org
**wiktionary.org
**commons.wikimedia.org
**meta.wikimedia.org
**species.wikimedia.org
**incubator.wikimedia.org
**wikiversity.org
**mediawiki.org
**wikiquote.org
**wikimediafoundation.org
**wikidata.org
**wikivoyage.org
===Known outstanding technical issues===
*Interaction of broken dual-stack clients and AAAA records. 
*Squid 2 IPv6 client support. (Avoidable by simply using another proxy in front of squid for v6) Squid 3.1 has full IPv6 support, however.
*MediaWiki user blocking support.
===statistics related to deploying AAAA records===
* results.labs.wikimedia.org
==How to add IPv6 to a node==
In site.pp:
 interface::add_ip6_mapped { 'main': }
or
 interface::add_ip6_mapped { 'main':
 interface => 'eth0',
 }
Category:Network
IRC

== About IRC and Wikitech ==
IRC is the primary mode of instant messaging for people working with the Wikimedia Foundation servers and network, such as the Wikimedia Foundation SRE team, and for people working with them. This page documents how IRC is used by this group of people. 
=== Why IRC? ===
IRC is preferred over other instant messaging protocols and software because it is an open, non-proprietary, mature, interoperable protocol supported by a variety of servers and clients, many of which are free software. Together with a logging tool, it provides all of the features required for persistent messaging.
== IRC Channels in Use ==
For a complete list of Wikimedia channels, see m:IRC/Channels.
== Planning ==
=== Winter/Spring 2019 SRE-related channels Cleanup effort ===
The Wikimedia Foundation SRE team would like to make the names and properties of IRC channels that the team uses to be consistent, meaningful, correctly scoped, and helpful for newcomers. See the Talk page for more information.
== See also ==
* m:IRC/Bots
Icinga/Runbooks

== How to add a runbook to an Icinga alert ==
1) Create a new page under Monitoring here on Wikitech, example: Monitoring/check_ferm
2) Add content and add the page to :Category:Runbooks (Look at the existing run books in there)
3) In Puppet add the full https URL of your page as a value of the notes_url parameter to one of the classes: monitoring::service, nrpe::monitor_service or monitoring::grafana_alert.
4) If you use a different class that does not have a notes_url parameter yet, consider adding it.
5) Merge and run puppet on the Icinga server. You should now see a new icon in the web UI with a "View Extra Service Notes" label. Clicking this should lead to your new page. The URL will also be added to IRC bot output.
* Related Phabricator ticket: https://phabricator.wikimedia.org/T197873
Category:Runbooks
Icinga1001

icinga1001 is the currently active production Icinga server and the equivalent of icinga2001 in codfw. Together they replaced einsteinium and tegmen which replaced neon. 
It's not active anymore, replaced with alert1001
== Services ==
* Icinga
* logmsgbot: tcpircbot IRC relay for syncs / scaps
== Also see ==
* icinga2001
* einsteinium
* tegmen
Iegreview.wikimedia.org

iegreview.wikimedia.org was a web-based application used to accept and process grants applications requests. See Grants Review app for details.
== How did I access it? ==
https://iegreview.wikimedia.org/
== Where could I report issues? ==
Bugs are tracked in phabricator.
== Where did it run? ==
The PHP application was deployed in an Apache vhost on krypton in the eqiad datacenter. SSL termination and front-end cache was provided by the misc varnish cluster. The database was served from db1048.eqiad.wmnet.
== Where's the code? ==
; Application
: Code was hosted locally in gerrit and diffusion
; Configuration
: The application was configured in the operations/puppet repo:
:* modules/iegreview
:* manifests/role/iegreview.pp
:* manifests/misc/deployment.pp
:* templates/varnish/misc.inc.vcl.erb
: Usernames and passwords were managed via the passwords::mysql::iegreview class in the private puppet repository.
== How was it deployed? ==
Deployment was managed with Trebuchet from the /srv/deployment/iegreview/iegreview/ directory on the deployment host.
$ ssh deploy1001.eqiad.wmnet
$ cd /srv/deployment/iegreview/iegreview/
$ git deploy start
$ get fetch origin
$ git rebase origin/master
$ git deploy sync
== Where were the logs? ==
Logs were written to mwlog1001 in the /srv/mw-log/iegreview.log file by Udp2log
Category:Services
Image-suggestion

image-suggestion is a service that provides read-only access to the Data Gateway Service. It is a relatively simple HTTP service running in Kubernetes. It was created by the Platform Engineering Team. The code lives in the generated-data-platform/datasets/image-suggestions repo on Gerrit. The task for setting it up was T304891. 
It is not to be confused with its predecessor, the image-suggestions-api service, which served a one-off dataset and has been discontinued. The task for providing a similar public API for live data is T306349.
== Configuration ==
The Helm chart used by the image-suggestion service is based on the cassandra-http-gateway chart. This chart is a generic chart to allow for the configuration of various services that will offer HTTP interfaces to data stored in Cassandra and is service independent for the most part. The image-suggestion service pods use Helmfile configuration to specify the relevant service-specific bits of configuration.
The service connects to Cassandra using a dedicated user configured in puppet. 
Access to the service is managed via the Kubernetes ingress.
== Deployment ==
Deployment of the service uses the standard Helm/Kubernetes process. A lightweight healthcheck is available at /healthz.
== API docs ==
See mw:Platform_Engineering_Team/Data_Value_Stream/Data_Gateway#Image_Suggestions
== Runbook ==
See Image-suggestion/Runbook
== See also ==
* ImageSuggestions on mediawiki.org
* Add Image
Incident documentation meeting/QR201403

= Quarterly Review of post-mortems - 2014-03 =
== Questions we want to be able to answer ==
* Have all of the issues that came out of the post-mortem been addressed? If not, why not?
* Are we satisfied with the current state of that part of the infra? Are there further actions to take (upon further reflection)?
* anything else?
== Agenda ==
* Go through the post-mortems and their respective action items and make sure they have been followed up appropriately.
** If you have details that are relevant to the post-mortem in BZ/etc, please link from the post-mortem.
* Discuss if there is anything else that we learned from the situation and follow up to better inform future decisions.
* Notes written up by all, collaboratively, so that others in the organization will learn from these as well.
== The post mortems ==
=== site outage ~ 2014-01-11 22:10 UTC ===
* TODO: Follow up with Sean and Tim about this. (Greg) - 
** greg pinged sean 20140320
=== ../20140113-Poolcounter/ ===
=== ../20140203-LVS/ ===
=== ../20131205-Swift/ ===
=== ../20140206-Math/ ===
=== ../20140211-Parsoid/ ===
=== ../20140228-Cirrus/ ===
=== ../20140313-API-Parsoid/ ===
=== ../20140313-Deploy/ ===
Incident documentation meeting/QR201407

= Quarterly Review of post-mortems - 2014-07 =
== Questions we want to be able to answer ==
* Have all of the issues that came out of the post-mortem been addressed? If not, why not?
* Are we satisfied with the current state of that part of the infra? Are there further actions to take (upon further reflection)?
* anything else?
== Agenda ==
* Go through the post-mortems and their respective action items and make sure they have been followed up appropriately.
** If you have details that are relevant to the post-mortem in BZ/etc, please link from the post-mortem.
* Discuss if there is anything else that we learned from the situation and follow up to better inform future decisions.
* Notes written up by all, collaboratively, so that others in the organization will learn from these as well.
== Groupings ==
* Group 1
** notes
* Group 2
** notes
== The post mortems ==
=== ../20140318-EventLogging/ ===
Sean, Nuria
=== ../20140328-DB-Queries/ ===
Sean
=== ../20140403-Deploy/ ===
Bryan, Reedy
=== ../20140503-Thumbnails/ ===
Antoine, Faidon
=== ../20140509-EventLogging/ ===
Nuria
=== ../20140517-bits/ ===
Timo
=== ../20140526-m1/ ===
Sean
=== ../20140529-appservers/ ===
Ori
=== ../20140607-Elasticsearch/ ===
Nik
=== ../20140608-Kafka/ ===
Andrew O
=== ../20140612-Math/ ===
Greg
=== ../20140613-Videoscalers/ ===
Giuseppe
=== ../20140618-Wikitech/ ===
Andrew O
=== ../20140619-parsercache/ ===
Sean
=== ../20140622-es1006/ ===
Sean
=== ../20140622-imagescaler/ ===
Filippo
=== ../20140625-CirrusSearch/ ===
Nik
Incident documentation meeting/QR201407/group1

= Quarterly Review of post-mortems - 2014-07 - Group 1 =
== Questions we want to be able to answer ==
* Have all of the issues that came out of the post-mortem been addressed? If not, why not?
* Are we satisfied with the current state of that part of the infra? Are there further actions to take (upon further reflection)?
* anything else?
== Agenda ==
* Go through the post-mortems and their respective action items and make sure they have been followed up appropriately.
** If you have details that are relevant to the post-mortem in BZ/etc, please link from the post-mortem.
* Discuss if there is anything else that we learned from the situation and follow up to better inform future decisions.
* Notes written up by all, collaboratively, so that others in the organization will learn from these as well.
== Notes ==
HERE!
== The post mortems ==
=== ../../20140318-EventLogging/ ===
Sean, Nuria
=== ../../20140328-DB-Queries/ ===
Sean
=== ../../20140509-EventLogging/ ===
Nuria
=== ../../20140526-m1/ ===
Sean
=== ../../20140607-Elasticsearch/ ===
Nik
=== ../../20140613-Videoscalers/ ===
Giuseppe
=== ../../20140619-parsercache/ ===
Sean
=== ../../20140622-es1006/ ===
Sean
=== ../../20140622-imagescaler/ ===
Filippo
=== ../../20140625-CirrusSearch/ ===
Nik