nyuuzyou/wikis huggingface dataset:
Deployments/Yearly calendar/2022

See also the calendar for 2021 and the calendar for 2023.
 Deployment disruptions 2022
What 	When 	Deployment decision 	What happened?
New Year's Day (observed) 	Mon 3 Jan 	No deploys 	8 MW deploys (not added to calendar in time)
Martin Luther King Day 	Mon 17 Jan 	No deploys 	No MW deploys
Presidents' Day 	Mon 21 Feb 	No deploys 	No MW deploys
WMF Holiday 	Fri 18 Mar 	No deploys (not that there are ever deploys on Friday) 	No MW deploys
WMF Holiday 	Fri 1 Apr 	No deploys (not that there are ever deploys on Friday) 	No MW deploys
Earth Day 	Fri 22 Apr 	No deploys (not that there are ever deploys on Friday) 	2 MW deploys
RelEng GitLab-a-thon sprint 	Mon May 9–Fri May 13	No train, Backport and config as usual 	N/A
Memorial Day 	Mon 30 May 	No UTC Late Backport/Config window 	No late-UTC MW deploys
Juneteenth (observed) 	Mon 20 Jun 	No deploys 	No MW deploys
Fourth of July 	Mon 4 Jul 	No deploys 	5 MW deploys
Wikimedia Foundation Inspiration Week 	Mon 11–Fri 15 Jul 	No train, Backport and config as usual 	N/A
International Day of the World’s Indigenous Peoples	Tue 9 Aug 	No Train, and no deploys that day 	No MW deploys
Labor Day 	Mon 5 Sep 	No deploys 	1 MW deploy
Indigenous Peoples' Day 	Mon Oct 10 	No deploys 	8 MW deploys
Election Day 	Tue 8 Nov 	No Train (same week as US Veterans' Day), and no deploys that day 	8 MW deploys
Veterans' Day 	Fri 11 Nov 	No Train (same week as US Election Day), and no deploys that day (so a normal Friday)	No MW deploys
Thanksgiving Day 	Thu 24 & Fri 25 Nov 	No Train, and no deploys those days 	
End of Year Holiday 	Mon 19 Dec–Mon 2 Jan	No Train or deploys
Deployments/Yearly calendar/2023

See also the calendars for 2022 and for 2024.
Deployment disruptions 2023
What 	When 	Deployment decision 	What happened?
New Year's Day 	Mon 2 Jan 	No deploys 	✅ No MW deploys
Martin Luther King Day 	Mon 16 Jan 	No deploys 	1× Emergency MW deploy and 1× No-op
Presidents' Day 	Mon 20 Feb 	No deploys 	6× MW deploys
Earth Day (observed) 	Fri 21 Apr 	No deploys (not that there are ever deploys on Friday) 	1× MW deploys
Memorial Day 	Mon 29 May 	No deploys 	✅ No MW deploys
Juneteenth / RelEng deep work week 	Mon 19 Jun 	No Train, and no deploys Mon 19 Jun 	9× MW deploys
WMF Holiday 	Mon 3 Jul 	No UTC Late backport window 	✅ No late-UTC MW deploys
Fourth of July 	Tue 4 Jul 	No UTC Late backport window 	3× late-UTC MW deploys
International Day of theWorld’s Indigenous Peoples	Wed 9 Aug 	No Train, and no deploys that day 	3× MW deploys
Labor Day 	Mon 4 Sep 	No deploys 	✅ No MW deploys
Indigenous Peoples' Day 	Mon 9 Oct 	No deploys 	2× MW deploys
Veterans' Day 	Fri 10 Nov 	No deploys (not that there are ever deploys on Friday) 	✅ No MW deploys
Deployment freeze discussion reminder 	Mon 13 Nov 	Open discussion with deployers and stakeholders about backports week of Dec 18 (and partial weeks at the end of the year going forward)	wikitech-l announcement
Thanksgiving+ Day after Thanksgiving 	Thu 23 & Fri 24 Nov 	No Train, and no deploys those days 	✅ No MW deploys
Release Engineering Offsite 	Mon 04 Dec–Fri 08 Dec	No Train 	—
End of Year Holiday 	Mon 25 Dec–Mon 1 Jan 	No Train or deploys 	1× MW deploy
Deployments/Yearly calendar/2024

See also the calendar for 2023 and the forthcoming one for 2025.
Deployment disruptions, 2024
What 	When 	Deployment decision 	What happened?
New Year's Day 	Mon 1 Jan 	No deploys 	✅ No MW deploys
Martin Luther King Jr Day 	Mon 15 Jan 	No UTC Late backport window 	2× MW deploys
U.S. Presidents' Day 	Mon 19 Feb 	No UTC Late backport window 	✅ No MW deploys
Northward Switchover: Services + Traffic 	Tue 19 Mar 	Deployments run as normal. Switchover window: 14:00-15:00 UTC 	—
Northward Switchover: Mediawiki 	Wed 20 Mar 	Deployments run as normal. Switchover window: 14:00-15:00 UTC 	—
Earth Day 	Mon 22 Apr 	No deploys 	✅ No MW deploys
U.S. Memorial Day 	Mon 27 May 	No UTC Late backport window 	—
Juneteenth 	Wed 19 Jun 	No UTC Late backport window 	—
Fourth of July 	Thu 4 Jul 	No UTC Late backport window 	—
U.S. July Holiday 	Fri 5 Jul 	No deploys (not that there are ever deploys on Friday) 	✅ No MW deploys
International Day of the World’s Indigenous Peoples 	Fri 9 Aug 	No deploys (not that there are ever deploys on Friday) 	✅ No MW deploys
U.S. Labor Day 	Mon 2 Sep 	No UTC Late backport window 	—
Southward Switchover: Services + Traffic 	Tue 24 Sept 	Deployments run as normal. Switchover window: 14:00-16:00 UTC (target time: 15:00 UTC)	—
Southward Switchover: Mediawiki 	Wed 25 Sept 	Deployments run as normal. Switchover window: 14:00-16:00 UTC (target time: 15:00 UTC)	—
Columbus Day (also Indigenous Peoples' Day) 	Mon 14 Oct 	No UTC Late backport window 	—
Veterans' Day 	Mon 11 Nov 	No UTC Late backport window 	—
Developer Experience Offsite 	Mon 09 Dec–Fri 13 Dec 	No Train (1.44.0-wmf.7), backports as normal 	—
End of Year Holiday 	Tues 24 Dec – Wed 1 Jan 	No Train (1.44.0-wmf.9 & .10) or deploys 	
1st week of 2025 (silent work week (if the WMF keeps quiet week))	Tues 07 Jan, 2025 – Thu 09 Jan, 2025	No Train (1.44.0-wmf.11)
Deployments/Yearly calendar/2025

See also the calendar for 2024.
Deployment disruptions, 2025
What 	When 	Deployment decision 	What happened?
New Year's Day 	Wed 1 Jan 	No deploys 	✅ No MW deploys
Martin Luther King Jr Day 	Mon 20 Jan 	No UTC Late backport window	
U.S. Presidents' Day 	Mon 17 Feb 	No UTC Late backport window	
Northward Switchover: Services + Traffic 	~21 Mar 	? 	
Northward Switchover: Mediawiki 	~21 Mar 	? 	
Earth Day 	Tue 22 Apr 	No deploys (train tbd) 	
U.S. Memorial Day 	Mon 26 May 	No UTC Late backport window	
Juneteenth 	Thu 19 Jun 	No UTC Late backport window	
U.S. July Holiday 	Thu 3 Jul 	No UTC Late backport window	
Fourth of July 	Fri 4 Jul 	No deploys as normal 	
International Day of the World’s Indigenous Peoples	Fri 8 Aug 	No deploys as normal 	
U.S. Labor Day 	Mon 1 Sep 	No UTC Late backport window	
Southward Switchover: Services + Traffic 	~21 Sep 	? 	
Southward Switchover: Mediawiki 	~21 Sep 	? 	
Columbus Day (also Indigenous Peoples' Day) 	Mon 13 Oct 	No UTC Late backport window	
Veterans' Day 	Tue 11 Nov 	No UTC Late backport window	
Thanksgiving & Day-after-Thanksgiving 	Thu 27–Fri 28 Nov	No UTC Late backport window	
End of Year Holiday 	Wed 24–Wed 31 Dec	No trains or deploys
Depooling servers

Contains various ways to depool different servers. In a glorious future, we'd just need to do conftool depool <service> <node> but we aren't there yet!
== cp*** machines ==
These are the varnish frontends, and should be depooled via conftool. To fully depool a server, do as root on any puppetmaster frontend:
$ sudo confctl select dc=<dc>,cluster=<cluster-name>,name=cp*** set/pooled=no
You can find out <cluster-name> from looking at conftool-data/node/ in the operations/puppet repository.
An easier option is to ssh into the server and just run
$ sudo -i depool
Note the use of sudo -i. depool needs environment variables to get access to etcd.
== mw* machines (and others) ==
The mw* application servers and various other servers are managed by PyBal directly in Etcd. These can be controlled via conftool as well, including with the depool and pool utilities.
See also Application servers/Runbook#From the host itself.
To learn how this works, refer to LVS#Etcd as a backend to Pybal (All of production).
== Re-pooling ==
Using the same example host from above, re-pooling the hosts consists of a similar command from the de-pooling:
$ sudo confctl select dc=<dc>,cluster=<cluster-name>,name=cp*** set/pooled=yes
Alternatively, ssh into the server and run
$ sudo -i pool
Category:How-To
Category:Runbooks
Developer.wikimedia.org

developer.wikimedia.org (Developer Portal) is a central entry point for Wikimedia technical documentation. The portal is a static site built using MkDocs and Material for MkDocs. It uses a custom plugin to generate PO files and integrate with translatewiki.net. For more information about the project, see mw:Developer Portal.
== Service ==
* Maintainer: mw:Wikimedia Technical Documentation Team
* Issue tracker: #wikimedia-developer-portal
* Source code: https://gerrit.wikimedia.org/g/wikimedia/developer-portal/
* Launch task: T261510
The static site runs from the Kubernetes infrastructure.
== Production deployment ==
# Create a patch to update main_app.version in to the desired tag of the wikimedia/wikimedia-developer-portal container. (example)
# CR+2 the helmfile.d patch in gerrit to start the merge process.
# Ssh to the currently active deployment server (deployment.eqiad.wmnet).
# cd /srv/deployment-charts/helmfile.d/services/developer-portal
# Verify that your changes from step 1 are checked out. The git clone is updated once a minute by a timer, so if your changes are not present yet double check that zuul has merged the change and then wait patiently for the update to be fetched.
# Deploy the chart to the staging cluster: helmfile -e staging -i apply --context 5
# Deploy the chart to the codfw cluster: helmfile -e codfw -i apply --context 5
#* Check the production site and build info.
# Deploy the chart to the eqiad cluster: helmfile -e eqiad -i apply --context 5
# When in doubt, check k8s status with kube_env developer-portal <cluster> && kubectl get all
== Demo server ==
A demo server is available at https://developer-portal.wmcloud.org/. This proxy points to an instance in the "devportal" Cloud VPS project which uses Podman to run the latest developer portal container as built by PipelineLib. A systemd timer runs podman auto-update every 5 minutes to poll for newer container versions.
The demo server was manually built and does not currently have any Puppet automation to build a new instance. The manual work needed however is a relatively small amount of effort thanks to the magic of podman:
== See also ==
* https://developer.wikimedia.org/
* mw:Developer Portal
Category:Services
Dickson

== Usage ==
* Hosted dickson.freenode.net
== History ==
* 2013-09-20: Rename manganese -> dickson and prep for debian
* 2014-09-13: freenode announces servers compromised
* 2014-12-30: Removed from public DNS access
* 2015-07-29: Returned to Server spares listing
* 2015-12-07: determine future of dickson - wmf hosted irc server (after compromise)
* 2016-08-11: server was decommissioned
Dickson.freenode.net

Dickson, i.e. dickson.wikimedia.org or dickson.freenode.net, was the Wikimedia-hosted IRC server for Freenode. As of October 2013 it was not yet widely advertised and used mainly by Toolforge bots, etc. It was later added to the freenode public list of servers. It was later compromised along other freenode servers, as detected on September 2014. Initially kept for forensic investigation, it was decided to decommission it on 2016-08-03. In 2017, providing freenode with a new server was declined. 
== See also ==
* dickson: Documentation for the physical hardware box known as dickson.
Category:Decommissioned services
Discovery/Analytics/Glent

Glent is a search platform project that generates query suggestions based on search logs in a batch process. The suggestions are regenerated once a week and shipped to the production cirrussearch clusters to be presented to users when their search query is similar to a query glent has suggestions for. This documention is limited to the analytics portion of the glent suggestions pipeline. Querying the glent data and presenting suggestions to users is implemented in the CirrusSearch MediaWiki extension.
== Development Environment ==
You will need java and maven for the analytics portion of glent.
=== Code ===
The source code is in gerrit project search/glent. To start working with glent clone the repository:
 git clone https://gerrit.wikimedia.org/r/search/glent
== Build ==
You can build the distribution package by running:
 ./mvnw package
and the package will be in the target/ directory.
== Release ==
All deployments of glent code must be accompanied by a release to make the jars available to production services. Releases are handled by the maven-release-plugin and can be run in a two stage process:
#./mvnw release:prepare in the source repository which updates the version numbers. If your system username is different that the one in gerrit, use -Dusername=... option.
#./mvnw release:perform in the source repository - this deploys the artifacts to archiva. 
Note that for the above you will need archiva deployment credentials in your ~/.m2/settings.xml file for the archiva.releases and archiva.snapshots repositories.
If there is some problem with preparing the release the local repository can be reset with:
./mvnw release:rollback
== Deployment ==
The analytics portion of glent is run by an airflow dag in the gerrit project wikimedia/discovery/analytics. Once a new version of glent has been released you will need to update the deployed jars.
Once the jars are updated, deploy the jars to Airflow production.
Distributed tracing

Distributed tracing starts with a single incoming request from a user, and tracks and records all the sub-queries issued between different microservices to handle that request.
If you are totally new to distributed tracing in general, then we recommend beginning with Tutorial/Start
We implement distributed tracing with the OpenTelemetry Collector running on each production host as a collection point, and then using Jaeger as an indexer and search/display interface to the trace data. The data at rest lives on the same OpenSearch cluster that also backs Logstash.
As of August 2024, the only thing in production that emits data to otelcol is Envoy proxy. Our Envoy configuration offers simple opt-in tracing of both incoming and outgoing requests, as long as your application is propagating context -- see #Enabling tracing for a service and also /Propagating tracing context.
Any interested service owners who want to emit OTel themselves should get in touch at TODO (irc, phab, and/or slack?)
== Enabling tracing for a service ==
TODO simple configuration instructions for helm charts in production
After enabling tracing, you should also do a brief audit for any easily-removable PII embedded in traces. Some PII is inevitable, but especially-sensitive data may be scrubbed by writing an otelcol processor rule. SRE is happy to assist with this, but you can find existing examples under transform/scrub: in 
TODO a way for service owners to e2e test in staging? no otelcol deployment there https://phabricator.wikimedia.org/T365809
== More user-facing documentation ==
== Tutorial sub-page: how to read a trace ==
Category:Distributed tracing
/Tutorial/Start
== External link ==
* https://trace.wikimedia.org/
Category:Services
Category:SRE Observability
Distributed tracing/Propagating tracing context

= How to make distributed tracing work for your service =
If you want to make distributed tracing work for your service, you MUST follow this spec.
== General applicability ==
Any services run in production by Wikimedia SRE which both:
* receive requests in the user-facing query path, and
* submit RPC calls or other RPC-like HTTP requests to other services in order to answer those requests
SHOULD follow this specification.
== Context propagation requirements ==
When we say "propagate" a header, we mean that, if a header with that name is set on an incoming request to the service, then it must also be set on all outgoing RPCs made "on behalf" of that incoming request.
Wikimedia services MUST make outgoing RPCs to other services via the Envoy services proxy running as a sidecar.
Wikimedia services MUST propagate the x-request-id header.T201409 Harmonise the identification of requests across our stack
Wikimedia services SHOULD propagate the W3C standard traceparent and tracestate headers.
Following all three of these recommendations means distributed tracing works across service boundaries.
== Easy ways to implement ==
Within Mediawiki core and extensions, use Wikimedia\Http\TelemetryHeadersInterface. When constructing an outgoing request you can call Telemetry::getInstance()->getRequestHeaders(). One of the multiple built-in HTTP clients that you might be already using may do this for you automatically.
Within other microservices, there is not yet an easy, drop-in solution, but we hope to have one soon. Some resources: 
* service-template-node shows you how to do this but there's not a scaffolding/library
* see also https://phabricator.wikimedia.org/T371120
====== References & footnotes ======
Category:Distributed tracing
Distributed tracing/Tutorial/Exploring attributes

=== Attributes ===
== Exploring Attributes in the Jaeger UI ==
The Jaeger UI provides detailed information about the various attributes associated with each span.
=== Accessing the Attributes Panel ===
* Select a Trace - From the trace results list, click on the trace you want to explore
 
* View the Trace Graph - You will see a graphical representation of the trace, showing all spans and their relationships.
* Select a Span - Click on a span in the trace graph, this will expand the span details where you will see clickable groups of attributes e.g. "Tags" and "Process". Click to expand into a full list.
=== Operation and Timing ===
At the top of the span details you'll see these fields in larger text:
* Operation Name: The specific operation being performed by the service (e.g., GET /foo).
* Service Name: The name of the service that executed the span.
* Duration: The total time taken by the span to complete.
* Start Time: The time when the span began executing relative to the beginning of the trace.
Below this, attributes are shown e.g. "Tags" and "Process"
=== Attributes Overview ===
The Attributes details presents a list of key-value pairs that provide detail about the selected span. The data displayed can vary based on the application tracing implementation, for example:
==== HTTP Attributes ====
* http.method: The HTTP method used for the request (e.g., GET, POST)
* http.url: The full URL of the request made
* http.status_code: The HTTP status code returned (e.g., 200, 404, 500).
=== Visual Demo - Expanding Attributes (click to play) ===
=== Filtering and Searching Attributes ===
* Searching - You can search for specific attributes using the tags selector in the Search form. 
=== Next up ===
Distributed tracing/Tutorial/Searching
3
Distributed tracing/Tutorial/Reading complex traces

=== Traces as documentation ===
It's sometimes the case that the architecture of a system — or the design of interactions between systems — can be read from a single trace.
In this example trace, we see a user querying 
=== Serial vs parallel ===
=== Multiple hops in the service mesh ===
=== Next up ===
Distributed_tracing/Tutorial/Exploring_attributes 
2
Distributed tracing/Tutorial/Searching

== Searching ==
When you first load https://trace.wikimedia.org you'll be presented with a search menu. It'll take a moment to load as it fetches the values for several of the attributes that can be searched. Once populated you'll have a number of options to select from via drop down menu.
=== Service ===
* Select the service you want to search traces for. This filters based on the service.name as reported by the application.
* Note: traces ingested without a service name may appear under OTLPResourceNoServiceName. Be sure to check this when troubleshooting/onboarding a new service.
=== Operation ===
* In the Operation dropdown, you can narrow down the search to specific operations or endpoints within the selected service. By default 'all' are searched.
=== Tags ===
* Filter by key-value pairs. For example to search for traces relating to errors try searching for tags:
** error=true
** http.status_code=500 
=== Lookback (Time Range) ===
* Use the Lookback dropdown to choose a time range:
** Predefined ranges such as "Last 1 Hour," "Last 2 Days."
** Custom range: Specify start and end times manually
=== Duration ===
The Duration Search feature in Jaeger UI allows users to filter and find traces based on their execution time. Which can be useful for finding slow traces, performance issues, etc
* Use the "Min Duration" and "Max Duration" fields to specify the length of the trace.
=== Limit Results ===
The maximum number of search results (traces) to fetch
=== Next up ===
Distributed tracing/Tutorial/Instrumenting your own application
4
Distribution upgrades

This page provides some advice and guidelines on how to upgrade distributions on servers.
First of all, decide whether you want to do a dist-upgrade or rather a reinstallation. Reinstallation is a better option where feasible, as the result is a more consistent system. Any system that is setup and configured completely automatically (e.g. using Puppet) should generally be reinstalled rather than upgraded.
Please only follow the supported Ubuntu upgrade paths, i.e. only upgrade between adjacent releases, or from one LTS release to the next.
Distribution upgrades should be done using
 # do-release-upgrade -p
...after first upgrading all packages before you start.
During the upgrade some questions will come up about configuration files that need to be replaced (or not). In general, we want to stay as close to a fresh, new Ubuntu install as possible. This means that we should use the new configuration files whenever possible. Also, most of the changes that were made should be done in Puppet, so will be redone automatically after the upgrade. This should be examined carefully on a case by case basis.
== Things to be aware of ==
* do-release-upgrade will disable the Wikimedia APT repository, as it's unknown. This should be put back immediately after the upgrade, with the proper distribution name of course (e.g. "wikimedia-trusty")
* if, on an internal host, do-release-upgrade stalls for a long time to check for a new release, it's not using the proxy server. Set the environment variable before you run it:
 # export http_proxy=http://webproxy.eqiad.wmnet:8080 (replace 'eqiad' with codfw, ulsfo or esams if needed)
== Distribution-specific notes ==
=== precise -> trusty ===
* python-diamond has transitioned to diamond, after the trusty-wikimedia repo has been re-enabled upgrade with:
 apt-get update ; apt-get -y install diamond python-diamond
=== jessie -> stretch ===
* Distribution upgrades/jessie stretch