nyuuzyou/wikis huggingface dataset:
Deployments/Features Process/E3 Notes

* we have a lot of noisy errors
** syncdir/scap, timeout erros
** you're conflicted LOLNEWB vs OMGWHYDIDN'TYOUSAY?!
** some of them aren't even mediawiki servers (eg: spence)
** the machines should be a list, when it is taken out of rotation it should be removed from this list as well
** treat it like compiling (either no warnings or they are surpressed)
** syncdir, "no syntax errors found..."
*** phplint isn't invoked correctly
** recoverable exceptions are aggregated on fluorine in logs
*** 3 people looking at that file
*** make this more attractive and accessible
*** TODO: ask Antoine about this
** the mystery of scap
*** distinction between syncdir, syncfile, and scap
*** what's going on there
** edit them so they echo out more useful indicators of what is *actually* happening
* TODO: get antoine to talk more about beta to E3
* all test local
* qunit on suacelabs - need this
* easy browser testing on all browsers
* pyramido - testing for next deployment (nearest thurs)
* toro - anything they want to test without messing up pyramido
** get qunit against these
Krinkle/Timo on qunit
* if you have +2 it runs the test right away
** all php unit tests are run, but only on sqlite
*** only php
** none of cucumber or qunit are run in jenkins
* E3 does a/b tests for specific time spans
* reluctant to add deployment steps, even when precautionary
* wish people understood the nature of the work better, time-sensitivity especially
** just the fact that people notice it
* no more manual steps, more automated if their doing smart things
* weirdest with two branches
** for a short period of time only one (last days of cycle)
** people want to deploy more often, but we don't do new branch cuts
** a huge simplificatioin if we had a deploy and a dev branch
* Documenting caching
** we use it on so many levels
** problems in the first 5-10 minutes after deploy could go away
** what are the implications of the message cache?
* people who go in and ssh in to specific apaches vs those who don't
** some animosity
** we aren't training new employees to do it
** TODO: how to poke around using shell?
* Lightning deploy
** there are needs to sync small things that are low risk but still need to go out
** the cost of waiting until the next branch is too much
** an attempt to create designated time to do it, "you break it, you bought it" - karma based ;)
** others will be around
** TODO: write a post to wikitech-l and wikitech.wm.org page about the purpose
* big picture overview of server infrastructure
** repurpose kraken diagram ;)
** who would use it?
* Subdomains on betalabs ?
** certain things turned on/off for testing
* when you modernize you feel you should say you don't need to know the details, it's all automated
** shouldn't shield people too much
Deployments/Features Process/Mobile Notes

Mobile Dev/Deploy
* recently switched to dedicated deploy branch
** called production
** simultaneously
* deploy by recreating production branch of master
** merge freeze 24hours-ish before deploy
** deploy fixes without needing to mess with complicated merges/changes we don't want to deploy yet
* had problems when a new wmf branch was created, it grabed the mobile frontend from master, and it had problems with quality
* currently still need to spend some time testing during deployment
** use test.wikipedia.org heavily
** since they have the 2 hour window why not use it for testing
*** usual devs poke around basic functionality or stuff that has changed
*** michele has a test plans, which is run against 3 version
* normal unstable, beta, alpha
* dont' want to do mobile with core, because it has caused problems in the past
** last major problem about a month ago
** due to recent changes, the possibility of it needed any more is much less
* modeling after the separate branch?
** depends on problems
** what is the tech plan of the product managers
** switching to dedicated production branch helped us
* sometimes they depend on core changes and thus, as a part of their deployment they have to deploy core changes as well, usually quite isolated
* I (Michelle) have a testing environment that I deploy merged code to so that the Designers and Product can see changes prior to the release it's http://staging.wmflabs.org/. This staging environment and test.m are only a piece of the puzzle because of caching which as you know adds a bit of a wrench in out deployments.
Chat with Maryana:
* two staging areas
** alpha and beta mobile site
** all three are different
* have to test each separately
* big challenge
** what is a real bug but not just a product of test environment
** eg test wiki
** betalabs is still wonky
** sometimes find out stuff a couple weeks later because of lack of intense testing on beta
* always the actual live production needs
* always begging to flush cache/sync stuff etc
* there is a more regular
Deployments/Features Process/TODOs

* - Ask Timo/Antoine re QUnit tests
* - Ask James F re VisualEditor using beta
* Ask QA re VE using more browser tests
* Ask Reedy if the VE deploy across both live MW versions messes him up
* Follow up with Ryan Lane re git-deploy and large files (git-annex style? something else?)
* Make the recoverable exceptions that are aggregated on fluorine more readable/accessible
* Make clear (document somewhere) the distinction between syncdir, syncfile, and scap 
* get antoine to talk more about beta to E3 
* qunit on suacelabs - need this (now on Jenkins)
* Get QUnit running against piramido and toro Labs instances for E3
* Create a "how to poke around using the shell" for developers (see E3 notes)
* write a post to wikitech-l and wikitech.wm.org page about the purpose of the Lightning Deploys
* Big Picture overview of the servers
Deployments/Features Process/VisualEditor Notes

= Deploying? =
VisualEditor rides the platform train
* they have milestone for every second monday
* assigning bugs to milestones, retargets
* operate ignorantly of deployments, kinda
* occasionally modifications in core
* deployment with news coverage, keep close watch
Parsoid is different story
* nodejs
* only thing being deployed with git-deploy!
* unversioned blobs of data
** external npm modules
** 39 megs
** talk with Ryan about git-annex re this
*** git-annex with bittorrent?
* develop on master
* James pays attention to deploys, talk with him
Every friday look at master vs every other
= Testing? =
* everyone has the connection between visual editor/local mediawiki setup?
* not on beta at all
* ask james about going to beta
* vague ideas of browser testing
* ask Chris M about that
* other than manual and Wikipedia :) testing
* fairly good unitest coverage
* Ed is working on more unittests
* much of the UI stuff is now getting unittested through 
* passed 100 tests
* timo and antoine working on qunit to run on Jenkins
** currently tests jslint
** would be good for qunit
* Ask Timo and Antoine re that
= Pain Points =
* parsoid is a separate service that is deployed separete
** making parsoid backwards compatible
** not runnable since mid-January
** parsoid is now rewriting the serializer
** hopefully in a couple weeks it should be ready
** code in gerrit ready but not able to be merged because parsoid is old
* sometimes upgrade VE on all deployed mediawikis
** messes up sam?
** if every week, then less need to do out of band deploys
The deployments "just run" and it is working well
= James and Beta =
TODO (Greg)
* Ask Timo and Antoine re qunit tests
* ask james about going to beta
* talk with Ryan about git-annex re git-deploy/git-annex and VE's use-case of 39 megs of npm's in git
Deployments/Inclusion criteria

The general outline of changes that require scheduling on the Deployments calendar as their own window, outside of Backport windows.
== Things that require their own deploy window ==
=== Long running tasks ===
While not strictly a deployment, performing a long running task (more than 1 hour), eg. migration scripts, are required to be scheduled on the calendar. These can encounter issues when code is updated while a script is being run. The window can be short (e.g. 30min), and cover the anticipated start time and mention the estimated length for the task.
=== Could make the cluster fall over ===
* Database schema changes.
** Any change altering the databases schema must be asked on the schema changes page prior to deployment.
* Backend/underlying software updates (e.g. PHP, HHVM, MariaDB, etc.)
* Data-center changes (e.g. new DC coming online or datacenter going offline.
=== Notable production code change ===
* Out-of-sequence MediaWiki and extension code updates (including wikitech)
* BetaFeatures
** Anything going into BetaFeatures (new features for users)
** Anything going out of BetaFeatures (i.e. will now be user-default, or included in User Preferences)
* Adding a new extension (either cluster-wide or per wiki)
* Anything which introduces a new cookie (or renames an existing one)
* Big default behavior changes
=== Needs read-only time ===
Including on a per-wiki basis, like database shard migrations.
=== Cross-cutting impact or large unknowns ===
* e.g. changes to deployment system
* changes that affect load characteristics substantially or in unknown ways, e.g. performance
* this section is informed by past breakages, as well
== How to add a special deployment window ==
# Ensure you have the support for deploying you need (e.g., someone to run maintenance scripts, someone to deploy, someone to run sql changes)
# Find a date/time in the Deployment Calendar
# Add a new Deployment calendar event card with the details of your deployment
== Unsure ==
If in doubt, talk to Release Engineering.
__NOTOC__
Category:Deployment
Deployments/Risky change template

You have a patch that you feel might cause problems on deployment: now what?
Below is an example template to add as a comment on a Phabricator train blocker task to alert the deployers that there are risky patches riding the deployment train.
== Template ==
##### Risky Patch! 🚂🔥
* **Change**: https://gerrit.wikimedia.org/r/0000000
* **Summary**:
** !! Why is it risky? !!
* **Test plan**:
** !! How will it be verified? !!
* **Places to monitor**:
** [Logstash: mediawiki-errors](https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors)
** [Logstash: mediawiki-client-errors](https://logstash.wikimedia.org/app/dashboards#/view/e51e9de0-9339-11eb-944f-51f280f5c035)
** [Grafana: mediawiki-errors](https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts)
* **Revert plan**: !!Rollback train, or Revert patch, or …!!
* **Affected wikis**:
```lines=10
testwiki
test2wiki
```
* **IRC contact**: `your_name`, `someone_else` (timezone UTC+-??)
* **UBN Task Projects/tags**: #your_team
* **Would you like to backport this change rather than ride the train?**: Yes/No
== Example ==
##### Risky Patch! 🚂🔥
* **Change**: https://gerrit.wikimedia.org/r/0000000
* **Summary**: (Why is it risky?)
** Big refactor in Foobar.
** Lots of callers to Foobar in extensions that are not all covered by tests.
* **Test plan**:
** FlaggedRevs are enabled on test2wiki. Mostly manual testing unfortunately and log watching.
* **Places to monitor**:
** [Logstash: mediawiki-errors](https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors)
** [Grafana: mediawiki-errors](https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts)
* **Revert plan**: Rollback train.
* **Affected wikis**:
```lines=10
alswiki
arwiki
bewiki
bnwiki
dewiki
enwiki
```
* **IRC Contact**: `Pchelolo`, `DannyS712`
* **UBN Task Projects/tags**: #core_platform_team
Deployments/Train

The Deployment Train is a weekly process led by the Release Engineering Team, to deploy the latest alpha version of MediaWiki at WMF to production.
Typical numbers as of September 2021:
* 198 code repositories (mediawiki/core, mediawiki/vendor, 188 MW extensions, and 8 MW skins).
* 978 wikis.
* between 150 and 1000 patches since the last train.
== Current week ==
* Find the currently deployed branch/version of a wiki at https://versions.toolforge.org/
* Train blocker task: https://train-blockers.toolforge.org/
* Change logs for WMF releases: https://www.mediawiki.org/wiki/Category:WMF_Releases
== Schedule ==
See also MediaWiki /Roadmap on mediawiki.org
NOTE: The mention of Tuesday, Wednesday and Thursday in the diagram are for the following week, not the week the patch was merged in. So, after 10 mins of the patch merged, it goes live on beta for people to test things and make sure to catch bugs in case there are any.
Assuming everything goes well, the changes that got merged during a particular week only ride the train the week after from Tuesday to group0, Wednesday to group1 and Thursday to group2.
Sunday	Monday	Tuesday 	Wednesday 	Thursday 	Friday	Saturday
 	 	Deploy to group 0	Deploy to group 1	Deploy to group 2	 	
== Groups ==
=== Tuesday ===
Each Tuesday a Release Engineering team member cuts a new branch of all 198 repositories in a Wikimedia Production deployment. The branch is always prefixed with wmf/ and then a version number. For example: wmf/1.40.0-wmf.1.
The train conductor deploys that to Group0 wikis (see ). This group includes:
*www.mediawiki.org (mediawikiwiki)
*test.wikipedia.org (testwiki)
*test.wikidata.org (testwikidatawiki)
*office.wikimedia.org (officewiki)
* all closed wikis
=== Wednesday ===
On Wednesday Release Engineering leads a log triage session to find and file any new errors.
Following the log triage, the train conductor for the week makes a go/no-go decision to deploy to Group1 wikis.
Group1 is non-Wikipedias, see .
Group1 includes:
*www.wikidata.org (wikidatawiki)
**.wiktionary.org (all Wiktionary sites)
**.wikisource.org (all Wikisource sites)
**.wikinews.org (all Wikinews sites)
**.wikibooks.org (all Wikibooks sites)
**.wikiquote.org (all Wikiquote sites)
**.wikivoyage.org (all Wikivoyage sites)
**.wikiversity.org (all Wikiversity sites)
*commons.wikimedia.org (Wikimedia Commons)
*meta.wikimedia.org (Meta-Wiki)
*incubator.wikimedia.org (Incubator)
*ca.wikipedia.org (Catalan Wikipedia)
*he.wikipedia.org (Hebrew Wikipedia)
*it.wikipedia.org (Italian Wikipedia)
*test2.wikipedia.org (test2wiki)
*wikifunctions.org (wikifunctionswiki)
*and others…
=== Thursday ===
If no blockers are found, the train conductor pushes code to all other Wikipedias, see .
Group2 includes en.wikipedia.org (English Wikipedia).
Deployments/Train/Transition-201306

This is here for historical reasons. This was the page used to describe two different one-week deploy cycle proposals as we were wanting to transition from a two-week cycle. The final decision was to go with Option B. This page below is as "/Deployments/One week" was as of May 21st, 2013.
== Current deploy schedule stats ==
* lifespan of any particular wmfXX branch: 24 days
* days of development per wmfXX branch: 14
== Proposals to get to a one-week deployment cycle==
=== Option A ===
Summary: 2 deploys a week, in 2 phases, starting on Monday
==== Deploy calendar ====
 	Sunday	Monday 	Tuesday	Wednesday	Thursday 	Friday	Saturday
week0	 	wmf1 to set 0	 	 	wmf1 to set 1	 	
week1	 	wmf2 to set 0	 	 	wmf2 to set 1	 	
week2	 	wmf3 to set 0	 	 	wmf3 to set 1	 	
* Wikis pointing to set 0
** MediaWiki.org
** testwiki
** test2wiki
** All non-Wikipedia sites (Wiktionary, Wikisource, Wikinews, Wikibooks, Wikiquote, Wikiversity, and a few other sites)
*** Some Wikipedias may move to this slot if the community desires to help debug issues earlier in the cycle. Tentatively this might include Chinese Wikipedia.
* Wikis pointing to set 1
** All Wikipedia
==== Lifespan ====
: Lifespan of any particular wmfXX branch (ie how long it stays on production): 10 days
==== Stats ====
* Days of development per wmfXX branch: 7 days
==== Pros/cons ====
Pros 	Cons
Short lifespan 	Monday Holidays
Simple to explain 	Short Maturation, ie: time to all wikis
3 days of two versions live	No window for automated tests against the production environment before going to all Non-Wikipedias
=== Option B ===
Summary: 2 deploys a week, in 3 phases, starting on Thursday
==== Deploy calendar ====
 	Sunday	Monday 	Tuesday	Wednesday	Thursday 	Friday	Saturday
week0	 	 	 	 	wmf1 to set 0	 	
week1	 	wmf1 to set 1	 	 	wmf1 to set 2
wmf2 to set 0	 	
week2	 	wmf2 to set 1	 	 	wmf2 to set 2
wmf3 to set 0	 	
* Wikis in set 0
** MediaWiki.org
** testwiki
** test2wiki
* Wikis in set 1
** All non-Wikipedia sites (Wiktionary, Wikisource, Wikinews, Wikibooks, Wikiquote, Wikiversity, and a few other sites)
*** Some Wikipedias may move to this slot if the community desires to help debug issues earlier in the cycle. Tentatively this might include Chinese Wikipedia.
* Wikis in set 2
** All Wikipedia
==== Lifespan ====
: Lifespan of any particular wmfXX branch (ie how long it stays on production): 14 days
==== Stats ====
* Days of development per wmfXX branch: 7 days
==== Pros/cons ====
Pros 	Cons
Longer maturation 	Longer lifespan
Longer window (3 days) for automated tests on production	always two versions live
 	Monday Holidays
Deployments/Train vs backport

There are types of changes that should be deployed via the train, types of changes that should be deployed via backport windows, and types of changes that should be deployed via dedicated deployment windows. This document is an attempt to codify the criteria used to determine the method by which a change should be deployed.
This change deployment criteria uses IETF RFC 2119 keywords.
== Train ==
* New extensions MUST be deployed via the train (before they are enabled)
** Justification: Localisation (l10n) is generated for the whole site not for only a single version. L10n logic currently expects new extensions to be deployed via the train (see: mw:Writing_an_extension_for_deployment#Preparing_for_deployment for more details)
* Changes to l10n SHOULD be deployed via the train
* Changes requiring library upgrades in mediawiki/vendor SHOULD be deployed via the train
* Changes requiring several commits across many branched repositories SHOULD be deployed via the train
* Any change to MediaWiki core, extensions, or skins MAY be deployed via the train
** NOTE: For many types of changes it is often more appropriate to deploy these changes via backports and that SHOULD be the default choice.
== Backports or Dedicated windows ==
* Changes to operations/mediawiki-config MUST be deployed via backport or a dedicated deployment window
* Changes meeting the criteria for dedicated windows MUST NOT be deployed via the train and SHOULD be deployed via dedicated windows
** Justification: train conductors don't usually understand the aspect or subsystem being changed, and having people who do understand it in dedicated deployment windows means it's less likely to break things badly, and any breakages are more likely to be noticed as breakages.
* Risky changes SHOULD be deployed in dedicated windows but MAY ride the train provided that the developer comments on the train task
* Fixes for critical regressions SHOULD be deployed as soon as possible via a dedicated window or a backport window.
* Changes that are incompatible between wikiversions, such as changes to job serialization format SHOULD be deployed via dedicated windows rather than deployed via that train
* Non-trivial changes to CSS, JavaScript, PHP, or configuration of MediaWiki core, extensions, or skins that does not require PHP library upgrades SHOULD be deployed via backport or dedicated window
* Trivial changes to CSS, JavaScript, PHP, or configuration of MediaWiki core, extensions, or skins that does not require PHP library upgrades MAY be deployed in backport windows (provided the backport deployer is comfortable deploying the change)
Deployments/Training/About

==About deployment trainings==
Welcome and we hope you will join the deployers' club!
Come often to trainings to learn and/or help out, it's great to have you!
Become a trainer too, we'd love that as well!
If things go wrong, people will be there to answer questions or to pitch in.
Help is ALWAYS available.
===Prereqs===
If you will be hands on at a training, here is what you need.
* can ssh to deployment host
* familiar with git log, fetch, rebase (not "expert" but "familiar with"!)
* can get around phabricator
* can get audio/video working for google meet
* can lurk in the libera.chat irc channel
If you will just be following along, here is what you need.
* can lurk in the libera.chat irc channel
* can get audio/video working for google meet if you decide to join
===Getting Ready to Deploy===
We'll discuss the steps involved in getting ready to deploy:
* how to check the calendar, show up in the irc channel, and make sure the patch owners are around
* how to see who your co-deployers are, who can take turns or help out if needed
* how to check the associated phabricator tasks briefly, and the patches themselves
* how to make sure patches are already merged in to master for MediaWiki core
* how to make sure patch owners know how to test and check that their fix or patch worked via mwdebug host
===Deploying a patch===
We'll discuss what you need to do to deploy one patch.
Deploying for limited testing:
* how to merge the backport or the config change
* how to check that there's not uncommitted stuff on the deployment host
* how to get the patch onto the host
* how to push it to the mwdebug host for testing
* how to watch logs and have the patch owner test
Completing the deployment: 
* how to push the patch everywhere
* how to watch logs and make sure things look ok
Dealing with trouble:
* how to ask for help, which is ALWAYS available
* pointers to docs for how to revert, if we encounter no trouble
===When you're done===
When you've gone through a training, if you don't feel ready to deploy, come back! Do multiple trainings!
If you do feel ready, come back! Help us train more deployers!
WE NEED DEPLOYERS.
WE NEED YOU. <3
Deployments/Yearly calendar/2021

Deployment disruptions 2021
What 	When 	Deployment decision 	What happened?
Martin Luther King Day 	Mon 18 Jan 	No deploys 	No MW deploys
Presidents' Day 	Mon 15 Feb 	No deploys 	One MW deploy
WMF Holiday 	Mon 29 Mar 	No deploys 	One MW deploy, one MW train group deploy
WMF Holiday 	Fri 9 Apr 	No deploys 	No MW deploys
WMF Holiday 	Mon 12 Apr 	No deploys 	Sixteen MW deploys
Earth Day 	Thu 22 Apr 	No Train (1.37.0-wmf.2), and no deploys that day 	No MW deploys
WMF Holiday 	Fri 28 May 	No deploys (not that there are ever deploys on Friday) 	One MW deploy
Memorial Day 	Mon 31 May 	No deploys 	No MW deploys
Engineering Productivity Team Offsite 	Mon 31 May–Fri 4 Jun	No Train (1.37.0-wmf.8), but deploys OK 	—
WMF All-Hands 	Tue 15–Thu 17 Jun 	No Train or deploys all week (1.37.0-wmf.10) 	One MW deploy
Juneteenth 	Fri 18 Jun 	No deploys (not that there are ever deploys on Friday) 	No MW deploys
Fourth of July + following WMF holiday 	Mon 5–Fri 9 Jul 	No Train or deploys all week (1.37.0-wmf.13) 	Many deploys
International Day of the World’s Indigenous Peoples	Mon 9 Aug 	No deploys 	Many deploys – NOTE: all before 17:00UTC—9am PDT)
Labor Day + Release Engineering focus week 	Mon 6–Fri 10 Sep 	No Train (1.37.0-wmf.22), but deploys OK except on Monday	—
Indigenous Peoples' Day 	Mon Oct 11 	No deploys 	No MW deploys
Election Day 	Tue 2 Nov 	EU Train week 	—
Veterans' Day 	Thu 11 Nov 	No Train (1.38.0-wmf.8), and no deploys that day 	Ten MW deploys
Thanksgiving Day 	Thu 25 & Fri 26 Nov 	No Train (1.38.0-wmf.10), and no deploys those days 	No MW deploys
End of Year Holiday 	Mon 20 Dec–Mon 3 Jan	No Train or deploys (1.38.0-wmf.14, 1.38.0-wmf.15) 	One MW deploy