nyuuzyou/wikis huggingface dataset:
Deployment pipeline/FAQ

=== What does this mean for me? ===
* If you are currently running a service in production, we'll talk to you about migration to kubernetes.
* If you are building a new service to go into production, please contact us to make sure your service is "born kubernetes", rather than having to migrate a month after deployment.
* If you don't have or plan to have a service, don't worry about things for now. Changes to non-services will take a while.
* What is the pipeline intended to be?
* What is the purpose of the pipeline?
* How does the Wikipedia movement benefit from the pipeline?
* How does a developer benefit from the pipeline?
* How does the WMF release engineering team benefit from the pipeline?
* How does the pipeline work, in broad strokes?
* How does a developer use the pipeline?
** What software does a developer need to install locally to use the pipeline?
* How can one check the status of the pipeline?
* How can one get alerted about problems of things in the pipeline?
* How can one see statics of the pipeline in operation?
** Number of successful builds and deployments to production?
** Number of build failures?
* How can one learn how the pipeline is implemented?
=== What steps are part of the pipeline? ===
1. It starts with using Blubber to create a Docker image.
also see: Blubber/Pipeline
=== How can I view application logs from my service in production Kubernetes? ===
These logs are available in Logstash#Kibana. This dashboard should allow you to select your service (and optionally Kubernetes cluster) to view logs.
Deployment prep

This is a collation of the best practices steps you should take to prepare your new API or Service for deployment by the SRE team.
We are not writing anything new here just structuring what already exists.
What's the value of following this process?
* Makes it easier and faster to be deployment ready, the intent is that you won't run into gotchas
* Makes us compliant with Security and Legal
When should you follow this process?
* Once your new API/Service is ready to be served via our production systems
Get your repo on gerrit 
In order to get the benefits of automated CI/CD already setup on Gerrit, Request new gerrit repo You have to already have a gerrit account that the repo creator will assign permissions to inorder to push to the repo Consult on repo naming conventions. Wrong naming will result in archiving of the repo and you will have to create a new rightly named one
Update gerrit repo with latest code
Once you have the new empty gerrit repo, in order to be able to push other people’s code commits you will require special powers on Gerrit by joining the Trusted Administrators Group
Ping someone already in the Trusted Administrators Group to add you there
Push the code commits
Security and Legal Compliance
Use Security Review - Standard Operating Procedure to find our if your new API/Service requires a security review. Not all Services/APIs will require this Security review
Security reviews all new review requests on a quarterly basis. It will be most nice to get your request in before the tasks of the quarter are finalized.
If Security team decide you require a Legal review they will be the ones to forward you to Legal for review, there is no need to ping Legal directly
Can we own our own security checking?
To some extent yes it is possible check for security vulnerabilities in your codebase by yourself to minimize on the back and forth feedback from Security team. For this you can consult
| Security Review - Some tools to help
Security team will still have to do the checks from their end.
Handle tech debt
There’s a certain threshold that all Wikimedia systems must meet in order to be deployable by SRE. Currently no example deployable Wikimedia service exists
It may not be possible to exhaust the list here so budget some time to fix stuff on the fly as you receive feedback from SRE
Some confirmed components that must definitely be there
# Service name
#* No underscores, use hyphens
#* Consistency in the naming of the service within the codebase itself and all docs.
# Unit tests
#* Your service must have true unit tests that don’t have external dependencies.
#* These should be runnable from a basic `make test` build step
# Metrics dashboards
#* Accessible via /metrics endpoint 
#* A /healthz endpoint
#* Provide metrics gathered via Prometheus and dashboards to monitor and assess
#* For AQS 1.0 for example, | Example Grafana dashboard
* Documentation updated
** Revise all endpoint and overview docs for completeness and consistency
** Add make docs command to update the docs
*** Uses `swag` —-- swag init --markdownFiles
Deployments

This page tracks upcoming deployments of software to the Wikimedia Foundation servers.
== Getting started ==
Ensure you joined the IRC channel as all deployment-related communications happen there.
If you need help, contact Release Engineering on IRC at ; and ping Tyler (thcipriani).
* MediaWiki is deployed weekly through the Deployment Train. Other services follow their own schedule.
* Times are pinned to San Francisco, thus the UTC time changes in March and November per DST.
* Prefer regular Backport windows over adding new windows. To request deployment of a config change or backport, add your username and Gerrit URL to one of the backport windows on this page. You must be online in #wikimedia-operations on IRC during your deployment and install WikimediaDebug ahead of time. The #wikimedia-operations channel requires you to register your nickname before you can join.
** You can use the to more easily edit this page.
* Tasks that meet Inclusion criteria require their own windows, which includes long-running tasks. Schedule more time than you think you need to account for delays and set backs, we recommend one hour for most tasks.
**To create or modify a recurring deploy window, send a patchset to deployments-calendar.yaml file in repos/releng/release.git.
**To create an one-off window, simply edit this page accordingly
** Announce changes to the ops mailing list ahead of time if they are likely to affect HTTP caching, introduce new cookies, or utilize new database tables.
** Announce deployments of major features to the community via Tech News and/or via other Product communication channels.
* Something went wrong? See Incident response. Is there a user-impacting problem? Communicate in the IRC channel. If there is a Phabricator task, ensure #Wikimedia-Incident is tagged, and consider setting the Unbreak Now priority.
__TOC__
Category:Deployment
==Week of February 17==
======
======
======
======
======
======
======
==Week of February 24==
======
======
======
======
======
======
======
Deployments/Archive/2014/01

=== Week of January 6 ===
; Week of:
* Upgrade Zuul (Antoine) 
; By day
=== Week of January 13 ===
; Week of:
* nothing up
; By day
=== Week of January 20 ===
; Week of:
* bring ULSFO (west coast caching center) back online (Opsen)
* mw:Architecture Summit 2014 on Thurs/Friday
; By day
=== Week of January 27 ===
; Week of:
* Jan 31st: Mobile Android Beta release (no specific time)
; By day
Deployments/Archive/2014/02

=== Week of February 3 ===
; Week of:
* ULSFO going online (Brandon/Faidon), in stages
; By day
=== Week of February 10th ===
; Week of:
* nothing yet
; By day
=== Week of February 17th ===
; Week of:
* nothing yet
; By day
=== Week of February 24th ===
; Week of:
* Monday morning (European time): Antoine and a European Opsen, fix for doc.wikimedia.org and integration.wikimedia.org http->https redirects - https://gerrit.wikimedia.org/r/#/c/111917/
* Brandon B will be upgrading Varnish instances to a newer version (previously deployed/tested at ulsfo) slowly over the week, avoiding maint windows
; By day
Deployments/Archive/2014/03

=== Week of March 3rd ===
; Week of:
* March 3rd: The migration of WMF Labs from pmtpa to eqiad begins
** new instance creation disabled in pmtpa, only available in eqiad
; By day
=== Week of March 10th ===
; Week of:
* nothing yet
; By day
=== Week of March 17th ===
; Week of:
* 17 March: End of 'soft-migration' of WMF Labs to eqiad, beginning of hard migration
** Tool migration instructions
** Project migration instructions
; By day
=== Week of March 24th ===
; Week of:
; By day
=== Week of March 31st ===
; Week of:
; By day
Deployments/Archive/2014/04

=== Week of April 7th ===
; Week of:
* nothing yet
; By day
=== Week of April 14th ===
; Week of:
* nothing yet
; By day
=== Week of April 21st ===
; Week of:
* Tampa DC: moving remaining in-use machines up to the still in use floor, wiping/shipping still good machines to new DC, getting rid of the rest.
* CirrusSearch: New machines coming online, begin background indexing process on the big wikis.
; By day
=== Week of April 28th ===
; Week of:
* nothing yet
; By day
Deployments/Archive/2014/05

=== Week of May 5th ===
; Week of:
* 7th - 12th: Mediawiki Hackathon in Zürich
; By day
=== Week of May 12th ===
; Week of:
* nothing yet
; By day
=== Week of May 19th ===
; Week of:
* nothing yet
; By day
=== Week of May 26th ===
; Week of:
* nothing yet
; By day
Deployments/Archive/2014/06

=== Week of June 2nd ===
; Week of:
* nothing yet
; By day
=== Week of June 9th ===
; Week of:
* nothing yet
; By day
=== Week of June 16th ===
; Week of:
* nothing yet
; By day
=== Week of June 23rd ===
; Week of:
* June 25: Android Wikipedia app release to market
* Re-enable wmgGettingStartedRunTest with new version (spec, Trello card)
; By day
=== Week of June 30th ===
; Week of:
* nothing yet
; By day
Deployments/Archive/2014/07

=== Week of July 7th ===
; Week of:
* nothing yet
; By day
=== Week of July 14th ===
; Week of:
* July 14th: submit iOS app to AppStore (unknown review timeline from them)
* July 15th: Target date for Content Translation to be deployed on the Beta Cluster
; By day
=== Week of July 21st ===
; Week of:
* Deploy task recommendations (in GettingStarted extension) to English Beta Labs. 
; By day
=== Week of July 28th ===
; Week of:
* nothing yet
; By day
Deployments/Archive/2014/08

=== Week of August 4th ===
Week of Wikimania.
No planned deployments. Only emergency deploys.
=== Week of Aug 11th ===
; Week of:
* nothing yet
; By day
=== Week of Aug 18th ===
; Week of:
* nothing yet
; By day
=== Week of Aug 25th ===
; Week of:
* nothing yet
; By day
Deployments/Archive/2014/09

=== Week of September 1st ===
; Week of:
* nothing yet
; By day
=== Week of September 8th ===
; Week of:
* 2014-09-12 13:00 PDT, with , to fix . Approved by Greg G. and Erik M.
; By day
=== Week of September 15th ===
; Week of:
* nothing yet
; By day
=== Week of September 22nd ===
; Week of:
* nothing yet
; By day
=== Week of September 29th ===
; Week of:
* nothing yet
; By day
Deployments/Archive/2014/10

=== Week of October 6th ===
; Week of:
* October 6: HHVM serving 1% of reader traffic
* October 6 (tentatively): Internal (non-public) RESTBase deploy (), followed by load testing
; By day
=== Week of October 13th ===
; Week of:
* October 13: 5% of reader traffic will be served by HHVM (app servers migrated: 15)
; By day
=== Week of October 20th ===
; Week of:
* October 20: 10% of reader traffic (Number of app servers: TBD based on our experience up until that point)
* TBD: IEG review system deployed (needs ops support)
; By day
=== Week of October 27th ===
; Week of:
* Localisation update will be temporarily disabled starting from 2014-10-27 because of migration of plural rules to CLDR 26 (bug 62861).
; By day
Deployments/Archive/2014/11

=== Week of November 3rd ===
; Week of:
* Fundraising tests throughout the week
* HHVM back to 10% of anonymous traffic
; By day
=== Week of November 10th ===
; Week of:
* Fundraising tests throughout the week
; By day
=== Week of November 17th ===
; Week of:
* HHVM:
** Monday 17/11: 25% of anonymous users
** Thursday 20/11: Remove cache separation between HHVM and Zend, convert 1 appserver in the Zend pool
* November 21st - 24th: Phabricator Bugzilla migration: Tracking task and Timeline
* Fundraising tests throughout the week
; By day
=== Week of November 24th ===
; Week of:
* Firefox:
** Nov 25: Firefox 34 with default HTTPS search on Wikipedia (and sometimes Wiktionary) 
* mw:HHVM:
** Nov 25: dissolve the distinction between HHVM and Zend appservers and progressively move every appserver to HHVM
* Fundraising tests throughout the week
; By day
Deployments/Archive/2014/12

=== Week of December 1st ===
; Week of:
* Fundraising tests throughout the week
** 100% to English
; By day
=== Week of December 8th ===
; Week of:
* Fundraising throughout the month
* HHVM: reimaging servers, should be completed by EOY
; By day
=== Week of December 15th ===
; Week of:
* Fundraising throughout the month
* HHVM: reimaging servers, should be completed by EOY
; By day
=== Week of December 22nd ===
* No planned deploys
* HHVM
** Should be 100% of all mediawiki servers by EOY
* Fundraising 
** Now through End of Year: tests throughout the week
If you do need to deploy something, please add it to this calendar, as usual.
=== Week of December 29th ===
* No planned deploys
** Permanently enable unregistered users editing on it.m.wikipedia.org (no-op if done before 2014-12-31T23:59 +0:00).
* HHVM
** Should be 100% of all mediawiki servers by EOY
* Fundraising 
** Now through End of Year: tests throughout the week
Deployments/Archive/2015/01

=== Week of Jan 5th ===
; Week of:
; By day
=== Week of Jan 12th ===
; Week of:
* HHVM rolled out to the jobrunners
; By day
=== Week of 19th ===
; Week of Summary:
* No MediaWiki train deploy due to WMF All Hands
* No deploys Mon/Wed/Thurs/Fri
* SWAT/etc on Tues OK
; By day
=== Week of 27th ===
; Week of Summary:
* No MediaWiki train deploy due to MediaWiki Dev Summit
* SWAT all week as normal
* no other deploys Mon/Tuesday
; By day
=== Week of Jan 12th ===
; Week of:
* HHVM rolled out to the jobrunners
; By day
Deployments/Archive/2015/02

=== Week of February 2nd ===
; Week of:
* Wednesday: Graphite HW migration - T85909
; By day
=== Week of February 9th ===
; Week of:
* nothing yet
; By day
=== Week of February 16th ===
; Week of:
* Tuesday-ish: RestBase deployment phab:T1228
; By day
=== Week of February 23rd ===
; Week of:
* Mid-week: RestBase deployment phab:T1228
; By day
Deployments/Archive/2015/03

=== Week of March 2nd ===
; Week of:
* Nothing yet
; By day
=== Week of March 9th ===
; Week of:
* Early week: Deploy RESTBase
** https://phabricator.wikimedia.org/T89481
* Mid week: Use RESTBase for VisualEditor from the MediaWiki Virtual Rest Service
** https://phabricator.wikimedia.org/T89066
; By day
=== Week of March 16th ===
; Week of:
* Progressive rollout to all wikis for restbase - phab:T89066
; By day
=== Week of March 23rd ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
* Friday, during EU working hours: Amsterdam datacenter network maintenance (should have no user impact)
; By day
=== Week of March 30th ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
; By day
Deployments/Archive/2015/04

=== Week of April 6th ===
; Week of
* See the corresponding "week of" column in the roadmap project workboard.
; By day
=== Week of April 13th ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
* Wed April 15th: SUL Finalization - The renaming of non-unique user accounts will begin - Announcement
* Thu April 16th at 16:00 UTC: Tool Labs - tools-db configuration update/maintenance - Announcement
; By day
=== Week of April 20th ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
; By day
=== Week of April 27th ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
; By day
Deployments/Archive/2015/05

=== Week of May 4th ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
; By day
=== Week of May 11th ===
; Week of:
* See the corresponding "week of" column in the roadmap project workboard.
* May 13: Elasticsearch rolling restart
* Enable remaining public projects in RESTBase: T93452
; By day
=== Week of May 18th ===
; By day
=== Week of May 25th ===
; By day
Deployments/Archive/2015/06

=== Week of June 1st ===
; By day
=== Week of June 8th ===
; By day
=== Week of June 15th ===
; By day
=== Week of June 22nd ===
; By day
=== Week of June 29th ===
* June 30th - Midnight UTC - Leap Second
* 1.26wmf12 - The old continuation mode of the API for action=query doesn't work any more. 
; By day
Deployments/Archive/2015/07

=== Week of July 6th ===
; By day
=== Week of July 13th ===
* Upgrade production Elasticsearch cluster to 1.6.0
** Starting Monday 7:30 am pacific time
** Expected to take 3 days
** Will not block any other deployments
; By day
=== Week of July 20th ===
* Week of:
** ...
; By day
=== Week of July 27th ===
* Week of:
** Upgrade/rolling restart of ElasticSearch - phab:T106090
; By day
Deployments/Archive/2015/08

=== Week of August 3rd ===
* Week of:
** nothing yet
; By day
=== Week of August 10th ===
* Week of:
** nothing yet
; By day
=== Week of August 17th ===
* Week of:
** nothing yet
; By day
=== Week of August 24th ===
* Week of:
** nothing yet
; By day
=== Week of August 31st ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2015/09

=== Week of September 7th ===
* Week of:
** Monday Sept 7th: US Holiday, no normal deploy windows
; By day
=== Week of September 14th ===
* Week of:
** nothing yet
; By day
=== Week of September 21st ===
* Week of:
** nothing yet
; By day
=== Week of September 28th ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2015/10

=== Week of October 5th ===
* Week of:
** nothing yet
; By day
=== Week of October 12th ===
* Week of:
** Operations Team Offsite, please don't do anything stupid
** No more deploys the rest of the week
; By day
=== Week of October 19th ===
* Week of:
** nothing yet
; By day
=== Week of October 26th ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2015/11

=== Week of November 2nd ===
* Week of:
** nothing yet
; By day
=== Week of November 9th ===
* Week of:
** nothing yet
; By day
=== Week of November 16th ===
* Week of:
** nothing yet
; By day
=== Week of November 23rd ===
* Week of Thanksgiving (US Holiday)
** High Priority SWATs only on Mon/Tuesday (Wednesday is "Friday")
*** "High Priority" means security and data loss. Anything else needs prior approval from at least Greg.
** No normal deploys
; By day
=== Week of November 30th ===
* No normal deploys
* High Priority SWATs only
** "High Priority" means security and data loss. Anything else needs prior approval from at least Greg.
; By day
Deployments/Archive/2015/12

=== Week of December 7th ===
* Week of:
** Move Catalan and Hebrew Wikipedias to "group1" (ie: Wednesday) deploy group - https://gerrit.wikimedia.org/r/#/c/253371/ - 
; By day
=== Week of December 14th ===
* Week of:
** nothing yet
; By day
=== Week of December 21st ===
* No normal deploys
* High Priority SWATs only
** "High Priority" means security and data loss. Anything else needs prior approval from at least Katie (K4-713) or Chad Horohoe (ostriches).
* Short week because of Christmas on Friday (aka Thursday == "No deploys Friday")
; By day
=== Week of December 28th ===
* No normal deploys
* High Priority SWATs only
** "High Priority" means security and data loss. Anything else needs prior approval from at least Katie (K4-713) or Chad Horohoe (ostriches).
* Short week because of New Years Day on Friday (aka Thursday == "No deploys Friday")
; By day
Deployments/Archive/2016/01

=== Week of January 4th ===
* No normal deploys
* High Priority SWATs only
** "High Priority" means security and data loss. Anything else needs prior approval from Greg (greg-g).
; By day
=== Week of January 11th ===
* Week of:
** nothing yet
; By day
=== Week of January 18th ===
* Week of:
** nothing yet
; By day
=== Week of January 25th ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2016/02

=== Week of February 1st ===
* Week of:
** nothing yet
; By day
=== Week of February 8th ===
* Week of:
** nothing yet
; By day
=== Week of February 15th ===
* Week of:
** nothing yet
; By day
=== Week of February 22nd ===
* Week of:
** nothing yet
; By day
=== Week of February 29th ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2016/03

=== Week of March 7th ===
* Week of:
** nothing yet
; By day
=== Week of March 14th ===
* Week of:
** March 16th: Day long switching default search to completion suggester - 
; By day
=== Week of March 21st ===
* Week of:
** nothing yet
; By day
=== Week of March 28th ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2016/04

=== Week of April 4th ===
* Week of:
** nothing yet
; By day
=== Week of April 11th ===
* Week of:
** nothing yet
; By day
=== Week of April 18th ===
No Deploys
This week we will not being doing any scheduled deploys on the Wikimedia cluster as the WMF Operations team (in coordination with other teams) will be switching the active/master datacenter. See Switch Datacenter for more details.
=== Week of April 25th ===
* Week of:
** nothing yet
; By day
Deployments/Archive/2016/05

=== Week of May 2nd ===
* Week of:
** nothing yet
; By day
{{#invoke:Deployment schedule|formatTable|hidedate=false|
|
|
{{#invoke:Deployment schedule|row
|when=2016-05-02 13:00 SF
|length=1
|window=Services – Parsoid / OCG / Citoid / 
Mobileapps / …
|who=, , , , Bernd () and Michael ({{ircni
ck|mdholloway}})
|what=See mw:Parsoid/Deployments, OCG/Deployments, Mobileapps deployments
}}|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
}}
=== Week of May 9th ===
* Week of:
** Nikerabbit: Running Translate/scripts/refresh-translatable-pages.php on Thursday between 0500-0900 UTC ()
; By day
=== Week of May 16th ===
* Week of:
** nothing yet
; By day
=== Week of May 23th ===
* Week of:
** …
; By day
=== Week of May 30th ===
* Week of:
** ...
; By day
Deployments/Archive/2016/06

=== Week of June 6th ===
* Week of:
** …
; By day
=== Week of June 13th ===
* Week of:
** …
; By day
=== Week of June 20th ===
* Week of:
** …
; By day
=== Week of June 27th ===
* Week of:
** …
; By day
Deployments/Archive/2016/07

=== Week of July 4th ===
* Week of:
** …
; By day
=== Week of July 11th ===
* Week of:
** …
; By day
=== Week of July 18th ===
* Week of:
** …
; By day
=== Week of July 25th ===
* Week of:
** …
; By day
Deployments/Archive/2016/08

=== Week of August 1st ===
* Week of:
** …
; By day
=== Week of August 8th ===
* Week of:
** …
; By day
=== Week of August 15th ===
* Week of:
** …
; By day
=== Week of August 22nd ===
* Week of:
** NOTE: This is the beginning of the new SWAT window schedule including the new mid-day Europe SWAT window (at 13:00 UTC).
; By day
=== Week of August 29th ===
* Week of:
** nothing yet...
; By day
Deployments/Archive/2016/09

=== Week of September 5th ===
* Week of:
** nothing yet...
; By day
=== Week of September 12th ===
* Week of:
** Granting editcontentmodel to mostly autoconfirmed users, probably on Thursday. legoktm (talk) 09:28, 12 September 2016 (UTC)
** nothing yet...
; By day
=== Week of September 19th ===
* Week of:
** : Deploy to all databases. In particular, this week we will be converting templatelinks and pagelinks on s1 (enwiki)
** FINISHED - : Deploy to centralauth (s7). As a central db, it could affect all wikis.
; By day
=== Week of September 26th ===
* No non-emergency deploys. Operations offsite all week.