nyuuzyou/wikis huggingface dataset:
Incidents/2018-01-10 swift

== Summary ==
Swift suffered a brief unavailability period in eqiad during roll restarts for kernel upgrades.
== Timeline ==
* 20180109 - Filippo is upgrading kernel on swift fleet, swift eqiad frontends is roll-restarted without incident, though the kernel package wasn't upgraded on those machines, requiring another roll-restart. During this operation ms-fe1008 is inadvertently not repooled as it should be. 3/4 machines are serving traffic.
* 20180110T16:11 - Roll restarts for ms-fe1* resumes, ms-fe1005 is depooled. 2/4 machines are serving traffic.
* 20180110T16:18 - ms-fe1005 repooled. 3/4 machines are serving traffic.
* 20180110T16:21 - ms-fe1006 depooled. 2/4 machines are serving traffic.
* 20180110T16:22 - ms-fe1007 cannot cope with the load. It is marked as down by PyBal and depooled. 1/4 machines are serving traffic.
* 20180110T16:29 - thumbor.svc.eqiad pages
* 20180110T16:29 - thumbor.svc.eqiad recovers
* 20180110T16:34 - ms-fe1005 also goes down and is depooled by PyBal. 0/4 machines are serving traffic.
* 20180110T16:36 - ms-fe.svc.eqiad pages
* 20180110T16:38 - ms-fe1008 repooled
* 20180110T16:39 - ms-fe1006 repooled
* 20180110T16:39 - ms-fe.svc.eqiad recovers
== Conclusions ==
Swift frontends safety margin is two machines out of four, though this margin was violated due to a combination of factors: namely one less machine in the pool than assumed and too fast traffic swings between the remaining machines in service. Further, PyBal does not take into account administratively depooled servers T184715 when checking whether a host can be depooled or not. The traffic swings ended up overloading ms-fe1007, the only machine fully in service at the time, and subsequently drop of traffic from all frontends in service at the time (ms-fe1005, ms-fe1007).
== Actionables ==
Mostly operator error and failure to verify preconditions before starting procedures on high traffic/critical services.
* Verify preconditions before start of operations (e.g. service pool is healthy)
* Patch PyBal to properly enforce depool-threshold phab:T184715
Category:Incident documentation
Incidents/2018-02-06 Phabricator

== Summary ==
Phabricator stopped working on Feb 6 2018.
Users saw either our standard error page from the caching layer or "Failed to `proc_open()`: proc_open() expects parameter 2 to be array, unknown given"''
The issue was fixed about 6 minutes later after restarting httpd.
== Timeline ==
* 16:51 <+icinga-wm> PROBLEM - https://phabricator.wikimedia.org on phab1001 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 2426 bytes in 2.329 second response time
* 16:56 < elukey> !log restart httpd on phab1001
* 16:57 <+icinga-wm> RECOVERY - https://phabricator.wikimedia.org on phab1001 is OK: HTTP OK: HTTP/1.1 200 OK - 31921 bytes in 0.225 second response time
== Conclusions ==
* There is a deadlock bug in PHP 5.6 which is triggered by Phabricator, leading to a deadlock in malloc as shown by this stack trace.
* The deadlock causes apache to slowly leak worker processes until the pool is filled some other system is exhausted.
* Until we can work around the bug or upgrade to a newer version of php, we will have to periodically restart apache on phabricator servers.
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* Set up a cron job to periodically kill deadlocked apache processes phab:T187790
* Upgrade phabricator to PHP 7.1 as soon as it is practical to do so. phab:T160714
Incidents/2018-02-14 labvirt1008-failure

== Summary ==
Labvirt1008 seems to have overheated and gone down. This effects tenants as well as virtual VPS infrastructure
== Timeline ==
* 7:20 UTC labvirt1008 rebooted
* 8:00 UTC moritzm logged a task about it
* There was no paging or alerting. This is a problem.
* 11:00 UTC Chase woke up and started investigating the extent of the outage, and looking for impact on Toolforge especially
* 12:58 UTC Chase sent an email about impact to cloud-announce https://lists.wikimedia.org/pipermail/cloud-announce/2018-February/000023.html with a list of affected instances.
== Conclusions ==
We know that our instance storage is local and ephemeral. We should ensure that is documented for tenants in easy to find places, and re-ensure that our mechanism that keep critical redundant components spread across labvirts are working. In our world though a single hypervisor is a special snowflake and I believe we should have been paged on this outage, but seem not to have been. It was my understanding that a full instance storage partition should have paged if nothing else, and in this case the failure of that check. 
== Actionables ==
* Coordinate with DC OPS to deal with overheating phab:T187292
* Look at moving tenant instances to another labvirt (we should have a standing spare)
* Investigate what should have paged and why it did not (and fix it)
Incidents/2018-02-20 thumbor

Thumbor was unavailable for a brief period of time following a configuration deployment.
== Summary ==
Thumbor, the image scaling service, became unavailable following a configuration deployment.
== Timeline ==
* 14:29 Filippo merges https://gerrit.wikimedia.org/r/407608 
* 14:44 The first errors of puppet failed on thumbor machines start coming in.
* 14:48 thumbor.svc.eqiad.wmnet pages
* 14:49 thumbor.svc.eqiad.wmnet recovers
* 14:51 Revert of https://gerrit.wikimedia.org/r/407608 is merged on the puppet masters
* 14:56 thumbor.svc.codfw.wmnet pages
* 14:59 thumbor.svc.codfw.wmnet recovers
== Conclusions ==
The puppet module for thumbor is configured to restart the thumbor instances (via thumbor-instances systemd unit, a meta-unit used to do mass actions on all thumbor instances configured on the host) upon config change. puppet issued a 'stop' to 'thumbor-instances' and subsequently a 'start'. Upon receiving a 'stop' all thumbor instances exited with status code 1 and considered in 'failed' state by systemd. The subsequent 'start' to 'thumbor-instances' failed to start the instances properly.
Recovery of thumbor happened on the next puppet run when the "ensure" mechanism of puppet made sure all thumbor instances configured were running as expected.
During the outage there has been also a fair amount of monitoring spam due to individual alerts for each thumbor instance in 'failed' state.
During the incident, about 0.08% of requests for thumbnails failed and end users experienced the 5xx emitted during unavailability for about 10 minutes.
== Actionables ==
* Don't issue uncoordinated restarts of thumbor from puppet: gerrit:412934
* Eliminate per thumbor-instance icinga checks and related spam: gerrit:412948
Category:Incident documentation
Incidents/2018-02-28 puppetmaster

== Summary ==
Puppet master suffered unavailability in eqiad, causing puppet failures
== Timeline ==
* 17:38 Filippo merges change to depool rhodium and repool puppetmaster1002 from puppet master duties
* 17:40 Filippo runs puppet-merge on puppetmaster1001 to make the change above effective in apache configuration
* 17:45 Spam from icinga-wm about puppet failures begins
* 17:47 nitrogen is suspected being the culprit of puppet spam
* 17:53 icinga-wm is stopped due to spam
* 17:57:06 apache2 is restarted on puppetmaster1002
* 17:57:55 puppetmaster1002 is allowed to talk to puppetdb, following a puppet run
* 18:12 A puppet run is forced in eqiad on hosts with failed puppet runs
* 18:14 puppetmaster1002 has its full load back
* 18:29 ircecho / icinga-wm is restarted and joins IRC again
== Conclusions ==
The root cause is a change deemed to be safe that wasn't: namely pool a puppet master backend, and needing only a puppet run on the puppet master frontend for taking full effect. The list of backends also controls access to puppetdb, in other words a puppet run on the puppetdb host is needed to allow access before repooling the backend.
Multiple factors complicated troubleshooting, including alert spam from puppet failures, leading to muting IRC alerts notifications (and potentially masking other unrelated problems). Long standing issues with puppetdb running out of memory was suspected first () which also contributed to slowing down troubleshooting.
== Actionables ==
* Document steps to repool a puppet master backend. https://wikitech.wikimedia.org/wiki/Puppet#Puppetmaster
* Apache httpd mod_proxy logging level should be raised to gain visibility. phab:T188601
* Decrease the amount of spam in case of widespread puppet failures. phab:T188602
Category:Incident documentation
Incidents/2018-02-29 Train-1.31.0-wmf.27

== Summary ==
Train deployment of 1.31.0-wmf.27 rolled back due to a increase in replication wait errors.
== Timeline ==
* 19:24 Synchronized php: group1 wikis to 1.31.0-wmf.26 (duration: 01m 17s)
* 19:22 rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.26
* 19:20 Rolling back to wmf.26 due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query"
* 19:19 rolling back to wmf.26
* 19:18 icinga-wm: PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [50.0] 
* 19:17 twentyafterfour: | I'm seeing quite a few "[{exception_id}] {exception_url} Wikimedia\Rdbms\DBExpectedError: Replication wait failed: Lost connection to MySQL server during query
* 19:06 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.27 (duration: 01m 17s)
* 19:05 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.27
Error graph from the same time period:
https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen&from=1522263260081&to=1522265839537
After rollback Mukunda filed to document the incident. The culprit was determined to be this commit which was made by Aaron Schulz. That commit was intended to address the issues described in .
== Conclusions ==
What weakness did we learn about and how can we address them?
It is exceedingly difficult to thoroughly test some changes outside of production. Testing replication lag detection properly would require a simulation of production databases plus realistic traffic to stress them. Our current deployment process prevented this from having an impact on production databases or site reliability, however, we spent a lot of time deploying and then reverting changes and we blocked testing of other changes in the pipeline. This points to weaknesses in the weekly branching strategy that we currently employ as well as weaknesses in our testing environments. A change such as this one should really have it's own staged deployment process rather than "riding the train" concurrent with a bunch of unrelated changes.
== Actionables ==
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
* phab:T193258
Incidents/2018-03-02 Train

== Summary ==
Train for 1.31.0-wmf.23 was rolled back on two occasions:
* 2018-02-28 05:43:37 due to deletion logs on MediaWiki recording incorrect users doing deletions 
* 2018-02-28 22:11:xx due to a noisy notices/all pages listed in Special:Newpages showed the current date and time 
== Timeline ==
=== T188479 ===
* 05:21 Stemoc reported a problem with the deletion logs (Special:Log/delete) showing the wrong user in #mediawiki
* legoktm investigates and creates a task: 05:29, 28 February 2018 Dharmadeepa V (talk | contribs | block) deleted page User:Dharmadeepa V (spam (this is legoktm)) (view/restore)
* 2018-02-28 05:39:05 <legoktm> I'd suggest reverting the train, like immediately
* 05:43:37 +logmsgbot | !log demon@tin rebuilt and synchronized wikiversions files: (no justification provided)
=== T188555 ===
* 1.31.0-wmf.23 was rolled out to group1 wikis: 2018-02-28 21:56 <thcipriani@tin> rebuilt and synchronized wikiversions files: Group1 to 1.31.0-wmf.23
* Wed, Feb 28, 21:59 thcipriani noticed an increased error rate, notices pointing to stdclass::$rc_timestamp and created 
* thcipriani rolled group back to 1.31.0-wmf.22 2018-02-28 22:11 <thcipriani@tin> rebuilt and synchronized wikiversions files: Group1 back to 1.31.0-wmf.22 T188555
* Overnight the problem was resolved and a patch merged in master, that change was deployed after train the following day: [2018-03-01T20:15:45Z] <thcipriani@tin> Synchronized php-1.31.0-wmf.23/includes/specials/pagers/NewPagesPager.php: SWAT: NewPagesPages: Use array_merge rather than + for RC query info fields T188555 (duration: 01m 14s)
== Conclusions ==
* A test case probably should have caught the first problem that led to an emergency rollback.
* The second problem seems like something automated browser tests or manual testing could have caught.
== Actionables ==
* phab:T188773 - Test to validate deletion log entries and ArticleDeleteComplete hook performers
Incidents/2018-03-06 MasterMergedInWMFBranch

== Summary ==
During deployment of a patch for mediawiki/core@wmf/1.31.0-wmf.23, the deployer found out the difference between HEAD the origin/wmf/1.31.0-wmf.23 had too many commits. That prompted for an immediate halt.
The root cause has been determined to be the commit from master being sent for review to the wmf branch. In such a case, Gerrit will implicitly merge master into wmf branch. The Gerrit UI was not showing that and when the patch got merged the master branch made it to the wmf branch.
== Timeline ==
* Mar 05 - CR for mediawiki-core submitted and merged into the master branch
* Mar 06 13:18 UTC - The change cherry-picked from the master branch to wmf/1.31.0-wmf.23 branch with CR and merged. Gerrit implicitly merged all the ancestors of the commit from master branch to wmf/1.31.0-wm.23 branch
* Mar 06 13:18 UTC Attempt to deploy revealed that the diff is too big and deploy was halted.
* Mar 06 14:46 UTC A revert of the erroneous commit created and merged, the state of gerrit returns back to normal.
== Conclusions ==
* Deployer followed the documentation (git diff HEAD..HEAD@{u}) and properly halted on the first oddity
* Gerrit is being too helpful
The reason behind is:
Given a git tree such as:
A -- B -- C (master)
 \
 \-- X (wmf)
One is locally on the master branch and does a git push HEAD:refs/publish/wmf. Gerrit will create a new change Y that implicitly merge master into the wmf branch:
A -- B -- C (master)
 \ \
 \-- X (wmf)--\-- Z (new change)
The Gerrit GUI fails to reflect that the new change (Z) is a merge commit. One can tell though since the parent commit will be B from the master branch.
When Z is submitted the master branch ends up having been merged in wmf since B and C got already merged in master.
Gerrit 2.13 offers the option receive.rejectImplicitMerges to prevent that implict merge. From | Gerrit 2.13 release notes :
: The ‘Reject Implicit Merges’ option can be enabled to prevent non-merge commits from implicitly bringing unwanted changes into a branch. This can happen for example when a commit is made based on one branch but is mistakenly pushed to another, for example based on refs/heads/master but pushed to refs/for/stable.
== Actionables ==
* In Gerrit disable receive.rejectImplicitMerges https://gerrit.wikimedia.org/r/Documentation/config-project-config.html#receive.rejectImplicitMerges phab:T189024
** Done via gerrit:416704
Category:Incident documentation
Incidents/2018-03-14 ORES

== Summary ==
We had a small and planned user facing outage during maintenance
== Timeline ==
* 14:18 UTC Aaron Halfaker and Alex discuss the need to do a scheduled reboot of the oresrdb hosts for kernel upgrades. Decision is taken to proceed with it ASAP
* 14:25 UTC Alex starts with the slaves in codfw and eqiad. No impact
* 14:34 UTC Both slaves had caught up to the masters
* 14:34 UTC Alex starts reboots for the master redis on codfw. Errors start
* 14:35 UTC Reboot is done but redis is loading the dataset in memory
* 14:37 UTC oresrdb2001 has the dataset in memory, jobs can be submitted once more.
* 14:45 UTC After having gauged the effects in codfw, alex starts the same process in eqiad
* 14:47 UTC Icinga notices
* 14:50 UTC Up and running again. Icinga notices
== Conclusions ==
We were unable to serve ~2500 requests total in eqiad (~250 external ones) and ~6000 requests in codfw (~240 external ones). The reason behind this is that the redis hosts that acts both as a queue and as a cache is now a SPOF and this should be addressed.
== Actionables ==
* Use twemproxy for the cache redis at least in order to be able to serve at least a portion of requests during a downtime (Phab: TODO)
* Try to figure out way that the queue redis can be made highly available (Phab: TODO)
Category:Incident documentation
Incidents/2018-04-04 mathoid

== Summary ==
The Mathoid service in eqiad experienced an outage due to increased load. An operator mistake in provisioning more capacity extended the total outage time by 10 mins.
 
== Timeline ==
* 22:26 UTC The SRE team gets paged for a mathoid outage.
* 22:27 UTC Pages about recovery arrive
* 22:36 UTC Paged again about a failure. 
* 22:37 UTC Filippo calls Alex, who happens to be awake.
* 22:40 UTC Alex logs in, realizes that extra traffic is being directed to the mathoid service (reason unknown, probably some pages/template edited), and decides to increase the capacity allocated to the cluster
* 22:54 UTC alex issues the command to increase the number of pods from 4 to 16.
* 22:55 UTC mathoid pages again. 
* ~23:00 UTC Giuseppe advises that the service IP in the kubernetes Service resource is wrong
* 23:01 UTC alex fixes the above mistake, still no luck
* 23:05 UTC Alex realizes the there is no network policy for the service applied so all traffic is denied
* 23:08 UTC Alex removes the helm release and installs a new one
* 23:09 UTC Mathoid recovers
== Conclusions ==
An operator mistake when increasing the number of pods from 4 to 16 in the invocation caused a wrong service IP in the kubernetes service configuration. We need better tooling to avoid such things in the future. The helm release did not have a network policy applied after a pod increase. The exact cause is yet unknown, more testing is needed. 
== Actionables ==
* Create documentation for the most common operations of the Kubernetes cluster and their services, like modifying the cluster size.
== Notes ==
Usually, most requests for Math formulae are handled by RESTBase (as most are asks for renders); Mathoid itself handles only responding to checks for erroneous formulae and renders for new formulae. Because of that, traffic to Mathoid itself is rather limited (compared to actual external traffic). However, it is not uncommon for editors to import large numbers of formulae at once (e.g. by copy/pasting math code from one wiki page to another and then modifying for language/style), causing surges in traffic to the service itself. During the move of the service from SCB to Kubernetes, however, the number of workers able to handle requests was dropped from 96 to 4, which can easily cause the service to stop accepting requests as rendering MathML, SVG and PNG versions of a formula takes a while.
Category:Incident documentation
Incidents/2018-04-10 Deleting a page on enwiki

== Summary ==
An alter table running on the archive table on the English Wikipedia master database server caused deletion of pages to fail for en.wikipedia.org for ~40 minutes.
It was originally reported at: 
According to logtash 85 errors happened during that time
== Timeline (IN UTC) ==
*5:20 an ALTER table on enwiki master (db1052) on the externallinks table was started
*8:20 an ALTER table on enwiki master (db1052) on the archive table was started
*8:30 Some errors pop in the error log: https://logstash.wikimedia.org/goto/becc429ddb975af71624b66402c3f6bb
Example of a failure:
Read timeout is reached (10.64.16.77)	INSERT INTO archive (ar_namespace,ar_title,ar_timestamp,ar_minor_edit,ar_rev_id,ar_parent_id,ar_text_id,ar_text,ar_flags,ar_len,ar_page_id,ar_deleted,ar_sha1,ar_comment,ar_comment_id,ar_user,ar_user_text,ar_content_model,ar_content_format) VALUES ('14','Articles_needing_sections_from_August_2015','xx','0','xx','0','xx',,,'xx','xx','0','xx','Creating monthly dated maintenance category for current month','xx','xx','xx',NULL,NULL)
*9:19: Alter table finished and errors are gone
*9:22 it is confirmed that everything works again at: 
A total of 85 errors happened between 08:20 and 09:19
== Conclusions ==
The following ALTER table caused issues on db1052 (enwiki master) when deleting (and probably when moving) a page but not on the other 6 masters it was executed before:
SET SESSION innodb_lock_wait_timeout=1; SET SESSION lock_wait_timeout=30; ALTER TABLE archive MODIFY COLUMN ar_text mediumblob NULL, MODIFY COLUMN ar_flags tinyblob NULL;
There were a total of 85 errors during the time of the incident. 
It was also noticed that the master had ongoing queries () for the archive table, that might have also contributed to this issue. Whether it is the cause or the consequence is still unknown. 
As described at: this ALTER is fully online (and has not caused issues anywhere else)
== Actionables ==
We are still investigating why this has been caused, but so far everything looks like a race condition.
The following task has also been created to try to check if the new queries may be creating additional contention (although it is not yet clear this is the root/only cause of the incident)
* Reduce locking contention on deletion of pages (phab:T191892)
Category:Incident documentation
Incidents/2018-04-10 Routing

== Summary ==
A configuration change on routers located in the Ashburn and Singapore datacenters caused a service interruption of ~10min (22:53-23:03UTC) for users redirected to Ashburn, and ~40min for users redirected to Singapore. (22:47-23:24 UTC)
More details on: 
== Timeline ==
* 22:47 Change pushed to cr1-eqsin
* 22:53 Change pushed to cr2-eqiad
* 22:58 cr2-eqiad rolled-back
* 23:03 eqiad full recovery (after routing convergence)
* 23:22 cr1-eqsin rolled-back (partial recovery)
* 23:31 eqsin de-pooled
* 23:36 eqsin full recovery
== Conclusions ==
* Changes, even if already live in part of the infrastructure, need to be better discussed with the team
* POPs (especially non redundant ones) should be depooled before applying changes, if any doubt
* The same change had different results across the deployment:
** No issues, working as expected (eg. switches, cr2-esams)
** Partial failure (cr1-eqsin), connectivity to the router and rpd appeared in a healthy state, user traffic was being dropped
** Full failure (cr2-eqiad), instantly lost connectivity to the router
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* Tickets have been opened with the vendor phab:T191667 (update: crash reason found)
Incidents/2018-04-23 wdqs

== Summary ==
Wikidata Query Service experienced slow down and timeouts for ~30 minutes
== Timeline ==
* 7:30 UTC: wdqs.svc.eqiad.wmnet start experiencing slowdowns (95 %-ile is reaching the 1 minute timeout)
* 7:39 UTC: icinga alert: LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds
* 7:41 UTC: icinga recovery: LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 434 bytes in 0.029 second response time
* 8:08 UTC: restart of wdqs1003, situation back to normal
== Conclusions ==
During the slowdown, we see multiple clients being aggressively throttled. This seems to be a consequence of the slowdown, not a cause. While we do have clients who don't backoff when receiving HTTP 429, they seem to be blocked correctly.
GC logs show mostly no GC activity after 7:28 UTC, which indicates that for once, the issue is somewhere else.
== Actionables ==
* investigate the cause of the slowdown of wdqs1003 phab:T192759
* create a dedicated WDQS cluster for internal traffic phab:T178492 (mostly done, still need to move clients to use this new cluster)