nyuuzyou/wikis huggingface dataset:
Incidents/2019-08-02 api-overload-parsoid

document status: 
== Summary ==
The api-appserver cluster has been overwhelmed with expensive parsoid-batch calls from particular pages, resulting in brownouts and frontend availability at ~97%.
=== Impact ===
The mediawiki API has suffered the most impact, with internal services and external API users impacted as well. See API frontend graph, the slowdown was severe enough that a depression in request rates was observed.
=== Detection ===
Alerts for appservers "high CPU" went off on IRC during brownouts through the first half of the day (6-12 UTC) followed by general widespread alerts for affected internal systems (restbase, recommendation, mobileapps alerts went off) starting around 15 UTC
== Timeline ==
All times in UTC.
* 7:00 / 9:30 / 12:30 / 13.00 High CPU load API appservers alerts and recoveries for brownouts, no widespread impact
* 13:25 investigation on the general issue begins
* 13:45 weights of mw12[23].* found not to be aligned with the rest of the fleet and fixed
* 14:50 High CPU load on API appservers start coming in again, this time more severe and affecting other services
* 14:54 Low HTTP availability alerts fire, together with recommendation_api, restbase, mobileapps
* 14:55 Investigation on the ongoing incident begins
* 15:02 Recoveries start coming in by themselves, and investigation continues
* 15:34 Offending page identified as one on itwiki using Lua, and requested by Google's rest api crawler
== Conclusions ==
API appservers are susceptible to storms of many long-running parsoid-batch requests, in particular external requests which end up exhausting the cluster's resources and affecting all API users.
=== What went well? ===
An outage condition was identified quickly, and investigation began quickly.
=== What went poorly? ===
* Identifying the expensive API calls could have been faster.
* Unbounded parallelism of expensive API calls must be limited, both in rate and concurrency. Internal update requests like this from Change-Prop are both rate-limited and blacklisted automatically if the rate of errors is too high, but external requests are not.
* I (Ppchelko) assume that Varnish would attempt to retry a request on timeout, which in turn multiplies the load.
=== Where did we get lucky? ===
The cluster recovered by itself once requests ended and recovery has been fast.
== Links to relevant documentation ==
Where is the documentation that someone responding to this alert should have (runbook, plus supporting docs). If that documentation does not exist, there should be an action item to create it.
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
* Limit external requests concurrency/parallelism ()
* Icinga alerts for appserver high CPU should be less spammy ()
* Bigger idea: it would be nice if there was, aggregated across the whole cluster, some report available of what request types, users, user-agents, etc were using the most appserver CPU.
* …
Incidents/2019-08-20 logstash

document status: 
== Summary ==
For about 30 minutes, Logstash was not getting any messages from the MediaWiki servers. 
=== Impact ===
During the Logstash outage, we were partly blind in terms of operational monitoring. It also meant developers were unable to use WikimediaDebug, and unable to deploy new code for MediaWiki and most other services.
While this impacted scheduling and developer productivity, it did not directly affect end-users of any public services. Also, the logs were eventually recovered into Logstash after it was restarted (the Logstash-Kafka consumer picks up where it left off).
=== Detection ===
* Icinga alerts.
== Timeline ==
All times in UTC.
* 23:24 <+icinga-wm> PROBLEM - Too many messages in kafka logging-eqiad on icinga1001 is CRITICAL: cluster=misc exported_cluster=logging-eqiad group={logstash,logstash-codfw} instance=kafkamon1001:9501 job=burrow partition={0,1,2} site=eqiad topic=udp_localhost-info https://wikitech.wikimedia.org/wiki/Logstash%23Kafka_consumer_lag https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag
* 23:40 <fgiunchedi> Logstash stabilized and returned to normal
== Conclusions ==
* Logstash consumer failed, which does not recover by itself. Details at https://phabricator.wikimedia.org/T230847#5427615
=== What went well? ===
* Detected early. Quickly fixed by restarting.
=== How many people were involved in the remediation? ===
* 1 SRE.
== Actionables ==
* None. See https://phabricator.wikimedia.org/T230847
Category:Incident documentation
Incidents/2019-08-23 network codfw

document status: 
==Summary==
A provider outage on our primary transport link between eqiad and codfw caused it to be in a constant flapping (going down and up) state.
This flapping caused routing re-convergence churn and packet loss between the two sites.
On the application level, this translated to elevated 5xx/s from Varnish from ulsfo, eqsin, and codfw from 21:20 to 21:55 UTC. Varnish reported "No backend" for many of the requests. Host checks in Icinga were flapping "TTL exceeded" and service checks flapping "No route to host."
===Impact===
Surfaced a bit more than 52,000 5xx responses.
https://grafana.wikimedia.org/d/000000479/frontend-traffic?panelId=2&fullscreen&from=1566594998796&to=1566597283616&var-status_type=5
===Detection===
Monitoring caught and reported the issue via SmokePing and Icinga.
== Timeline==
All times in UTC.
*2019-08-23 21:20 OUTAGE BEGINS
*21:25 Investigation begins
*21:33 Zayo (the link's provider) reports issue with service (email unnoticed)
*Lots of errors and recoveries - flapping
*21:41 Arzhel starts investigating
*21:46 Brandon called
*21:47 Decided to depool codfw (ended up not needing it)
*21:48 Arzhel promotes backup link to primary
*21:55 OUTAGE ENDS
*2019-08-25 01:37 Link stops flapping
==Conclusions==
=== What went well?===
*The root cause was quickly worked-around once the cause (network link) was identified.
===What went poorly?===
*Due to the frequency of the flapping Icinga checks for link status, OSPF and BFD didn't trigger, causing SREs to think of an application layer issue
*The work-around (failing over to the backup link) is not documented and requires Netops to be done.
*Nothing paged even though it had user facing impact
===Where did we get lucky?===
*Giuseppe, and Filippo responded outside of their office hours.
===How many people were involved in the remediation? ===
*No Incident coordinator appointed - 5 SREs
==Links to relevant documentation==
*Global traffic routing#Disabling a Site 2
*Network monitoring#OSPF status
==Actionables==
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
*Those two will help mitigate the consequences of an overly flapping link:
**Configure interface damping on primary links - T196432
**ospf link-protection - T167306
*This one will make it easier (down the road) to a non-netops to failover a link if the need arises:
**Configuration management for network operations - T228388
*This one is about having better monitoring and alerting by replacing Smokeping by something Prometheus based
**Investigate/setup prometheus blackbox_exporter - T169860
*
Incidents/2019-10-16 network eqsin

document status: 
==Summary==
An Equinix Singapore IXP peer flapped heavily, which overwhelmed the routing daemon on cr1-eqsin and caused all its BGP and OSPF sessions to flap or go down.
In addition to the external connectivity issues, as the primary transport link to codfw is on cr1, it caused the local caches to not be able to reach their peers in the main datacenters and serve 500 errors instead.
===Impact===
Estimated ~170k errors surfaced to users of eqsin PoP.
https://grafana.wikimedia.org/d/000000479/frontend-traffic?orgId=1&from=1571245200000&to=1571248800000&var-site=eqsin&var-cache_type=text&var-cache_type=upload&var-status_type=5
===Detection===
The following automated alerts got triggered:
*Varnish traffic drop between 30min ago and now at eqsin
*HTTP availability for Nginx -SSL terminators- at eqsin
*HTTP availability for Varnish at eqsin
*BFD status on cr1-codfw
*LVS HTTPS text-lb.eqsin.wikimedia.org - PAGE
This quickly pointed to an network issue in eqsin.
Was the alert volume manageable? yes
Did they point to the problem with as much accuracy as possible? yes
==Timeline==
All times in UTC.
*17:15 SSL terminator alerts in eqsin fire, non-paging -- OUTAGE BEGINS
*17:28 First page fires -- LVS HTTPS text-lb.eqsin.wikimedia.org
*17:29 eqsin depooled
*17:29 Recovery on its own -- OUTAGE ENDS
==Conclusions==
===What went well?===
*The issue was quickly identified
*The issue recovered on its own
===What went poorly?===
*A router's routing daemon should not behave that way but that router's model is known to be weak
*The logs didn't have any information on why OSPF and BGP were behaving that way.
===Where did we get lucky?===
*Several SREs were around when the issue started
===How many people were involved in the remediation?===
*6 SREs
==Links to relevant documentation==
Depooling the site: DNS#Change GeoDNS
==Actionables==
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
*T236878 Improve resiliency of the eqsin transport link by either:
**Terminating it on cr2-eqsin
**Adding a 2nd link
**Configuring link damping
*Replace cr1-eqsin with a better router (next FY)
Incidents/2019-10-31 wikidata

document status: 
== Summary ==
Wikidata API calls were not getting responses (getting timeouts) due to DB read load due to backported changes reducing deadlocks around writing to the new terms store for wikibase.
=== Impact ===
Wikidata editors received timeouts to API requests, API response time for writes went through the roof. It seems like most edits from API calls were actually made, but the clients didn't get a response confirming that.
=== Detection ===
Humans in Telegram chat.
Confirmed by Addshore.
== Timeline ==
All times in UTC.
* 16:00 - Maintenance script migrating wb_terms data restarted
* 16:02	<ladsgroup@deploy1001>	Synchronized php-1.35.0-wmf.3/extensions/Wikibase: Wikibase deadlock reduction, Stop locking and use DISTINCT when finding used terms to delete (T236466) (duration: 01m 05s)
* 16:05	<ladsgroup@deploy1001>	Synchronized php-1.35.0-wmf.4/extensions/Wikibase: Wikibase deadlock reduction, Stop locking and use DISTINCT when finding used terms to delete (T234948) (duration: 01m 04s)
* 16:12 - Read rows on 1 db slave shot up https://phabricator.wikimedia.org/T236928#5620434
* ~16:18 - Edit rate on wikidata really started dropping
* 16:30 - Maintenance script migrating wb_terms data restarted (picking up code changes)?
* 16:38 - Reported timeout in UI editing on wikidata.org in Telegram chat
* 16:54 - Phabricator task created after Addshore spotted this message - https://phabricator.wikimedia.org/T236928
* 16:59	<jynus>	killed rebuildItemTerms on mwmaint1002
* ~17:00 Edit rate on wikidata recovering, but drops again - https://phabricator.wikimedia.org/T236928#5620364
* 17:26	<ladsgroup@deploy1001>	Synchronized php-1.35.0-wmf.3/extensions/Wikibase: Revert 16:02 UTC T236928 (duration: 01m 04s)
* ~17:26 Edit rate on wikidata recovering again
* 17:29	<ladsgroup@deploy1001>	Synchronized php-1.35.0-wmf.4/extensions/Wikibase: Revert 16:05 UTC T236928 (duration: 01m 05s)
== Conclusions ==
* We could do with more alarms on things that often indicate a problem
=== What went well? ===
* Was not a total outage, just severe slowness
=== What went poorly? ===
* No alarms went off
* Only a message in Telegram alerted us to an issue (not even a phab task)
=== Where did we get lucky? ===
* People were on hand that knew what the problem was (as the issue did not coincide with deployment time)
=== How many people were involved in the remediation? ===
* 2 SREs?
* 2 Wikidata Devs
== Actionables ==
* More wikidata alerting https://gerrit.wikimedia.org/r/#/c/547404/ https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts
* Add alerting to API response times for wikidata
* Add alerting for wikidata edit rate (if below 100 per minute something somewhere is wrong)
* Add alerting for MASSIVE database read rate on s8
Incidents/2019-12-04 MediaWiki

== Summary ==
Roll out of wmf.8 to group1 broke the world.
=== Detection ===
Initial indicators of the issue were picked up in logstash and via logspam-watch on mwlog1001. A large number of Icinga alerts followed.
It seems likely that the primary issue was obscured during the initial deploy by a focus on Parsoid errors.
== Timeline ==
All times in UTC.
* 20:12 brennen: Train wmf.8 roll fowards from group0 to group1 as well (try 1) 
* 20:12 Large amounts of logspam noticed, especially from Parsoid/PHP, and Icinga issues many alerts.
* 20:28 brennen: Train wmf.8 rolled back to just group0 
[Fixes to exclude Parsoid/PHP]
* 23:30 brennen: Train wmf.8 roll fowards from group0 to group1 as well (try 2) 
* 23:30 OUTAGE BEGINS
* 23:30 Large spike in database errors in logstash (T239877), shortly thereafter large amounts of Icinga alerts go off.
* 23:30+ Production group1 and group2 wikis become noticably sluggish, eventually stopping working entirely.
* 23:35 brennen: Attempted train wmf.8 roll back thwarted by canary failures 
* 23:38 brennen: Train wmf.8 rolled back to just group0, again 
* 23:38 OUTAGE ENDS
== Links to relevant documentation ==
* Heterogeneous_deployment/Train_deploys#Rollback
Incidents/20190308-maps

== Summary ==
A monkey patch phab:T214350 deployed during the jessie -> stretch migration phab:T198622 was erased by mistake. This broke the geoshape service for some use cases. Which in turn broke a number of map frames.
The failure happened with Geoshapes v1.0.3 which didn't have the fix published, that caused the Pull Request that changed it to be ineffective.
For context: the migration from jessie to stretch requires a re-generation of tiles, which is taking a long time (weeks). During that time, the cluster is in a mixed jessie / stretch configuration. Some code changes were needed for the stretch upgrade, so different application versions are deployed on different servers.
== Timeline ==
* 2019-01-22 14:34Z: 'monkey patch' deployed to production
* 2019-03-07 18:32Z: deployment of kartotherian 1.0.0 on maps100[1-4] + maps2004 (servers already migrated to stretch), this erased the previous monkey patch
* 2019-03-07 18:32Z: repool of maps2004
* 2019-03-07 14:14Z: phab:T217898 created for an issue on geoshape service
* 2019-03-07 15:05Z first report of wiki pages with blank maps on IRC, in staff channel
* 2019-03-07 15:25Z: issue investigated and reproduced on a number of wiki pages with mapframe + geoshape
* 2019-03-07 15:45Z: issue identified as affecting the nodes upgraded to stretch, but not the ones on jessie
* 2019-03-07 15:47Z: issue identified as an existing undeployed patch
* 2019-03-07 16:22Z: patch packaged and properly deployed on all stretch nodes except maps1004 (typo in deployment config)
* 2019-03-07 16:43Z: missing deployment on maps1004 identified
* 2019-03-07 16:47Z: patch deployed on maps1004, situation back to normal
== Conclusions ==
* our monitoring did not catch this issue, no test is performed on the geoshape service as part of the usual spec.yaml phab:T217910
During investigation, a number of issues were raised, which are not all directly related to this specific incident. Still, it make sense to track and address those:
* kartotherian acts as a SPARQL proxy to wikidata query service, this should be constrained in some way and not expose the full power of SPARQL
*there was little confidence that codfw alone could cope with production load while in a "degraded" state (with maps2004 down) highlighting potential need to grow cluster / to have a dashboard showing service capacity
== Links to relevant documentation ==
Some (not enough) documentation on maps Maps. There is no specific documentation for this particular problem, and since it is unlikely that the same situation happen again (the same undeployed patch), none will be created.
== Actionables ==
* Improve monitoring of geoshapes phab:T217910
*Review maps server capacity
Category:Maps outages, 2019
Incidents/20190321-acmechief

== Summary ==
After upgrading to acme-chief 0.14 and restarting uwsgi-acme-chief service in acmechief1001, acme-chief-api wrongly signaled to puppet the /etc/acmecerts files as directories.. effectively wiping the TLS certs used by the services whose certificates are managed by acme-chief.
The issue in acme-chief API is solved by https://gerrit.wikimedia.org/r/c/operations/software/acme-chief/+/498046
== Timeline ==
All times are in UTC
* 08:54 uwsgi-acme-chief is restarted in acmechief1001 making effective the acme-chief upgrade to 0.14.
* 08:58 slapd crashes in seaborgium after puppet runs and destroys the TLS files in /etc/acmecerts
* 09:09 toolschecker pages CRITICAL for Test LDAP on checker.tools.wmflabs.org
* 09:13 acme-chief downgraded to 0.12 and uwsgi-acme-chief restarted
== Affected servers ==
The following servers have been affected by this issue:
* sodium.wikimedia.org
* seaborgium.wikimedia.org
* cobalt.wikimedia.org
* gerrit2001.wikimedia.org
* netmon2001.wikimedia.org
* netmon1002.wikimedia.org
* mx1001.wikimedia.org
* mx2001.wikimedia.org
* ldap-eqiad-replica01.wikimedia.org
* ldap-eqiad-replica02.wikimedia.org
* fermium.wikimedia.org
* dbmonitor1001.wikimedia.org
* dbmonitor2001.wikimedia.org
Incidents/20190603-maps

==Summary==
Work was (and still is) in progress to address out of disk space issues on maps codfw servers (). This means that maps2001 and maps2004 were depooled at that time. A spike is requests caused overload of those servers and request timeouts. The suspicion was that we had other issues related to this disk out of space, so maps2002 and maps2003 were depooled. For unknown reason, eqiad was depooled in dnsdisc, causing all maps traffic to be directed to codfw, the depool of maps200[23] meant that no server was available to serve traffic. Switching all traffic to eqiad fixed the issue.
===Impact===
Service unavailability for users as tiles request timed out (for how long?)
===Detection===
* Icinga alerts for kartotherian endpoints and varnish 5xx alerts
* checking pybal to realize only maps codfw was pooled at some point
==Timeline==
All times in UTC.
*9:46: First alert seen on irc (#-operations). It showed tile request timed out for kartotherian. PyBal backends health check on lvs2006 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps2003.codfw.wmnet are marked down but pooled
*9:48: Maps200[23] were depooled. This was done in believe that maps eqiad was pooled but it was not.
*9:49: More alerts were seen on icinga and this time, varnish started reporting errors too.
*9:59: We detected only maps codfw was pooled
*10:02: maps eqiad was pooled
*10:03: Maps codfw was depooled.
*10:05: 503s in varnish started recovering
==Conclusions==
What weaknesses did we learn about and how can we address them?
The following sub-sections should have a couple brief bullet points each.
===What went well?===
* It was easy to point maps codfw problems to postgresql lag for slaves.
===What went poorly?===
* We should have pooled maps eqiad.
* Depooling maps200[23] meant that no server was available to serve traffic. The intent was to depool kartotherian / codfw, which should have been done via dnsdisc.
===Where did we get lucky?===
*
==Links to relevant documentation==
Where is the documentation that someone responding to this alert should have (runbook, plus supporting docs). If that documentation does not exist, there should be an action item to create it.
* how to depool kartotherian codfw (or more generically, how to depool a service for a datacenter using confctl) [TBD]
==Actionables==
* Maybe a check to confirm maps eqiad and codfw are always pooled?
* fix maps codfw disk space issues fast as we only have maps eqiad for now
* refresher on how dnsdisc works
Category:Maps outages, 2019
Incidents/2020-01-27 app server latency

== Summary ==
External demand for an expensive MW API query, caused the MW API web servers to become overall slower to respond to other queries as well. Some of these queries became sufficiently slow as to trigger our execution timeout of 60 seconds. The Recommendation API service was partially unavailable for about 35-40 minutes as it uses the MW API for part of its work.
== Timeline ==
* (Grafana) RESTBase backend errors: <https://grafana.wikimedia.org/d/000000068/restbase?orgId=1&from=1580159395388&to=1580164542571>
* (Logstash) MediaWIki errors & timeouts: <https://logstash.wikimedia.org/goto/613fac39acd1638eca41fd1649afe1da>
Internal Google Doc: <https://docs.google.com/document/d/1H1HOA49S0pATzXlD1yQLQiSR12qsFkcagtLWg5Slz34/edit#>
Incidents/2020-05-28 commons read-only

== Summary ==
s4 primary database master (db1138) had a hardware memory issue, mysqld process crashed and came back as read-only for 8 minutes.
Impact: commonswiki didn't accept writes for 8 minutes. Reads remained unaffected
== Timeline ==
All times in UTC.
* 01:33 First signs of issues on the DIMM are logged on the hosts's IDRAC
* 20:21 mysql process crashes OUTAGE BEGINS
* 20:24 First page arrives: PROBLEM - MariaDB read only s4 #page on db1138 is CRITICAL: CRIT: read_only: True, expected False: 
* 20:24 A bunch of SREs and a DBA start investigating and the problem is quickly found as a memory DIMM failure and mysql process crashed
* 20:28 <@marostegui> !log Decrease innodb poolsize on s4 master and restart mysql
* 20:29 MySQL comes back and read_only is manually set to OFF
* 20:29 OUTAGE ENDS
== Conclusions ==
This was a hard to avoid crash - hardware crash on a memory DIMM.
Masters start as read-only by default (to avoid letting more writes go through after a crash, until we are fully sure data and host are ok and still able to take the master role).
We did see traces of issues on the idrac's error log, if we could alert on those, maybe we could have performed a master failover before this host crashes. If this crash happens on a slave, the impact wouldn't have been as big, as slaves are read-only by default and MW would have depooled the host automatically.
=== What went well? ===
* The read_only alert that pages when a master comes back as read-only worked well (this was an actionable from a similar previous issue)
* Lots of people made themselves available to help troubleshooting, from both EU (late UTC evening already) and US (from different TZs)
=== What went poorly? ===
* We had signs of this memory failing on the idrac's hw log ardound 19 hours before the actual crash, we could have alerted on those beforehand, scheduled an emergency failover and prevent this incident.
=== Where did we get lucky? ===
* The host and the data didn't come back corrupted, otherwise we'd have needed to do a master failover at that same moment
=== How many people were involved in the remediation? ===
* 1 DBA even though lots of SRE made themselves available in case they were needed.
== Links to relevant documentation ==
* There is not specific documentation on what to do when a master crashes and comes back as read only. Action point: https://phabricator.wikimedia.org/T253832
== Actionables ==
* [DONE] Documentation on how to proceed if a master pages for read-only = ON: https://phabricator.wikimedia.org/T253832
* [DONE] Failover db1138 to its candidate master (scheduled for Friday 29th at 05:00 AM UTC): https://phabricator.wikimedia.org/T253808
* [DONE] Replace failed DIMM on that host: https://phabricator.wikimedia.org/T253808
* Alert on ECC warnings in SEL https://phabricator.wikimedia.org/T253810
* Create a script to move replicas between hosts when the master isn't available https://phabricator.wikimedia.org/T196366