nyuuzyou/wikis huggingface dataset:
Incidents/2022-03-10 MediaWiki availability

==MediaWiki availability on all wikis for logged-in users / uncached content==
The root cause seems to have been db1099, a replica database in the s8 section (Wikidata), that was rebooted for maintenance shortly before the incident and it was slowly repooled into production while at the same time there was a file transfer over the network from the same host. The load caused by the repooling, although at a very small percentage of the traffic, in addition to the existing bandwidth used by the file transfer, the host became slow to respond to queries, but not enough to be considered down and depooled automatically by other systems. This caused a cascade effect on DBs on the same section (s8) which got overloaded, and, because practically every page render involve reads from s8, it had a secondary cascade effect on all wikis, causing the exhaustion of workers at the application layer. The user-facing side of the outage was seen as slow or unavailable access to uncached render pages or perform read-write actions.
Documentation:
*Dashboard with the query throughput for all databases showing the dip of query processed on all database sections.
*RED dashboard for the application servers (MediaWiki) showing that the problem was related to the S8 database section.
*API Gateway SLO dashboard
==Actionables==
*Incident tracking task
*Investigate if stopping mysql with buffer_pool dump between 10.4 versions is safe
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Score	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? (score 1 for yes, 0 for no) 	 	
People 	Were the people who responded prepared enough to respond effectively (score 1 for yes, 0 for no) 	1 	
People 	Were more than 5 people paged? (score 0 for yes, 1 for no) 	0 	
People 	Were pages routed to the correct sub-team(s)? (score 1 for yes, 0 for no) 	 	
People 	Were pages routed to online (business hours) engineers? (score 1 for yes,  0 if people were paged after business hours) 	1 	
Process 	Was the incident status section actively updated during the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Was the public status page updated? (score 1 for yes, 0 for no) 	0 	
Process 	Is there a phabricator task for the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Are the documented action items assigned?  (score 1 for yes, 0 for no) 	0 	
Process 	Is this a repeat of an earlier incident (score 0 for yes, 1 for no) 	0 	
Tooling 	Was there, before the incident occurred, open tasks that would prevent this incident / make mitigation easier if implemented? (score 0 for yes, 1 for no)	0 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? (score 1 for yes, 0 or no) 	0 	
Tooling 	Did existing monitoring notify the initial responders? (score 1 for yes, 0 for no) 	1 	
Tooling 	Were all engineering tools required available and in service? (score 1 for yes, 0 for no) 	 	
Tooling 	Was there a runbook for all known issues present? (score 1 for yes, 0 for no) 	 	
Total score	Total score 	5
Incidents/2022-03-31 api errors

==Summary==
After a code change rolled out in this week's train, the GlobalUsersPager class (part of CentralAuth) produced expensive DB queries that exhausted resources on s7 database replicas.
Backpressure from the databases tied up PHP-FPM workers on the API servers, triggering a paging alert for worker saturation. The slow queries were identified and manually killed on the database, which resolved the incident.
Because the alert fired and the queries were killed before available workers were fully exhausted, the impact was limited to s7. Full worker saturation would have resulted in a complete API outage.
Because only two engineers responded to the page and the response only took half an hour, we decided not to designate an incident coordinator, start a status doc, and so on. We didn't need those tools to organize the response, and they would have taken time away from solving the problem.
Documentation:
* Phabricator task detailing the slow query
* API server RED dashboard showing elevated latency and errors; php-fpm workers peaking around 75% saturation; and s7 database errors
* Same dashboard for the app servers showing measurable but lesser impact
==Actionables==
* Revert the patches generating the slow queries - done 
* Later (2022-04-06) it was discovered the query killer was using the old 'wikiuser' name, which prevented it from acting. Fixed in , deploying soon.
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Score	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? (score 1 for yes, 0 for no) 	0 	Info not logged
People 	Were the people who responded prepared enough to respond effectively (score 1 for yes, 0 for no) 	1 	
People 	Were more than 5 people paged? (score 0 for yes, 1 for no) 	0 	paged via batphone
People 	Were pages routed to the correct sub-team(s)? (score 1 for yes, 0 for no) 	0 	paged via batphone
People 	Were pages routed to online (business hours) engineers? (score 1 for yes,  0 if people were paged after business hours) 	0 	paged via batphone
Process 	Was the incident status section actively updated during the incident? (score 1 for yes, 0 for no) 	0 	
Process 	Was the public status page updated? (score 1 for yes, 0 for no) 	0 	
Process 	Is there a phabricator task for the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Are the documented action items assigned?  (score 1 for yes, 0 for no) 	1 	
Process 	Is this a repeat of an earlier incident (score 0 for yes, 1 for no) 	1 	
Tooling 	Was there, before the incident occurred, open tasks that would prevent this incident / make mitigation easier if implemented? (score 0 for yes, 1 for no)	1 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? (score 1 for yes, 0 or no) 	1 	
Tooling 	Did existing monitoring notify the initial responders? (score 1 for yes, 0 for no) 	1 	
Tooling 	Were all engineering tools required available and in service? (score 1 for yes, 0 for no) 	1 	
Tooling 	Was there a runbook for all known issues present? (score 1 for yes, 0 for no) 	1 	
Total score	Total score 	9
Incidents/2022-04-06 esams network

==Summary==
This was due to an issue at the Amsterdam Internet Exchange (AMS-IX).
Documentation:
* https://www.wikimediastatus.net/incidents/jnqvz8gljzhy
*Restricted document
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Score	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? (score 1 for yes, 0 for no) 	1 	
People 	Were the people who responded prepared enough to respond effectively (score 1 for yes, 0 for no) 	1 	
People 	Were more than 5 people paged? (score 0 for yes, 1 for no) 	0 	Paged via batphone
People 	Were pages routed to the correct sub-team(s)? (score 1 for yes, 0 for no) 	0 	Paged via batphone
People 	Were pages routed to online (business hours) engineers? (score 1 for yes,  0 if people were paged after business hours) 	0 	Paged via batphone
Process 	Was the incident status section actively updated during the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Was the public status page updated? (score 1 for yes, 0 for no) 	0 	
Process 	Is there a phabricator task for the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Are the documented action items assigned?  (score 1 for yes, 0 for no) 	0 	
Process 	Is this a repeat of an earlier incident (score 0 for yes, 1 for no) 	1 	
Tooling 	Was there, before the incident occurred, open tasks that would prevent this incident / make mitigation easier if implemented? (score 0 for yes, 1 for no)	1 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? (score 1 for yes, 0 or no) 	1 	
Tooling 	Did existing monitoring notify the initial responders? (score 1 for yes, 0 for no) 	1 	
Tooling 	Were all engineering tools required available and in service? (score 1 for yes, 0 for no) 	1 	
Tooling 	Was there a runbook for all known issues present? (score 1 for yes, 0 for no) 	1 	
Total score	Total score 	10
Incidents/2022-05-02 deployment

==Summary==
Parts of /srv/deployment were lost on active deployment server (deploy1002) due to the wrong command, rm -rf, being executed. This halted deployments for some time, until we were able to restore the directory from a backup and we checked it for correctness.
Documentation:
*T307349
*Bacula backups
==Actionables==
* https://phabricator.wikimedia.org/T309162
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	yes	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	no 	
People 	Were pages routed to the correct sub-team(s)? 	/ 	
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	/ 	
Process 	Was the incident status section actively updated during the incident? 	no 	
Process 	Was the public status page updated? 	no 	
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	no 	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	yes	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	yes	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	no 	
Tooling 	Were all engineering tools required available and in service? 	yes	
Tooling 	Was there a runbook for all known issues present? 	yes	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incidents/2022-05-20 Database slow

==Summary==
On 2022-05-14 at 8:18 UTC there was a 3 minute impact on uncached traffic (high latency, unavailability) related to application server worker thread exhaustion caused by slow database response. There was no clear root cause at the time. The incident occurred again on the same database host on 2022-05-20 at 09:35 UTC, this time lasting for 5 minutes. After further investigation the likely root cause is a MariaDB 10.6 performance regression under load, further researched in https://phabricator.wikimedia.org/T311106.
Documentation:
*MediaWiki Exceptions
==Actionables==
*Investigate mariadb 10.6 performance regression during spikes/high load
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	no 	
People 	Were pages routed to the correct sub-team(s)? 	no 	
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	
Process 	Was the incident status section actively updated during the incident? 	no 	
Process 	Was the public status page updated? 	yes	
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	yes	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	yes	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	yes	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	yes	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	11
Incidents/2022-05-21 varnish cache busting

==Summary==
A flood of API traffic from an AWS instance caused caching servers to be overloaded. Services behind our caching layer were up, but not reachable during this time.
==Actionables==
*T308952 - get a legend for haproxy "anomalous session termination states"
*T308940 - follow-up on user reported ticket with public incident report
*T308941 - semi related: Klaxon redirects to http
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.		
Process 	Was the incident status section actively updated during the incident? 		
Process 	Was the public status page updated? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were all engineering tools required available and in service? 		
Tooling 	Was there a runbook for all known issues present? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incidents/2022-05-26 Database hardware failure

==Summary==
For approximately 12 minutes some internal services (e.g. Bacula and Etherpad) were not available or operated at reduced capacity. This was caused by a faulty memory stick leading to a reboot of db1128, which was at the time the primary host of the m1 database section.
Documentation:
*Full list of potentially affected services
==Actionables==
*Failover m1 primary db from db1128 to db1164
*db1128 faulty memory
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	no 	
People 	Were pages routed to the correct sub-team(s)? 	no 	
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	
Process 	Was the incident status section actively updated during the incident? 	no 	
Process 	Was the public status page updated? 	no 	
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	yes	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	no 	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	no 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were all engineering tools required available and in service? 	yes	
Tooling 	Was there a runbook for all known issues present? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	7
Incidents/2022-06-01 Lost index in cloudelastic

==Summary==
During a reimage operation, the cloudelastic Elasticsearch cluster lost a shard and went into red status, indicating data loss.
Until the data was restored, search results were incomplete on Cloudelastic. Restoration from production snapshots, using the previously understood and documented process, failed consistently, requiring a different approach to be devised which is why restoration was delayed by a month. Restoration was completed on 12 July. 
Documentation:
*https://phabricator.wikimedia.org/T309648#8072778
==Actionables==
*Restore data to cloudelastic
*Document cloudelastic cluster (what is its purpose, who are the stakeholders, etc)
*Document restore process
*Review monitoring for cloudelastic
*Inform stakeholders of the current situation
TODO: Add the #Sustainability (Incident Followup) and the #SRE-OnFIRE (Pending Review & Scorecard) Phabricator tag to these tasks.
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.		
Process 	Was the incident status section actively updated during the incident? 		
Process 	Was the public status page updated? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were all engineering tools required available and in service? 		
Tooling 	Was there a runbook for all known issues present? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incidents/2022-06-10 overload varnish haproxy

==Summary==
Lots of HTTP 429 from Varnish (due to ongoing rate-limiting) caused overload at the HAProxy level and general service disruption.
Documentation:
*HTTP 503s reported on phab https://phabricator.wikimedia.org/T310368
*https://www.wikimediastatus.net/incidents/5k90l09x2p6k
==Actionables==
*(Re) evaluate effectiveness / usefulness of varnish/haproxy traffic drop alerts https://phabricator.wikimedia.org/T310608
*Mitigate/fix overload situations between varnish and haproxy https://phabricator.wikimedia.org/T310609
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.		
Process 	Was the incident status section actively updated during the incident? 		
Process 	Was the public status page updated? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were all engineering tools required available and in service? 		
Tooling 	Was there a runbook for all known issues present? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incidents/2022-06-12 appserver latency

==Summary==
Documentation:
*https://www.wikimediastatus.net/incidents/nb6v1zxp86ns
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.		
Process 	Was the incident status section actively updated during the incident? 		
Process 	Was the public status page updated? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 		
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)