nyuuzyou/wikis huggingface dataset:
Incidents/2021-04-06 partial rack codfw

==Summary==
Around 17:00 UTC, a faulty network switch caused partial failure of one rack in the Codfw cluster. As 8 individual hosts become unreachable, this led to reduced redundancy or capacity for some of the affected services. The hosts were moved to new switch ports, and were unreachabe for about 1 hour.
* Elastic search: Three elastic nodes down, mild impact to redundancy (yellow state).
* Swift media storage: Three ms-be nodes down in the standby cluster, no user impact (should catch up once reachable).
* Edge cache: One cp node unreachable for the Codfw PoP, automatically depooled by healthcheck.
* DNS: One dns node unreachable, effectively depooled automatically per lack of BGP advertising.
== Notes ==
https://wm-bot.wmflabs.org/logs/%23wikimedia-operations/20210406.txt
 [17:01:22] <icinga-wm>	 PROBLEM - Host cp2036 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:01:26] <icinga-wm>	 PROBLEM - Host elastic2045 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:01:28] <icinga-wm>	 PROBLEM - Host ms-be2034 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:01:28] <icinga-wm>	 PROBLEM - Host ms-be2035 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:01:56] <icinga-wm>	 PROBLEM - Host elastic2046 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:02:44] <icinga-wm>	 PROBLEM - Host elastic2047 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:03:14] <icinga-wm>	 PROBLEM - Host dns2001 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:03:48] <icinga-wm>	 PROBLEM - Host ms-be2055 is DOWN: PING CRITICAL - Packet loss = 100%
 [17:13:19] <elukey>	 seems a rack down
 [17:13:46] <elukey>	 yep C2 
 …
 [17:38:21] <elukey>	 papaul: so I can connect to cp2035, that is in the same rack
 [17:53:24] <bblack>	 (cp2036 was pulled from service by external healthchecks, and dns2001 was pulled from traffic flow when it became unable to advertise BGP to routers)
 [18:14:51] <bblack>	 !log dns2001 - re-enabling and running puppet agent to restore service
 [18:18:15] <logmsgbot>	 !log bblack@cumin1001 conftool action : set/pooled=yes; selector: name=cp2036.codfw.wmnet
* Grafana: Host overview: cp2036
* Grafana: Frontend traffic (16:00 - 18:30)
==Actionables==
Tracking task for the incident: https://phabricator.wikimedia.org/T279457
Incidents/2021-04-15 appserver latency

==Summary==
From 07:31 to 08:16 UTC, there was increased latency and error rates for some MediaWiki cache misses and authenticated requests. The incident predominantly affected API users and bots on commons.wikimedia.org. API request latency for some requests went up by 5 seconds, and error rates upto 25%.
The issue was found to be caused by a bot causing increased load on the commonswiki databases, and consuming API webserver resources. Later in the time range, regular users and other wikis were affected as well through cross-wiki features involving Commons.
==Actionables==
Tracking task: https://phabricator.wikimedia.org/T280232
Incidents/2021-04-20 MediaWiki API slowdown

== Summary ==
From approximately 7:31 to 8:16 UTC, there has been instability serving Wiki uncached content (mostly impacted would be authenticated users and bots, as well as POST requests), affecting at first commonswiki, but later most other wikis, albeit at a lower rate.
The impact between those timestamp was increased latencies (up to 5+ seconds to return results) and high level of errors (approximately 1/4th of request lost). This was due to overload on commonswiki requests blocking resources on most api & database servers, which in turn created contention on most other wikis.
== Actionables ==
* Uncached wiki requests partially unavailable due to excessive request rates from a bot https://phabricator.wikimedia.org/T280232
* Determine safe concurrent puppet run batches via cumin https://phabricator.wikimedia.org/T280622
Incidents/2021-04-26 thumbor

== Summary ==
At 9:21, ms-be1062 had a crash in its network stack (tracked at T281107) which caused the server to blackhole traffic. This caused a lot of requests to swift to reach timeouts, causing the consequent starvation of resources for the thumbor cluster workers, which were mostly waiting for a response from swift. This happened during a rebalance operation of the swift cluster, which we've seen in the past can impose quite some stress on the swift backend in our current configuration. Traffic was diverted to the codfw cluster for generating thumbnails at 09:26, thus minimizing impact for users. Rebooting ms-be1062 at 9:38 solved the issue instantly.
== Actionables ==
* Understand why pybal would not depool any of the backends even if they were returning 503s even to monitoring (TODO: Create task)
* Track down the kernel bug that caused ms-be1062 to blackhole traffic. It’s also worrisome that a swift cluster would become unresponsive in such a situation. https://phabricator.wikimedia.org/T281107
Incidents/2021-06-15 Eqsin network

==Summary==
At 09:23 UTC, alerts indicated connectivity issues to the Eqsin cluster in Singapore. At 09:31 UTC, @Ema deployed a DNS change to depool the Eqsin cluster. This diverted most of its assigned traffic to Ulsfo, and some to Esams. At 09:35 UTC traffic started recovering, with traffic back to regular levels at 09:45 UTC. The 15-minute window is attributed to DNS caches expiring (e.g. at ISPs and on client devices). The connectivy issues were resolved later that day, and at 18:50 UTC @CMooney repooled the Eqsin cluster, with traffic back to regular levels in Eqsin by 19:00 UTC.
Impact: For about 35 minutes from 09:20 to 09:45 UTC, the wikis were largely unreachable from countries normally served by the Singapore DC (including India, Hong Kong, and Japan).
Documentation:
* Wikimedia DNS: DC geo map
* Grafana: Navigation Timing by Continent 2021-06-15
* Grafana: Traffic volume by DC 2021-06-15
==Actionables==
* Public tracking task: https://phabricator.wikimedia.org/T284986
* TODO Per-country Frontend Traffic dashboards T286554
Incidents/2021-06-29 trwikivoyage primary db

==Summary==
Shortly after the June 2021 switch over, the backend appservers were unable to serve requests to tr.wikivoyage.org for about 8 minutes from 14:22 UTC to 14:30 UTC. No other wikis were affected.
Impact: For 8 minutes, registered users on the Turkish Wikivoyage were consistently unable to load any pages or perform any actions. The general public may have noticed it to a lesser extent due to caching at our CDN layer, although any pages absent from the CDN cache would have also been been temporarily unavailable.
=== Details ===
The Turkish Wikivoyage project was created a few months ago on 19 January 2021 (T271260). The site configuration for it includes an assignment from the wiki to one of our primary databases (config patch), specifically on the s5 cluster. These assignments are done separately for each of our core datacenters: Eqiad, and Codfw, despite being identical.
The assignment for Codfw, our then-inactive datacenter, contained a typo, so MediaWiki would look for the trwikivoyage database in the s3 cluster instead of s5. This was not spotted in code review, and that particular configuration key was not subject to data validation, and after deployment the unavailability of this wiki within the inactive Codfw cluster was not covered by local monitoring.
Today at 14:21 UTC, we switched traffic from Eqiad to Codfw, and thus the typo became apparent through PHP Fatal Errors for backend requests to tr.wikivoyage.org. This was found through Logstash, and also detectable with alerts from the Prometheus metrics for error logs.
 [{reqId}] {exception_url} Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'trwikivoyage'
 Function: Wikimedia\Rdbms\DatabaseMysqlBase::doSelectDomain
 Query: USE `trwikivoyage` 
[14:26:11] <marostegui> I am checking an error related to: Unknown database 'trwikivoyage'
[14:27:09] <legoktm> indeed, https://tr.wikivoyage.org/ is down
[14:27:38] <marostegui> urbanecm: looks like it is being searched for in s3
[14:27:41] <marostegui> but it is on s5
At 14:28, @Urbanecm uploaded a patch to correct the configuration.14:30     <legoktm@deploy1002>     Synchronized wmf-config/db-codfw.php: fix trwikivoyage (duration: 01m 01s)At 14:30, @Legoktm finished the deployment of that patch.
Relevant documentation:
* Server admin log: 2021-06-29
* Switch overs
* Data centers
* Appservers
==Actionables==
* T260297: db-eqiad and db-codfw sectionsByLoad can get out of sync
Incidents/2021-06-30 Ulsfo traffic surge

==Summary==
Sometime after 22:00 UTC, a traffic surge started to fill up the capacity of a network port in the Ulsfo cluster. By 23:00 UTC the issue was mitigated.
Impact: For up to an hour, a portion of requests may have received a slower response from regions served by the San Francisco DC. This includes New Zealand, and parts of North America.
Documentation:
*Wikimedia DNS: DC geo map
==Actionables==
None.
Incidents/2021-07-14 eventgate-analytics latency spike caused MW app server overload

== Summary ==
While working on updating EventGate to support Prometheus, Andrew Otto deployed the changes to eventgate-analytics in codfw (then-active DC). This change removed the prometheus-statsd-exporter container in favor of direct Prometheus support, as added in recent versions of service-runner and service-template-node.
The deploy went fine in the idle "staging" and "eqiad" clusters, but when deploying to codfw, request latency from MediaWiki to eventgate-analytics spiked, which caused PHP worker slots to fill up, which in turn caused some MediaWiki API requests to fail.
The helm tool noticed that the eventgate-analytics deploy to codfw itself was not doing well, and auto-rolled back the deployment:
$ kube_env eventgate-analytics codfw; helm history production
REVISION	UPDATED 	STATUS 	CHART 	APP VERSION	DESCRIPTION
[...]
4 	Wed Jul 14 16:07:12 2021	SUPERSEDED	eventgate-0.3.1 	 	Upgrade "production" failed: timed out waiting for the co...
5 	Wed Jul 14 16:17:18 2021	DEPLOYED 	eventgate-0.2.14	 	Rollback to 3
Impact: For ~10 minutes, MediaWiki API clients experienced request failures.
Documentation:
* Grafana: Envoy telemetry
* Grafana: Application Servers dashboard
* Grafana: Envoy telemetry / Upstream latency
== Actionables ==
* Figure out why this happened and fix. Based on this log message, it seems likely that a bug in the service-runner prometheus integration caused the nodejs worker process to die. [DONE]
** Further investigation uncovered that require('prom-client') within a worker causes the observed issue. Both service-runner and node-rdkafka-prometheus require prom-client. It was proposed to patch node-rdkafka-prometheus to handle passing in the prom-client instance. 
** node-rdkafka-prometheus is an unmaintained project, so we have forked it to @wikimedia/node-rdkafka-propetheus and fixed the issue there. Additionally, if this issue in prom-client is fixed, we probably won't need the patch we made to node-rdkafka-prometheus for this fix.
Incidents/2021-09-04 appserver latency

==Summary==
An increase in load on a database server resulted in many queries being much slower to respond. This in turn meant backend traffic occupies appserver php-fpm workers for much longer, and a proportion of those requests will fail entirely due to unavailable workers. The failed requests got an error page with the message "upstream connect error or disconnect/reset before headers. reset reason: overflow".
Impact: For 37 minutes, backends were slow (taking several seconds to respond) and 2% of requests failed entirely. This affected logged-in users, most bots/API queries, and some page views from unregistered users for pages that were recently edited or otherwise expired from the CDN cache.
Documentation:
* Public task about incident. T290373
*Grafana: Application RED dashboard
File:4-Sep-2021-http-status.png|HTTP error rates.
File:4-Sep-2021-latency.png|Latency buckets.
File:4-Sep-2021-latency-quantile.png|Latency quantile estimates.
==Actionables==
* T277416 (restricted)
Incidents/2021-09-18 appserver latency

==Summary==
Slow database queries resulted in php-fpm worker exhaustion.
Impact: For about 10 minutes, backends were slow or unavailable for all wikis. This affected logged-in users, most bots/API queries, and some page views from unregistered users (pages that were recently edited or otherwise expired from the CDN cache). 
Documentation:
* Public incident task: T291311 
* Similar to Incident documentation/2021-09-04 appserver latency and Incident documentation/2021-04-15 appserver latency.
==Actionables==
* T284419 (restricted)
Incidents/2021-09-26 appserver latency

==Summary==
Increased db load for enwiki (s1) resulted in slower responses, which in turn resulted in overall php-fpm worker limits being reached and thus affecting requests for all wikis. For requests above the limit, the error was "upstream connect error or disconnect/reset before headers. reset reason: overflow".
Impact: For about 15 minutes, backend appservers were slower or unable to respond for all wikis. This mainly affected logged-in users and most bot/API queries. Some page views from unregistered users were affected, for pages that were recently edited or otherwise expired from the CDN cache. 
Documentation:
* Public incident task: T291767
* Similar to Incident documentation/2021-09-04 appserver latency and Incident documentation/2021-09-18 appserver latency.
==Actionables==
* T291767 (restricted)
* T251885 (restricted)
Incidents/2021-09-29 eqiad-kubernetes

==Summary==
Network in the kubernetes cluster was briefly disrupted due to a rolling restart of calico pods. For 2 minutes, all service in kubernetes were unreachable. That manifested in elevated errors rates and latencies in mediawiki among other things as mediawiki relies on many of those service, e.g. sessionstore.
The incident was the result of an operator forcing a rolling restart of all pods in all namespaces (in order to pickup a docker configuration change), a process that has been done multiple times in the past and never caused issues. The process works by deleting the old pods one by one. It is without much risk mainly because:
* Each delete depools first the pod, draining it of traffic
* The pod is asked to stop first and then is killed and deleted
* In the meantime a new pod has been spun up, probably in a different node, has run the initialization process and is ready to receive traffic
* The initialization process of almost all our services is very fast.
However this time around, the process fell on a race condition when it entered the kube-system namespace, which contains the components that are responsible for the networking of kubernetes pods. All calico node pods were successfully restarted, however before they had enough time to initialize and perform their BGP peering with the Core routers (cr*-eqiad.wikimedia.org), a component they rely on called calico typha was also restarted. Unfortunately we run only 1 replica of the typha workload. The new replica was scheduled in a new node that did not have the image yet on the OCI imagefs. Enough time elapsed while the image was being fetched and the new pod started by the kubelet that the graceful BGP timers on the core routers expired, forcing the core routers to withdraw from their routing tables the IPv4/IPv6 prefixes for the pod networks, leading to them becoming unreachable. The typha pod started up eventually, calico node pods managed to connect to it, fetched their configuration and BGP peered with the core routers re-announcing their pod IPv4/IPv6 prefixes. Routing was restored and the outage ended. No operator action was necessary to fix the issue, the platform auto-healed.
On the plus side, due to this (crude admittedly) form of chaos engineering, we found out a change in our kubernetes clusters and a need to amend our process.
Impact: For 2 minutes fetches to mediawiki resulted in higher error rates, up to 8.51% for POSTs and elevated latencies for GETs (p99 at 3.82s). Some 1500 edits failed (although there is duplication there, those are not unique edits). Those seem to have happened later, when the services were restored. Kafka logging lag increased temporarily.
Documentation:
* Save failures graphs
* Sessionstore graphs
* Appservers graphs
== Actionables ==
* TODO: Document the rolling pod restart procedure and skip kube-system namespace
*Investigate whether running >1 replicas of calico-typha is feasible and prudent. T292077
Incidents/2022-03-01 ulsfo network

==Summary==
Multiple of our redundant network providers for the San Francisco datacenter simultaneously experienced connectivity loss. After 20 minutes, clients were rerouted to other datacenters.
Documentation:
*https://gerrit.wikimedia.org/r/c/operations/dns/+/767250/
*https://www.wikimediastatus.net/incidents/2rp6n2cpym3m
*https://phabricator.wikimedia.org/P21629#102776
==Actionables==
* T303219 Integrate DNS depools with Etcd and automate/remove the need for writing a Git commit.
* Can we increase fiber redundancy?
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Score	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? (score 1 for yes, 0 for no) 	0 	Info not logged
People 	Were the people who responded prepared enough to respond effectively (score 1 for yes, 0 for no) 	1 	
People 	Were more than 5 people paged? (score 0 for yes, 1 for no) 	0 	
People 	Were pages routed to the correct sub-team(s)? (score 1 for yes, 0 for no) 	0 	
People 	Were pages routed to online (business hours) engineers? (score 1 for yes,  0 if people were paged after business hours) 	0 	
Process 	Was the incident status section actively updated during the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Was the public status page updated? (score 1 for yes, 0 for no) 	1 	
Process 	Is there a phabricator task for the incident? (score 1 for yes, 0 for no) 	0 	
Process 	Are the documented action items assigned?  (score 1 for yes, 0 for no) 	0 	one appears to be an open question
Process 	Is this a repeat of an earlier incident (score 0 for yes, 1 for no) 	0 	
Tooling 	Was there, before the incident occurred, open tasks that would prevent this incident / make mitigation easier if implemented? (score 0 for yes, 1 for no)	0 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? (score 1 for yes, 0 or no) 	1 	
Tooling 	Did existing monitoring notify the initial responders? (score 1 for yes, 0 for no) 	1 	
Tooling 	Were all engineering tools required available and in service? (score 1 for yes, 0 for no) 	1 	
Tooling 	Was there a runbook for all known issues present? (score 1 for yes, 0 for no) 	1 	
Total score	Total score 	7
Incidents/2022-03-04 esams availability banner sampling

==Summary==
A particular banner was deployed via CentralNotice that was both enabled for all users and with 100% sampling rate for its event instrumentation.
This caused instabilities at the outer traffic layer. The large amount of incoming traffic for event beacons, each of which had to be handed off to a backend service (eventgate-analytics-external), resulted in connections piling up and Varnish was unable to handle it and other traffic as a result, thus causing wikis to be unreachable in the affected regions. Initially Esams datacenter clients (mostly Europe, Africa and Middle East), with some temporary issues on other datacenters (Eqiad) as well when we initially attempted to reroute traffic to there.
Documentation:
*Varnish traffic 08:00-12:00 (Grafana)
*Frontend traffic 2xx responses (Grafana)
*navtiming pageview sampling (Grafana)
*https://www.wikimediastatus.net/incidents/rhn1l6k33ynz
*Restricted documented
==Actionables==
* Avoid flood of CN banner analytics
* Set a maximum for configurable sample rate of CentralNotice events that use EventGate
==Scorecard==
Incident Engagement™ ScoreCard
 	Question 	Score	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? (score 1 for yes, 0 for no) 	1 	
People 	Were the people who responded prepared enough to respond effectively (score 1 for yes, 0 for no) 	1 	
People 	Were more than 5 people paged? (score 0 for yes, 1 for no) 	0 	
People 	Were pages routed to the correct sub-team(s)? (score 1 for yes, 0 for no) 	0 	
People 	Were pages routed to online (business hours) engineers? (score 1 for yes,  0 if people were paged after business hours) 	1 	
Process 	Was the incident status section actively updated during the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Was the public status page updated? (score 1 for yes, 0 for no) 	1 	
Process 	Is there a phabricator task for the incident? (score 1 for yes, 0 for no) 	1 	
Process 	Are the documented action items assigned?  (score 1 for yes, 0 for no) 	1 	
Process 	Is this a repeat of an earlier incident (score 0 for yes, 1 for no) 	0 	
Tooling 	Was there, before the incident occurred, open tasks that would prevent this incident / make mitigation easier if implemented? (score 0 for yes, 1 for no)	0 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? (score 1 for yes, 0 or no) 	1 	
Tooling 	Did existing monitoring notify the initial responders? (score 1 for yes, 0 for no) 	1 	
Tooling 	Were all engineering tools required available and in service? (score 1 for yes, 0 for no) 	1 	
Tooling 	Was there a runbook for all known issues present? (score 1 for yes, 0 for no) 	1 	
Total score	Total score 	11