nyuuzyou/wikis huggingface dataset:
Incidents/2019-01-22 maps

== Summary ==
After upgrading maps servers to stretch phab:T198622, there was need to bump the replicator factor on cassandra to increase resiliency and availability but this turned to be an issue which caused an outage on kartotherian service which is the front facing service clients access. This happened at eqiad cluster which was depooled and traffic was switched to codfw temporarily.
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
* 2019-01-22 19:22 UTC - first alert was seen on #wikimedia-operations IRC Channel. It reports response timeout on HTTP endpoint for maps100[2-4]
* 2019-01-22 19:37 UTC - maps1003 recovered.
* 2019-01-22 19:41 UTC - a paging alert was sent to ops about timeout on kartotherian.svc.eqiad.wmnet endpoint. Suggestions to switch to codfw cluster followed. 
* 2019-01-22 19:48 UTC - The issue continued as more alert was seen on #wikimedia-operations about socket timeout. Also further attempt to troubleshoot continued.
* 2019-01-22 20:03 UTC - patch to depool maps eqiad endpoint is ready for merge. 
* 2019-01-22 20:07 UTC - puppet was ran on lvs to enable the change which switch traffic to codfw cluster. 
* 2019-01-22 20:08 UTC - we were Ok to continue working on eqiad to find the possible solution.
* 2019-01-22 20:22 UTC - we discovered the analytics team have had similar issues - https://phabricator.wikimedia.org/T157354 which lead to an outage incident documented here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170223-AQS. We discovered that, after increasing the replicator factor for system_auth keyspace, Its is necessary to recreate the permissions on cassandra. permission was reset on the cluster and a task was created to track it - https://phabricator.wikimedia.org/T214434. also, codfw cluster remained the active cluster while we left eqiad to to further investigate what went wrong.
* 2019-01-23 10:32 UTC - steps to fix this issue after confirming its the same as https://phabricator.wikimedia.org/T157354 is documented here: https://phabricator.wikimedia.org/T214434.
* 2019-01-23 10:32 UTC - after confirming that endpoint work correctly, the previous day patch that disabled eqiad cluster was reverted. and puppet was ran manually.
* 2019-01-23 13:52 UTC - full recovery.
== Conclusions ==
What weakness did we learn about and how can we address them?
We still don't have an exact root cause. The Cassandra documentation seems consistent with the procedure followed but for some reason system_auth users/roles were wiped in the process. There are some remarks to highlight:
* please refer to https://wikitech.wikimedia.org/wiki/Incident_documentation/20170223-AQS
== Actionables ==
* Follow the safe procedure when increasing replicator_factor which means recreating the user/permission should immediately follow. This
* Document best practices for increasing system_auth replication factor on Wikitech and its pitfalls.
Category:Maps outages, 2019
Incidents/2019-02-08 logstash-mediawiki

== Summary ==
A single database host down caused an exceptional influx of log entries from MediaWiki, which in turn caused overload in logstash ingestion.
== Timeline ==
* 12:18 db1114 DOWN alert fires, see also 
* 12:30 DBAs are engaged
* 12:42 UDP packet loss alerts fire for logstash
* 12:44 db1114 is depooled
* 13:09 UDP packet loss for logstash recovers
== Conclusions ==
MediaWiki alone was able to cause a logstash overload, resulting in UDP packet loss. Applications using UDP as log transport have experienced loss of logs, while applications using the new logging pipeline (i.e. writing to Kafka, MediaWiki included) experienced a slowdown in log processing while logstash instances were catching up on the backlog.
Note that the length of this incident was a contributing factor in the UDP loss, a shorter reoccurrence (20min) happened on Feb 11th due to repool of db1118 but resulted in no UDP loss on the logstash side as instances were able to catch up. A deeper understanding of logstash performance characteristics is needed as well.
== Links to relevant documentation ==
* DB depooling: https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting
== Actionables ==
* A MediaWiki dependency being down (single database host in this case) should not cause log spam/overload 
* The logging pipeline will need some additional spam / ratelimit protection 
* Better understanding of Logstash performance
Incidents/2019-02-11 varnish

== Summary ==
A wrong change was merged cause varnish upload cache to spit out 301 to all requests. The change was soon reverted and service was restored. The impact was upload.wikimedia.org and maps.wikimedia.org were not really accessible for the duration of the incident.
== Timeline ==
All times are in UTC
* 16:43 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/488445/ gets merged after having received review and a PCC run was executed on it. The patch is being deployed on the usual automatic puppet schedule
* 16:55 First report of the problem on IRC, investigation starts. First reports are confining it to mobile, something that was not true. 
* 16:59 Corresponding spike on 301s noticed by bblack
* 17:02 bblack reverts the change and starts deploying it
* 17:02 akosiaris, _joe_, gtirloni have already jumped in and help with the investigation
* 17:03 icinga-wm notices the issue and alerts because the test returned 301 instead of 200. But the alert was specific to maps. But by now the nature of the problem was clear.
* 17:06 reports of desktop being broken arrive as well, further confirming the theory that the change was related.
* 17:07 icinga-wm reports CRITICAL for the nginx-ssl terminators as well
* 17:06 discussion in to whether we need to issue purges, bblack confirms that is not needed.
* 17:08 incident response documentation is started
* 17:12 icinga-wm reports recoveries
== Explanation ==
* bblack: 1) The patch was missing a bit for the cache_upload case, which caused the wikimedia_nets/wikimedia_trust ACLs to puppetize as empty sets
* bblack: 2) The caches only trust the "X-Forwarded-For: https" header based on those ACLs, so it was blanking out that header which it got from our local nginx TLS proxy
* bblack: 3) Therefore it thought HTTPS traffic was not HTTPS, and emitted a 301 redirect to https:// (same URI)
== Conclusions ==
* Humans are the weakest link.
* A more complete PCC might have caught that.
* icinga was a tad late in noticing this, delayed by some 10mins
== Actionables ==
* Investigate if icinga should report sooner on the failed check above https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/490612
* Standardize and/or automate doing these kinds of verifications for VCL patches against all cluster/dc combos T216140
** Possibly: add support to PCC to have pre-set groups of nodes, making this easy
Incidents/2019-02-13 PAWS

== Summary ==
The PAWS service was offline for several hours due to issues connecting to tools.db.svc.eqiad.wmflabs, related to the database being in the limit of usage ( T215993: tools.db.svc.eqiad.wmflabs hitting it's limit?).
See also the following day's related incident: 20190214-labsdb1005
== Timeline ==
All times are in UTC.
* 03:38 first icinga alert about PAWS being down.
* 04:26 tools.db.svc.eqiad.wmflabs issues reported in T215993: tools.db.svc.eqiad.wmflabs hitting it's limit?.
* 04:33 Bryan acknoledges the database issue.
* 04:43 icinga recovery for PAWS (flapping?).
* 05:38 second icinga alert about PAWS being down.
* 05:43 icinga recovery for PAWS (flapping?).
* 06:41 third icinga alert about PAWS being down.
* 06:42 Brooke and Arturo start looking into the PAWS issue. Very late/early for both.
* 07:27 detect an issue with the database (connection refused), didn't pay enough attention since it appeared transient in the logs at the time.
* 07:47 we start chasing issues regarding networking/proxying related to kubernetes since some parts of that had stopped working
* 08:37 Giovanni joins the debugging team.
* 08:45 Brooke goes to sleep, very late in her timezone. Arturo breakfast.
* 09:28 Giovanni realizes the database issue could be the root issue. Restarts the database.
* 09:42 icinga recovery for PAWS.
== Conclusions ==
The PAWS setup is really complex from the user-to-server path, lots of small spots to check & debug, at least 3 or 4 proxies are involved.
Also, these proxies are in different CloudVPS projects (project-proxy, paws, and tools).
We overlooked the issue in tools.db.svc.eqiad.wmflabs, probably due to being very late for Brooke and very early for Arturo.
The database issue itself is not weird (capacity saturation) given increased usage of our platforms.
It took us a lot of time to start debugging things properly because PAWS is poorly documented from the administration/design/architecture/operation point of view.
Since nobody in the WMF-WMCS team is really involved in mantaining PAWS, is not known which bits are important, what are the common things to check, etc. We weren't familiar with the setup.
In general, PAWS is in a weird state. It's a service that WMF/WMCS wants to offer to the Wikimedia Movement, but we don't have enough human capacity to fully support it.
The truth is that the project is mostly community-maintained, which is probably not enough for the importance of the service within the movement.
It's weird that we get paged at odd hours to fix a service that we aren't really supporting.
== Links to relevant documentation ==
We have a basic documentation page which clearly is not enough:
* PAWS
== Actionables ==
* Increase alerting level for tools.db.svc.eqiad.wmflabs. (TODO: create phab task)
* Increase capacity and robustness for tools.db.svc.eqiad.wmflabs. (TODO: create phab task)
** Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020
*** Degraded RAID on cloudvirt1019, Degraded RAID on cloudvirt1020
* Generate better documentation for PAWS. (TODO: create phab task)
* Clarify on the level of support offered for PAWS from WMF/WMCS.
Incidents/2019-02-20 irc-outage

== Summary ==
The IRC-based changes feed was not pushing out changes for 67 minutes following a restart of systemd-journald which was done as part of a security update.
== Timeline ==
* 11:01 - Moritz restarts systemd-journald on kraz.wikimedia.org as part of the rollout of a security update for systemd
* 12:00 - User sDrewth reports on the #wikimedia-operations channel "is it known that the RC IRC feeds have stopped?"
* 12:02 - Brian Wolff creates https://phabricator.wikimedia.org/T216607
* 12:08 - Moritz restarts the ircecho service on kraz and changes are propagated again
== Conclusions ==
* The Icinga status of kraz was checked following the restart of journald, but our monitoring didn't alert an error, despite being broken the udpmxircecho service continued to run and our Icinga check only validates the presence of the process, not whether it's functional
== Actionables ==
* phab:T216607 - Restarting systemd-journald breaks ircecho service
* phab:T216611 - Icinga check for ircecho should check for actual activity
* phab:T185319 - IRC RecentChanges feed: code stewardship request (not implemented yet)
Incidents/2019-02-28 eventbus extension and logstash

== Summary ==
An invalid mediawiki-config deployment caused EventBus extension to be misconfigured. This resulted in a spike in mediawiki errors which caused logstash instances to become overloaded and stop reliably processing logs until logstash persisted queueing was disabled.
See also: https://phabricator.wikimedia.org/T217385
== Timeline ==
* 19:40: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/492770/ was deployed as part of Morning SWAT. It was tested on mwdebug1002 before being deployed to the rest of fleet, but the misconfiguration would only manifest on group2 wikis.
*19:40 group2 wikis can no longer send events to eventlogging-service-eventbus.
*19:40: mediawiki log rate increases significantly
* 19:40: logstash queued events begin to increase
* 20:10: logstash udp loss begins to rise
*20:10: Mediawiki .wmf19 is deployed to group2 wikis. EventBus extension configuration now works everywhere.
* 20:15: logstash udp loss alerts fire
* 20:20: troubleshooting -- all associated services in stack appear healthy despite no logs being processed. logstash instances are logging normally with no errors, elasticsearch cluster health green, kafka cluster is in sync and healthy. logstash restarts do not improve the situation
*~20:30: Hashar notices that error logs are related to EventBus and notifies Ottomata. Hashar and Ottomata revert https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/492770/ just in case.
* 20:47: logstash persistent queue files are cleared and logstash instances restarted
* 20:47: logstash queued events begin to increase again
* 20:53: logstash udp loss alerts clear
* 21:07: logstash udp loss alerts fire again
* 21:09: logstash persisted queue disabled, instances restarted
* 21:13: logstash udp loss alerts clear
* 21:37: kafka-logging consumers back in sync
* 21:37: end of incident
*22:29: Ottomata realizes that the failed event data is in the udp_localhost-err topic in the Kafka logging cluster. He consumes the events and replays them to eventlogging-service-eventbus.
== Conclusions ==
* Logstash is not able to cope with a sudden significant increase in log volume, in this case mediawiki errors.
* The logstash persistent queue mechanism becomes unstable under significant load, resulting in broken logstash instances that remain in a broken state across restarts.
== Actionables ==
* Disable logstash persisted queues (done)
* Improve visibility of logging pipeline health by creating grafana dashboard with closely related metrics side by side (namely kafka consumer lag, mw log volume, logstash loss/queues) (done)
*Consider using Kafka logstash consumer throttling with Kafka quotas: https://kafka.apache.org/documentation/#design_quotas
Incidents/2019-03-13 MW API App Servers and EventGate

STILL A DRAFT waiting for phab to be back...
== Summary ==
On March 13, Petr and Otto enabled api-request logging to the new eventgate-analytics service for group1 wikis, bumping up volume to around 4000 messages / second. The HTTP requests from the EventBus extension were set to timeout after 60 seconds, and 1 (some?) out of 20 of the eventgate-analytics pods had errors producing to Kafka. This caused about 1/20th of POST requests to eventgate-analytics to wait 60 seconds before timing out. This caused MW API worker PHP threads to run low, resulting in 500 errors for MW API requests.
The produce errors seemed to be spaced about an hour apart, which leads us to believe that there is a produce volume threshold here that was being reached every hour. We are still investigating the cause of the produce problems.
Additionally, the volume of errors caused a significant increase in logstash messages. Logstash lagged. EventGate was logging the POSTed event data along with the error message, which caused logstash ElasticSearch indexing errors.
== Timeline ==
* 18:44 - otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging to eventgate-analytics for group1 wikis
* 19:12 - Icinga alerts
* 19:14 - Icinga recovery
* 20:12 - Icinga alerts
* 20:15 - Icinga Recovery (there are still EventGate and logstash errors, but MW API requests recover)
* ~20:20 - otto is notified in #mediawiki-security and realizes that the MW API 500s are caused by eventgate-analytics timeouts
* 20:24 - otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disabling api-request logging to eventgate-analytics for group1 wikis to investigate possible outage
== Conclusions ==
We are still investigating the cause of the EventGate Kafka produce errors, but the MW API and Logstash problems should have been avoidable.
There is no reason that analytics POST requests to eventgate-analytics need to have a 60 timeout. EventGate is configured to have a 'hasty' mode, where POST produce requests will be served without confirmation; i.e. the HTTP client will get a response immediately, and the Kafka produce will happen asynchronously. This should be enabled for analytics events.
Logstash ElasticSearch was erroring because EventGate was including event data as log fields in error messages. Event data should be serialized to a string rather than included as log fields.
== Links to relevant documentation ==
https://phabricator.wikimedia.org/T218255
https://etherpad.wikimedia.org/p/incident13Mar2019
== Actionables ==
* Enabling api-request eventgate to group1 caused minor service disruptions: https://phabricator.wikimedia.org/T218255
* EventBus extension should never log unserialized events: https://phabricator.wikimedia.org/T218254
* Decrease timeout for EventBus extension for analytics events: https://phabricator.wikimedia.org/T218260
* eventgate-analytics k8s pods occasionally can't produce to kafka: https://phabricator.wikimedia.org/T218268
* EventGate wikimedia implementation should emit rdkafka stats https://phabricator.wikimedia.org/T218305
Category:Incident documentation
Incidents/2019-03-20 ORES

ORES in CODFW stopped processing requests. The result was sustained overload errors and a growing backlog of requests to process. 
== Summary ==
This is a short (<= 1 paragraph) of what happened.
Please ensure to remove private information.
== Timeline ==
All times in UTC
* March 19th
** 0400 - We observe a very high, sustained request rate from Google Cloud in SFO The sustained request rate brings EQIAD/CODFW near capacity. (grafana of external requests)
* March 20
** 15:58 DNS oresrdb.svc.codfw.wmnet is switched over to oresrdb2002
** 14:02 - oresrdb2002 is rebooted for maintenance.
** 14:10 - oresrdb2002 comes back up.
** 14:12:22 - Score cache redis start and begins loading the redis databases from disk
** 14:13:57 - the score cache redis database loads the file and start accepting connections. It also begins a full resynchronization from master
** 14:12 - ORES codfw stops returning any scores (grafana of scores processed)
** 14:14 - ORES codfw begins to return overload errors (grafana of overload errors)
** 14:40 - Reversal of previous DNS change and forced restart of workers.
[14:40:21] <akosiaris> lemme reverst the switchover of the redis just in case
[14:43:21] <akosiaris> I 'll force a worker restart just to make sure it was that
* 14:43 - ORES codfw begins to return scores again (grafana of scores processed)
== Conclusions ==
Miscommunication between SRE team members ended up in the reboot of the backup redis server after it was switched to serve redis traffic.
== Links to relevant documentation ==
Where is the documentation that someone responding to this alert should have (cookbook / runbook). If that documentation does not exist, there should be an action item to create it.
== Actionables ==
* Implement a better redis HA solution https://phabricator.wikimedia.org/T122676
Incidents/2019-03-28 commons

== Summary ==
The main (gallery) namespace on Wikimedia Commons became uneditable for some 19 hours.
=== Impact ===
Editors on Wikimedia Commons were affected, though the gallery namespace is not as frequently edited as others (file uploads and categorization were unaffected, for instance).
=== Detection ===
End users reported the error on-wiki fairly soon after it became effective, and a Phabricator task was created as well.
However, it took over half a day for operations to become aware of the task.
== Timeline ==
All times in UTC.
* 2019-03-27 16:44 last edit in the gallery namespace (according to recent changes)
* 2019-03-27 16:55 config change deployed OUTAGE BEGINS
* 2019-03-27 18:43 first report on Commons’ Village pump
* 2019-03-27 20:43 second report on Commons’ Help desk
* 2017-03-27 20:53 phabricator:T219450 created
*2019-03-28 11:26 User:Yann mentions on the Phabricator task that this affects all NS0 edits, not just "some" as the task had been filed
* 2019-03-28 11:29 User:Yann mentions the Phabricator task in 
* 2019-03-28 12:02 config revert deployed OUTAGE ENDS
== Conclusions ==
=== What went well? ===
* Once the bug was reported in #wikimedia-operations, response was relatively quick
=== What went poorly? ===
* The bug was not detected on Test Commons or Beta Cluster Commons over months of testing.
* Wikibase in production (for Wikidata) uses NS0 for entities, but this is not true by default or on most development machines. This area of the code is thus not well-tested.
* A ticket about an UBN! situation had no response from the technical community for several hours, no escalation or notification method is in place for it.
=== Where did we get lucky? ===
* The main namespace is not frequently used on Commons, so work on files, categories etc. was unhindered
== Links to relevant documentation ==
* I followed SWAT deploys/Deployers when deploying the revert
* Wikibase config documentation?
== Actionables ==
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
* Fix the underlying bug and deploy the config again while testing *all* namespaces ()
* [task management] The largest issue was not the time between deployment and notice, but the time between the a ticket was first filed (untriaged, no tags) and it was acted on. Analyze what failed and either change Phabricator usage recommendations/reporting best practices, or something else (if considered necessary) when a clear outage is happening. ()
* Consider setting up production (regression) testing
Incidents/2019-04-07 Zotero

== Summary ==
On the morning of April 7, most of the zotero pods in codfw started using too much memory, and stopped responding to requests.
=== Impact ===
Given restbase and citoid are active/active, any user coming from eqsin, codfw or ulsfo and trying to get a citation would have seen a degraded service.
=== Detection ===
The LVS level check of zotero, and the service-checker test of citoid both reported the issue.
== Timeline ==
All times in UTC.
* 04:58 sudden increase in the memory used by the first zotero pod
* 05:05 all zotero pods in codfw have now high memory watermark. Service checker on citoid reports a problem
* 05:12 The zotero LVS endpoint has become unresponsive to monitoring. A page is sent. OUTAGE BEGINS
* 05:13 A recovery page arrives. The service will keep flapping and more pages are sent out 5:23 (recovery at 5:37), at 5:41 (recovery at 5:42) at 6:00
* 06:01 Alexandros, Giuseppe and marostegui respond to the page that is now being sent to people in EU timezones
* 06:03 Giuseppe depools zotero in codfw, even if the recovery arrives, so that the issue can be better analyzed. OUTAGE ENDS
* 06:23 After some log analysis, it is decided to kill the pods which still show a high memory watermark.
== Conclusions ==
There isn't much to conclude, apart from the fact we still have situations where zotero can fail because of sudden memory increases. This is a bug in the software and unless we can create a reproducible test case (as this seems to happen because of some user request), there aren't many chances to see it fixed. On the positive side, this hadn't happened in a long time.
The root cause of this specific incident is still TBD.
=== What went well? ===
* The problem is known, the monitoring adequate, the response clear. This isn't a new kind of outage and we're well equipped to respond to it.
=== What went poorly? ===
* No reponse to repeated pages/recoveries for almost 1 hour. 
* No runbook exists for this 
=== Where did we get lucky? ===
* It's fair to say a malfunctioning zotero doesn't have a huge impact. Requests rates average at 0.05 req/s. See https://grafana.wikimedia.org/d/NJkCVermz/citoid?refresh=5m&panelId=46&fullscreen&orgId=1&var-dc=codfw%20prometheus%2Fk8s&var-service=citoid&from=now-7d&to=now
* That some of us have a light sleep, probably?
== Links to relevant documentation ==
AIUI the documentation on how to operate on kubernetes for debugging should be written this quarter.
== Actionables ==
* Document kubernetes debugging procedures (TODO: Create task)
* Identify and reproduce the situation that caused the memory leak (TODO: Create task)