nyuuzyou/wikis huggingface dataset:
Incidents/2019-04-15 maps

== Summary ==
Tilerator and Kartotherian failed to restart during planned restart for OpenSSL upgrade. Issue was tracked down to config being not readable, permissions were manually reset and service recovered.
=== Impact ===
* maps.wikimedia.org was partially unavailable for about 10 minutes. See graph.
=== Detection ===
* problem was detected by multiple Icinga checks.
== Timeline ==
* 13:56Z: rolling restart of kartotherian + tilerator started
* 14:04Z: first icinga alert about kartotherian and tilerator being down on maps1001
* ~14:16Z: permission reset for /srv/deployment/kartotherian on maps100[23]
* 14:17Z: recovery of kartotherian on maps100[23]
* 14:18Z: permission reset on /srv/deployment/kartotherian and /srv/deployment/tilerator for all maps servers
* 14:20Z: last direct recovery message from Icinga
== Conclusions ==
* Restarting a service should be a trivial operation, but was not, and was not tested properly on a single host before restarting the whole clusters.
* The wrong permissions on config files is not yet explained.
=== What went well? ===
* good collaboration of multiple people to resolve the issue
=== What went poorly? ===
* a trivial operation went wrong
* lack of focus by the operator during the restart
* restart was not tested
=== Where did we get lucky? ===
* N/A
== Links to relevant documentation ==
It isn't clear yet what went wrong in this case and what should be documented. Maps runbook is available, but does not contain anything that would have helped in this case.
== Actionables ==
* main task 
Category:Maps outages, 2019
Incidents/2019-04-16 varnish

== Summary ==
For approximately an hour, the traffic layer served bursts of 503 errors: up to ~50k/minute for several minutes at a time. It is unclear why this happened, and whether misbehavior at the traffic layer or at the appserver layer was actually at fault.
=== Impact ===
Approximately 553,000 HTTP 503 errors were served across all sites. https://logstash.wikimedia.org/goto/accfe83bffa587f460110942361af4a1
=== Detection ===
Automated monitoring (Icinga alerts on traffic availability) plus multiple staff/user reports in #wikimedia-operations.
== Timeline ==
All times in UTC.
* 18:24: HTTP 503 error rate begins to rise OUTAGE BEGINS
* 18:26: first alert from Icinga <+icinga-wm>	PROBLEM - HTTP availability for Varnish at esams on icinga1001 is CRITICAL: job=varnish-text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
* 18:52: jynus phones bblack
* 18:55: cdanis depools cp1085 after looking at Varnish mailbox lag console and https://phabricator.wikimedia.org/T145661
* 19:04: bblack performs varnish-backend-restart on cp1085
* 19:07: bblack performs varnish-backend-restart on cp1083
* 19:08: 503s taper off to 0 OUTAGE ENDS
Possibly-relevant other graphs:
* Spikes of connections from Varnish backends to appservers correlate with the 503s: https://grafana.wikimedia.org/d/000000439/varnish-backend-connections?orgId=1&from=1555438200000&to=1555442400000
* Some very-long-running WDQS queries also correlate: https://grafana.wikimedia.org/d/000000489/wikidata-query-service?panelId=26&fullscreen&orgId=1&from=1555438200000&to=1555442400000
It is unclear what of the above are symptoms vs causes.
== Conclusions ==
=== What went well? ===
* automated monitoring detected the incident
=== What went poorly? ===
* unable to root-cause incident
=== Where did we get lucky? ===
* Whatever was causing the issue stopped happening.
* The outage was not more widespread
== Links to relevant documentation ==
Where is the documentation that someone responding to this alert should have (runbook, plus supporting docs). If that documentation does not exist, there should be an action item to create it.
== Actionables ==
* Continue working on moving to ATS. Similar incidents have happened before, and continuing to investigate these failures of Varnish is not a good use of time.
Incidents/2019-04-23 varnish

== Summary ==
Similar Varnish 'mailbox lag' problems as many times before.
=== Impact ===
Approximately 82k queries lost (HTTP 503 served instead).
source
=== Detection ===
Automated monitoring -- Icinga alerts on traffic availability.
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied. Include the lead-up to the incident, as well as any epilogue, and clearly indicate when the user-visible outage began and ended.
All times in UTC.
* 19:54 Varnish mailbox lag begins climbing on cp1083 OUTAGE BEGINS
* 19:56 first Icinga alert for HTTP availability PROBLEM - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga1001 is CRITICAL: cluster=cache_text site=ulsfo 
* 19:57 Varnish mailbox lag recovers on cp1083 but begins climbing on cp1085
* 20:02 Varnish mailbox lag recovers on cp1085 OUTAGE ENDS
Graphs:
Mailbox lag
HTTP availability
== Conclusions ==
See Incident_documentation/20190416-varnish#Conclusions
== Actionables ==
See Incident_documentation/20190416-varnish#Actionables
Incidents/2019-05-03 varnish

== Summary ==
The traffic layer in eqiad reported various HTTP failures fetching from the application layer. FetchError seemed different from the usual varnish-be scalability issue (e.g. not the same as in Incident documentation/20190416-varnish). Instead of "Could not get storage" we were getting "HTC status -1" and (less frequently) "http format error". See https://phabricator.wikimedia.org/P8470 and https://phabricator.wikimedia.org/P8469 respectively. HTC status -1 happens when the function HTC_RxStuff returns HTC_S_EOF. The "HTTP format error" case was verified to not be happening at the appserver layer. This initially led us to think of a potentially different issue. However, the varnish backends throwing 503 errors were confirmed to consistently be those running for the longest amount of time, and restarting them solved the problem. 
=== Impact ===
Approx 900,000 HTTP 503s served, mostly between 05:00 and 05:40.
=== Detection ===
Automated: Icinga paged.
== Timeline ==
All times in UTC.
* 05:01: 503 errors begin at a slow but increasing rate OUTAGE BEGINS
* 05:09 Icinga pages about LVS in eqsin and codfw
* 05:32 Joe restarted varnish backend on cp1077
* 05:33 failed fetches move to cp1085
* 05:37 ema summoned
* 05:41 failed fetches from cp1085 disappear
* 05:51 failed fetches move to cp1089
* 06:00 failed fetches from cp1089 disappear OUTAGE ENDS
* 07:03 failed fetches from cp1089 return OUTAGE BEGINS
* 07:16 ema restarts varnish backend on cp1089
* 07:20 error rate returns to normal OUTAGE ENDS
Graphs:
* https://grafana.wikimedia.org/d/000000479/frontend-traffic?orgId=1&from=1556857800000&to=1556868300000
* https://logstash.wikimedia.org/goto/71cf7b8e8fde9e790d999276b7a57d3d
== Conclusions ==
* The on-disk Varnish storage backend does not scale.
== Actionables ==
* There is unfortunately no action that can be taken immediately. In the medium-term, our strategy is moving all cache backends to Apache Traffic Server.
Incidents/2019-06-03 eqiad-port-saturation

== Summary ==
We received a LibreNMS alert for port utilization over 80% on cr2-eqiad.wikimedia.org in xe-3/3/3 interface.
After some investigation, we identified a single client running in AWS performing multiple requests for media content using "User-Agent: python-requests".
=== Impact ===
No user-facing impact.
=== Detection ===
LibreNMS alert on #-operations
== Timeline ==
(In UTC)
* 12:23: <librenms-wmf> Critical Alert for device cr2-eqiad.wikimedia.org - Primary outbound port utilisation over 80%
* 13:22: faidon@weblog1001:/srv/log/webrequest$ head -n 2560000 sampled-1000.json | jq -r '.ip + " " + (.response_size | tostring)' | awk '{ sum[$1] += $2 } END { for (ip in sum) print sum[ip],ip }' |sort -nr | head -10
* 13:35: cdanis opens abuse report with AWS abuse team
* 13:50: crawler begins scraping larger objects again, network usage increases to max
* 14:00: ema merges https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/514005/ to block their User-Agent (also unavoidably blocking some legitimate traffic)
* 14:01: ema runs puppet on cp1084
* 14:05: outbound network returns to normal
== Actionables ==
* phab:T224884 - Rate limit requests to cache_upload
* phab:T224888 - Network port saturation should page
* phab:T224891 - Return HTTP 403 to requests violating User-Agent policy
* Begin enforcing our existing meta:User-Agent policy, after notifying community (TODO: file task). Some summary of discussion:
** Ratelimit/block 'default' UAs, like "curl", "python-requests", "python-urllib2", etc.
*** Probably allow only so many reqs/sec from a given IP for these 'default' ones
** First investigate how much traffic this would hurt
** wikitech-l@, mediawiki-api-announce@, and m:Tech/News are reasonable venues for announcements
Category:Incident documentation
Incidents/2019-06-11 maps

document status: 
==Summary==
While https://phabricator.wikimedia.org/T224395 is on which led to depooling codfw. Codfw was pooled back and only two nodes were available (see Incident documentation/20190603-maps). It was necessary to reimage all codfw servers to pickup the new partitioning scheme. And when maps2003 was reimaged which was the last to reimage, kartotherian codfw which is pooled had no node to route traffic to hence request timeouts and SREs were paged. Icinga started alerting from 8:21am
===Impact===
Users who were currently served from codfw were affect. They were affected for about 6min(from logstash)
===Detection===
Icinga alerts for kartotherian endpoints. Also from pybal, no nodes were pooled at codfw due to
==Timeline==
All times in UTC.
* 8:18 Reimaging of maps2003 started. OUTAGE BEGINS
* 8:21 Icinga alerts on IRC and pages were sent out''
* 8:26 maps200[124] was pooled OUTAGE ENDS
==Conclusions==
What weaknesses did we learn about and how can we address them?
The following sub-sections should have a couple brief bullet points each.
===What went well?===
* Outage was at least detected quickly
===What went poorly?===
*
===Where did we get lucky?===
* It was easy to link outage to the ongoing reimage of maps2003
==Links to relevant documentation==
We should streamline processes for maps to reduce outages. Maybe via a cookbook to ensure we capture all activities and steps before performing any activity on maps. 
==Actionables==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
*Streamline all maps activities via a script(cookbook) to make sure we don't miss anything?
*Always check all asumptions (never trust any)
Category:Incident documentation
Category:Maps outages, 2019
Incidents/2019-06-13 wdqs

== Summary ==
From June 13 ~15:10UTC to ~15:50 UTC the public WDQS endpoint in eqiad was overloaded by a bot to the point where it was not serving user queries. There is no reason to think that this bot was malicious. To mitigate this, the python-requests user agent is temporarily banned from accessing WDQS, consistent with our user agent policy.
=== Impact ===
The WDQS public endpoint in eqiad was unavailable from ~15:25 to ~15:45 UTC.
The python-requests user agent is still being banned, we are waiting to implement a more gentle solution before removing this ban.
The internal WDQS endpoint was not impacted.
=== Detection ===
Problem was detected by the Icinga LVS probe.
== Timeline ==
All times in UTC.
* 15:10: load starts to increase on the public wdqs eqiad cluster
* 15:31: Icinga LVS alert for wdqs.svc.eqiad.wmnet
== Conclusions ==
* identifying and throttling bots is a hard problem
* we need to take more drastic action to protect the stability of the service (aggressively throttle generic user agents)
=== What went well? ===
* problem was detected automatically in a timely manner
* good collaboration and clear communication between 
=== What went poorly? ===
* while we do have logic to throttle abusive bots, this throttling was not sufficient to protect the service
* we are still banning python-requests as a user agent, which affects a number of bots
=== Where did we get lucky? ===
* This happened during SRE offsite, when most SRE are in the same timezone. Luckily this wasn't when all of them were sleeping!
== Actionables ==
* We know that our throttling of bot is far from perfect (this is a hard problem). Some idea are already being discussed on phab:T219477
* ban logs are too verbose on disk (mitigated by https://gerrit.wikimedia.org/r/c/operations/puppet/+/516837)
* our user agent policy discourages the use of generic user agents, we should start to be more agressive in throttling generic UA or ban requests with no UA (https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/516834)
* our own clients should follow our policy, for example our icinga checks should have a meaningful UA (https://gerrit.wikimedia.org/r/c/operations/puppet/+/517032)
* remove the ban on the python-requests UA once we have a more gently throttling solution in place
Incidents/2019-06-19 JobQueue

document status: 
== Summary ==
Jobs stopped being processed after rollout of MediaWiki 1.34.0-wmf.10. 
With the Kafka job queue the jobs are serialized into JSON and then via EventBus and ChangeProp are delivered to an internal endpoint on job runners for execution. in a page_title property was removed from the job specification. We have been preparing for this, removing the property from the job JSON schema, however, we've forgot that the internal endpoint checks for the field to be present and returns 400 if it's not. The check is done very early in the codepath, so no logging was done, and change-prop ignored the 400. So the issue was not observable. It was fixed by 
=== Impact ===
Jobs were not processed for group0 and group1 wikis for 12 hours. Jobs that go around all the wikis, like GlobalRename, got stuck reaching a group0 or group1 wiki.
=== Detection ===
The issue was not detected automatically by monitoring tool, logs or graphs, it was found by accident due to MassMessage job being stuck
== Timeline ==
* 06-18 19:00-20:00 group0 and group1 deployed to wmf.10
*06-19 07:30 Issue discovered and ticket created
*06-19 07:40 group1 rolled back to wmf.8
*06-19 08:40 group0 rolled back to wmf.8
*06-24 Patch for the job executor deployed, train unblocked.
== Conclusions ==
=== What went poorly? ===
* No indication of a breakage in logs, graphs or monitoring tools
*Due to cpt offsite a long time has passed between the issue discovery and a fix
=== Where did we get lucky? ===
* The outage was noticed quickly due to stuck MassMessage jobs
== Links to relevant documentation ==
* Kafka Job Queue
== Actionables ==
* Change-Prop should report 400 errors from endpoints
* Better observability for jobqueue (TODO: link to existing task)
* Automated integration test? (TODO: create task)
Category:Incident documentation
Incidents/2019-06-24 wdqs

== Summary ==
WDQS public endpoint in eqiad was overloaded between 11:50 UTC until 13:15 UTC, leading to HTTP 5xx being served to users. Updates were disabled to mitigate the issue.
=== Impact ===
The outage (or at least reduced service availability) went for ~1.5h, leading to ~7K HTTP 5xx being served.
=== Detection ===
The LVS check was alerting.
== Timeline ==
All times in UTC.
* 11:58: increased rate of HTTP 5xx on WDQS public endpoint
* 12:04: first Icinga alert for "PROBLEM - LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds"
* 12:36: ban of GuzzleHttp user agent (this seems to be the UA generating the most read traffic at the moment)
* 12:49: restarting blazegraph on wdqs1004 (JVM thread out of control)
* 13:00: shutting down wdqs-updater on wdqs-public / eqiad
* 13:02: last Icinga recovery for "RECOVERY - LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 448 bytes in 0.341 second response time"
== Conclusions ==
=== What went well? ===
* internal WDQS clusters were not affected, segregating use cases works
=== What went poorly? ===
* while we do have throttling in place to keep read load under control, this was not sufficient to prevent the issue
=== Where did we get lucky? ===
* Not sure there was much luck here.
== Links to relevant documentation ==
Wikidata_query_service/Runbook#Overload_due_to_high_edit_rate
== Actionables ==
* Better throttle generic user agents (should be deployed later today)
* Rate limit updates
Incidents/2019-07-15 maps

document status: 
== Summary ==
While reloading the OSM data on maps eqiad cluster, maps1004 was depooled. The cluster became overloaded and started to timeout. The codfw cluster was depooled at that time. Traffic was switched to the codfw cluster and this allowed the cluster to recover.
=== Impact ===
The maps service was having an increased error rate from ~12:40 UTC to ~13:05 UTC.
=== Detection ===
Alerts were raised by Icinga (LVS checks) and we were watching the service closely since this was during an ongoing maintenance operation.
== Timeline ==
All times in UTC.
* 12:35 start of OSM data reload procedure OUTAGE BEGINS
* 12:43 Icinga "Kartotherian LVS eqiad on kartotherian.svc.eqiad.wmnet (get a tile in the middle of the ocean)" start to fail with timeout OUTAGE BEGINS
* 12:45 PYBAL check starts failing
* 12:50 restart of kartotherian on maps1002, the load climbs back up after the restart
* 12:54 shutting down tilerator on all maps eqiad nodes to free some CPU, no effect
* 12:59 realize that maps codfw was depooled, repooling it to absorb some of the load
* 12:59 depool maps eqiad (all traffic now served by codfw)
* 13:00 LVS check recovers on maps eqiad
* 13:05 error rate back to normal OUTAGE ENDS
* 13:16 repooling maps eqiad
== Conclusions ==
Average load on the maps eqiad cluster increased significantly after June 1. The restart during this incident brought the load back to its usual value. Surprisingly the number of tiles served during the same period did not change much.
We need to do further investigation into the causes of that increased load. We might want to review the sizing of the maps clusters to ensure they are ready to support the full load in a single data center.
== Links to relevant documentation ==
Where is the documentation that someone responding to this alert should have (runbook, plus supporting docs). If that documentation does not exist, there should be an action item to create it.
== Actionables ==
* Validate the sizing of the maps clusters .
Category:Maps outages, 2019
Incidents/2019-07-23 logstash

document status: 
== Summary ==
Logstash became overloaded during a network outage that caused elevated error rate.  Problematic log messages were also observed, which were able to crash logstash, requiring manual filtering of the problem logs.
=== Impact ===
Logstash was unable to process incoming logs on all inputs until the the problem log type was identified and filtered.  This resulted in delayed logs, and missing logs of the affected type (MediaWiki's SlowTimer)
=== Detection ===
Icinga alerts notified us to the issue.
== Timeline ==
* 19:10 - First page (logstash failure, secondary fallout from error ingestion) - A flood of errors caused by network disruption during eqiad rack a6/a7 pdu maintenance overwhelmed logstash.
** troubleshooting of related network issue proceeds with higher priority.  logstash outage presumed to be secondary effect of network issues.
* 20:10 network issue begins to improve, however logstash does not recover on its own
* 20:23 logstash unable to process backlog and is crashing due to UTF-8 parsing error  Suspected message example: https://phabricator.wikimedia.org/P8790
**
[FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: invalid byte sequence in UTF-8> 
* 21:00 UTF-8 issue traced to kafka rsyslog-shipper input.  Logstash collectors restarted with kafka rsyslog-shipper input disabled.  Kafka-logging consumer lag begins to recover on non-rsyslog topics https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=1563904908998&to=1563929943574&var-datasource=eqiad%20prometheus%2Fops&var-cluster=logging-eqiad&var-topic=All&var-consumer_group=All
* 21:45 UTF-8 issue traced to SlowTimer messages and a temporary fix to drop [message] =~ /^SlowTimer/ deployed
* 22:00 temporary fix is active across eqiad/codfw logstash collectors, kafka rsyslog-shipper input re-enabled.  Kafka consumer lag is recovering now on all topics
* 00:16 (next day) logstash has caught up with kafka backlog
== Conclusions ==
=== What went well? ===
* We successfully isolated what stream was generating the problematic messages.
=== What went poorly? ===
* It took us a while to determine where to put and install a filter that would drop the problematic messages.
=== Where did we get lucky? ===
* Happened in the middle of the day for two of the engineers with context.
=== How many people were involved in the remediation? ===
* 2 SRE
== Links to relevant documentation ==
* https://wikitech.wikimedia.org/wiki/Logstash
== Actionables ==
* [DONE] Investigate separating logstash pipelines
**Possible with current architecture, but multiple pipelines support requires logstash 6.x or newer
* [DONE] Investigate filtering/parsing options to avoid UTF-8 error observed (https://phabricator.wikimedia.org/T233662)
**https://github.com/logstash-plugins/logstash-filter-mutate/commit/ac4c440fba70e2ddb2a6a664af755f79893cfabd
**https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/101
**Requires logstash 7.x or newer
*Upgrade to logstash 7.x or newer https://phabricator.wikimedia.org/T183281 or otherwise upgrade mutate plugin to 3.5.0