nyuuzyou/wikis huggingface dataset:
Incidents/2018-05-22 MediaWiki

== Summary ==
MediaWiki issues cause by Translate extension
Symptoms initially looked like db issues or network issues, then like a SWAT patch had caused the issue, but nothing seemed to line up that well.
See https://phabricator.wikimedia.org/T195293#4224220
== Timeline ==
Patchs Merged (as part of SWAT)
* 13:21 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert: Temp rate limit for arwiki due to mass vandalism T192668 (duration: 01m 18s)
* 13:25 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgUseRCPatrol on azwiki T194389 (duration: 01m 20s)
meta.wikimedia.org actions
* 13:26, 22 May 2018 FuzzyBot (talk | contribs) changed the state of Russian translations of Privacy policy from Needs updating to In progress
* 13:28, 22 May 2018‎ Kaganer (talk | contribs)‎ m . . (52,758 bytes) (+3)‎ . . (thank) - https://meta.wikimedia.org/w/index.php?title=Privacy_policy&diff=18066512&oldid=18063747
* 13:29, 22 May 2018 Kaganer (talk | contribs) marked Privacy policy for translation
* 13:32, 22 May 2018 Kaganer (talk | contribs) changed the state of Russian translations of Privacy policy from In progress to Needs updating
Issue
* 13:34 paladox@#wikimedia-operations: hmm https://meta.wikimedia.org/wiki/Privacy_policy is not loading for me
* 13:35 NotASpy@#wikimedia-operations: yeah, en.wp is crawling along for me.
* 13:35 addshore@#wikimedia-operations: *looks around*
* 13:35 addshore@#wikimedia-operations: I can see a bunch of db errors
* 13:35 addshore@#wikimedia-operations: spike in lag or issue with replication
* 13:37 https://phabricator.wikimedia.org/T195293 - 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly)
* 13:40 addshore@#wikimedia-operations: started at 13:31
* <discussion>
* 13:43 PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [50.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen
Reverts
* 13:44 addshore - Started first revert
* 13:46 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert Enable $wgUseRCPatrol on azwiki (duration: 01m 19s)
* 13:47 marostegui@#wikimedia-operations: all the connection errors I am seeing are on s7
* 13:48 addshore@#wikimedia-operations: s7 has ar wiki?
* 13:49 marostegui@#wikimedia-operations: addshore: actually yes
* 13:49 addshore - started second revert
* 13:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert Revert Temp rate limit for arwiki due to mass vandalism (duration: 01m 18s)
Recovery
* 13:52 marostegui@#wikimedia-operations: connections are decreasing on db1094
* 13:52 __joe__@#wikimedia-operations: yes, queues on the appservers are vanishing
* 13:52 volans@#wikimedia-operations: 500s goind down
* 14:11 RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 70.00% above the threshold [25.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen
== Conclusions ==
TODO
== Actionables ==
TODO
Incidents/2018-06-06 mx

== Summary ==
Between [2018/06/06 18:55 UTC] and [2018/06/07 02:00 UTC] outbound mail was interrupted for the below services (by puppet module/profile names).
Service (impact to outbound mail)
* Phabricator (outbound mail delayed)
* Gerrit (outbound mail lost)
* Hue (outbound mail lost)
* IEG grant review (outbound mail lost)
* Oozie (outbound mail lost)
* Wikimania scholarships (outbound mail lost)
* Sentry (outbound mail lost) 
* Snapshot wikidump (outbound mail lost)
== Timeline ==
* 18:55 UTC Keith stops Exim on mx1001 in preparation for OS reload planned the next day. 
* 01:27 UTC (June 7th) Gerrit and Phabricator email issues are reported via IRC.
* 01:34 UTC Sam Reed creates Phabricator task describing Phabricator and Gerrit email issues T196598
* 01:46 UTC Kunal pages SRE/Ops team via SMS.
* 01:54 UTC Faidon checks in on IRC and begins investigating.
* 02:00 UTC Keith checks in on IRC and begins investigating.
* 02:00 UTC Faidon restarts exim on mx1001 which fixes the issue . Affected services are able to send mail out once again. Deferred phabricator emails are dequeued and delivered.
== Conclusions ==
* Unexpected SPOFs and a lack of outbound mail queueing is present in several production services. The cause was tracked to two issues:
** Phabricator is configured to use mx1001 as primary, and mx2001 as a backup, for outbound mail. For an unknown reason failover to the backup mail server did not occur when mx1001 became unavaialble. 
** Some services are configured (via puppet) to use an SMTP server of mail_smarthost[0] which populates the service configuration with only the first mailserver listed for that realm, resulting in an SPOF.
== Actionables ==
* Investigate and address Phabricator backup email server configuration - phab:T196916
* Migrate services lacking built-in email queueing and failover to exim localhost SMTP server - phab:T196920
* Explore graphing of outbound mail volume on per-service or hostgroup level - phab:T197171
* Improve outbound mail service alerting - phab:T197172
Incidents/2018-06-25 wdqs

== Summary ==
Wikidata Query Service public SPARQL endpoint has been overloaded and started to timeout to LVS probes. Updates lagged for some time.
== Timeline ==
2018-06-24
* 15:27 icinga-wm: PROBLEM - LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds
* 15:28 icinga-wm: RECOVERY - LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 434 bytes in 0.018 second response time
* 15:33 godog: restart-wdqs on wdqs1004
* 15:35 godog: systemctl restart wdqs-blazegraph on wdqs1004
* 15:40 godog: restart wdqs-blazegraph on wdqs1005
* 16:09 godog: restart wdqs-updater on wdqs1005
* 16:31 icinga-wm: PROBLEM - LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout * after 10 seconds
* 16:32 icinga-wm: RECOVERY - LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 434 * bytes in 0.056 second response time
* 19:41 gehel: restart blazegraph on wdqs1004
* 21:51 gehel: restarting wdqs1005 after taking a few thread dumps - T198042
* 22:26 gehel: restarting wdqs1004 - T198042
2018-06-25
* 07:46 gehel: depooling wdqs1005 to allow it to catch up on updates - T198042
* 08:11 gehel: rolling restart of wdqs to enable async logging - T198051
* 12:00 gehel: repooling wdqs1005 it has catched up on updates - T198042
== Conclusions ==
Main tracking task: .
Our public WDQS endpoint is fragile (nothing new here). Investigating thread dumps, it looked like a lot of threads were stuck in logging. Blazegraph uses an unbounded executor service, having threads stuck in logging makes the situation worse. Logging asynchronously () and dropping log messages when under too much load would help, reducing the amount of logs would help as well. There is a deeper issue of expensive queries, but that one is hard to address directly.
During investigation we also realized that our banning mechanism was throwing HTTP 500 errors instead of HTTP 403 () due to a bug in throttling logging, which may have contributed to logging volume increase.
== Links to relevant documentation ==
A "runbook" page has been created for WDQS and explains the actions taken during this incident.
== Actionables ==
* log asynchronously (phab:T198051)
* banned users should be notified with HTTP 403, not with HTTP 500 (phab:T198055)
* If the server is severely backed up, it is hard for it to catch up and serve traffic at the same time. We need to look into the issue and see if we need to adjust cluster capability or find some other solution.
* WDQS logs are overly verbose, which both makes them hard to read and contributes to clogging the log pipeline. We should adjust them to be less verbose. (phab:T197645, phab:T198046)
Category:Incident documentation
Incidents/2018-06-26 LoadBalancers

== Summary ==
An attempt to use a broken functionality of scap, together with an unexpected behaviour of our load balancers made all wikis and all api endpoints fail almost completely for 6 minutes.
== Timeline ==
* 2018-06-26 08:45: Marko and Petr start a deployment window in which an important migration will take place
* 2018-06-26 08:48: A first full scap sync is run, but is aborted as mediawiki emits notices for Undefined variable: wmgUseEventBus
* 2018-06-26 09:03: A second full scap sync is attempted, again unsuccessfully (the notices persist)
* 2018-06-26 09:27: Given the situation doesn't improve, it is decided to run scap with the additional flag for restarting HHVM, and an heads-up is given in #-operations
* 2018-06-26 09:28: The scap command is run
* 2018-06-26 09:29-32: while Giuseppe acknowledges that he is sure the HHVM restart from scap is broken, and that it might even be harmful, he advises against stopping the command as the worst-case scenario (all servers get depooled and none is restarted) should be protected by pybal as T184715 is resolved. Moreover, since that feature is still there in scap, it might have been fixed in the meanwhile. 
* 2018-06-26 09:33: It is noticed that the worst-case-scenario behaviour from scap is happening. Scap is abruptly stopped, but by this time, it has removed more than 90% of all servers in both clusters from the pool. Comment on IRC from Giuseppe "let's hope pybal saves us"
* 2018-06-26 09:34: first reports of sites being down from users
* 2018-06-26 09:35: A quick glance at the state of the pools on one of the eqiad load balancers confirms that T184715 is indeed not fixed
* 2018-06-26 09:35: A shower of alerts start pouring - our monitoring confirms everywthing's more or less down
* 2018-06-26 09:36: A first mass-repool (for appservers) is issued
* 2018-06-26 09:37: A mass repool of API servers, first in eqiad, then in codfw is issued
* 2018-06-26 09:38: Users report the sites are now working again
* 2018-06-26 09:42: All alerts clear
The real outage, which has been almost complete for non-cached resources, lasted between 9:32 and 9:38, so 6 minutes.
== Conclusions ==
The scap function that caused this issue has long been broken and needs to be removed from the software. That, combined with the pybal bug, caused this outage. While removing the scap function should be easy, most efforts should be spent in really fixing T184715.
== Actionables ==
* Remove --restart from scap (DONE: phab:T198185)
* Make pybal not depool servers if it goes below depool_threshold (TODO: phab:T184715)
Incidents/2018-07-04 mathoid

Due to operator error there was a 4 min outage of the mathoid service in codfw
== Summary ==
During reimaging of the codfw kubernetes hosts in preparation for the 1.8 upgrade, an operator oversight caused the mathoid service, which is served by the codfw kubernetes cluster to experience an outage of 4 mins due to no pods running.
== Timeline ==
* 07:18 akosiaris starts reimaging kubernetes100{1,2}.eqiad.wmnet kubernetes200{1,2}.codfw.wmnet without swap. Kubernetes100{3,4} and kubernetes200{3,4} had been reimage the previous day successfully. The process is the same, that is sequential with a large sleep (45 mins) between hosts. 
* 08:04 kubernetes1001 is done. akosiaris interrupts the process and removes the --sequential argument to the reimaging tool. 
* 08:05:34 pages arrive for mathoid codfw, akosiaris responds
* 08:06:26 akosiaris notices the issue. Namely, due to the previous day's reimaging all the pods were scheduled on the codfw hosts being reimaged. So with the 2 hosts being reimaged simultaneously, no alive host has pods scheduled on them.
* 08:07 kubernetes has already noticed the issue and is rescheduling pods on other nodes. This unfortunately is taking longer than normal since the available nodes are all pristine and hence have to download the container from the registry
* 08:08:20 akosiaris notes that kubernetes2004 is already starting the first container.
* 08:09:47 icinga notifies of a successful recovery.
== Conclusions ==
Humans are always the weakest link in the chain. However there was some quite useful information gained from this.
* The time for an automatic recovery with 0 human interaction on the platform was 4 minutes. 
* kubernetes does not do currently automatic rescheduling of pods. That is expected behavior and some work to resolve this is on https://github.com/kubernetes-incubator/descheduler. 
* chaos engineering can be a useful method to detect issues like this.
* Some 24 requests for math rendering formulas seem have initially failed.
== Links to relevant documentation ==
 We need to create a lot of documention on the kubernetes platform anyway
== Actionables ==
 None at this point in time 
Category:Incident documentation
Incidents/2018-07-11 fundraising undeliverable mail

root@ account was mercilessly spammed by bounces from full fr-tech@ box
== Summary ==
Fundraising was running a "Big English test" which means banners on English wikis and heavy donation traffic. An error in the Amazon Pay integration lead to an email alert per donation, which quickly overwhelmed delivery.
== Timeline (UTC) ==
* 15:00 banners go up, failmail starts rolling in to fr-tech@
* 15:30 based on the error Casey assumes the firewall needs a hole
* 15:36 Manuel alerts Casey that the root@ alias is getting mail about bounces to fr-tech@
* 16:00 Arzhel deploys Casey's erroneous firewall config, Casey chases tail
* 17:00 Arzhel points out that he cannot connect to the IP from his laptop, indicating the firewall is not the issue
* 18:00 further investigation with fr-tech seems to confirm that Amazon was advertising broken endpoints
* 20:00 Elliott discovers that the error message was misleading and not reporting the endpoint IP after all
* 20:30 Elliott finds the actual timeout was to http://sns.us-east-1.amazonaws.com/ to retrieve the session certificate
* 20:35 Casey finds the changed IP for that URL has not been whitelisted in local iptables (pfw was not the problem)
== Conclusions ==
* I (Casey) should have a better grasp on the firewalls. This was a good learning opportunity.
* We have had problems for years with Amazon changing IPs (T119002). Most of it was resolved by using a proxy IP they created for us (T176654). However the certificate retrieval URL has up to now not been manually resolved to it.
* The Amazon integration had rusted since last use and should have been kicked, either by a manual test or an automatic check, before being turned on full steam.
* The fr-tech email alert system should be smart enough to not send 1000s of the same message at once.
== Actionables ==
* Try using our proxy IP for the cert URL: phab:T199382
* Create further health checks for PSP endpoints: TBD, I think there are tasks for this already, combing Phab
* Throttle failmail before this happens: phab:T161567
Category:Incident documentation
Incidents/2018-08-08 Network

==Summary==
Topology changes made to improve the redundancy and stability of the switch stack asw2-a-eqiad caused it to drop ~1/3 of the packets transiting through its members for about 1h. This packet drop caused internal services to timeout/retry, exact user facing issues TBD but at least an increase of 5xx errors.
==Timeline (UTC)==
17:14 - First topology change made
17:43 - Last topology change made (T201145#4489225)
17:47 - First Icinga alerts, some high API latencies, puppetfails, etc. IRC spam is bad, but no major pages or signs of broader user-facing issues yet.
18:07 - Replaced fpc1-fpc3 link for T201095 (Unaware of the alerts)
18:10 - Started investigating asw2-a-eqiad
18:30 - Disabled fpc1-fpc3 link
18:33 - Minor user-facing disturbances begin showing up as a low-but-unusual rate of 503s
18:42 - 503 rate begins climbing significantly, reaching ~5% of all cache_text request rate at peak (probably roughly all of the misses and passes (e.g. logged-in traffic), only cache hits being served). Grafana
18:47 - Disabled fpc2-fpc4 link
18:47 - First Icinga recoveries
18:50 - 503 burst that began at 18:42 comes back to normal near-zero rate.
19:18 - eqiad front edge depooled in DNS, to stabilize and reduce risk during follow-on investigations fixups (takes 10 minutes for DNS TTLs to expire as this comes into effect)
==Conclusions==
* Virtual Chassis are black boxes, which makes it more difficult to investigate issues
* Topology changes included cable move, which makes a rollback more difficult
* Our current topologies are unsupported, this outage revealed that any changes, even though toward a more supported configuration can have bad consequences.
* Logging work done in SAL could have reduced the response time
* This event caused a driver issue on new cp1* servers, causing their link to be up on the switch side, but down on the server side
==Actionables==
* - Fix asw2-a-eqiad topology phab:T201145
* - Repool eqiad front edge traffic once eqiad is stable
Category:Incident documentation
Incidents/2018-09-18 train

Covers deployment of 1.32-wmf.22 ().
== Summary ==
1.32-wmf.22 went mostly fine, there are a few post train actionables though.
== Timeline ==
* Slow access to Special:Contributions on mediawiki.org
** Requests to https://www.mediawiki.org/wiki/Special:Contributions were terribly slow. DBA ruled out any issue with the database. Brad Jorsh pointed out at the actor storage refactoring happening in MediaWiki. It has been enabled on Monday for group0 a small set of testing wikis which includes mediawiki.org. A few related issues have been filled and the feature flag has been promptly disabled. Actionables are now child tasks of Deploy refactored actor storage
* Echo and Translate caused some PHP notices, apparently without much user impact. They have been quickly hot fixed by the code owners.
* OAuth had a fatal error due to a misnamed method in a refactoring patch. Gergő Tisza had a patch ready before I have finished completing the task!
* , a known issue. Upon deployment, some requests made to ORES timeout.
* Each scap action triggers a spike of web request took longer than 60 seconds and timed out errors. The requests taking longer than usual probably always happened, but previously there was no timeout on the web requests (enabled since ).
* scap was checking canaries from the dormant datacenter (eqiad) instead of the active one (codfw). The list of hosts is hardcoded in puppet and has not been changed during the switch.
* Canaries would not catch the web request took longer than 60 seconds which happens after scap canarie check window of 20 seconds.
== Conclusions ==
* New features introduced with feature flag / limited to a group are priceless.
* We have proper web request timeout, which highlight an issue we probably had for ages.
* Adding forceprofile=1 to a wiki url is a fast way to pin point PHP code slowness.
* PHP notices are now reported in logstash at error level (thanks to Timo). They come with a stacktrace/context which is way better than the hhvm logbucket that simplies relays hhvm stdout/stderr without much infos.
* the train is so automated, Antoine did not even notice the canaries check were pointing the wrong host.
== Links to relevant documentation ==
* Heterogeneous deployment/Train deploys
== Actionables ==
* phab:T204871 Deployments of MediaWiki with scap cause a spam of "web request took longer than 60 seconds and timed out"
** To be raised to Scrum of Scrum for investigation.
* phab:T204907 Scap is checking canary servers in dormant instead of active-dc
** Switchover process needs an update. Long term the list of canaries should be in conftool.
* phab:T204961 ORES requests for wikidatawiki models=damaging end up with HTTP request timed out
** Acknowledged by Amir Sarabadani
Incidents/2018-10-02 maps

== Summary ==
Maps overloaded actinium web proxy with errors by trying to connect to eventbus through it. The issue was mitigated by shutting down tilerator and kartotherian on maps1004 and removing the overflowing logs on actinium. This affected all services using the proxy, in particular restbase and citoid failures were detected by Icinga.
== Timeline ==
* 6:39 2018-10-01: latest version of tilerator deployed to maps1004 (including new tile invalidation mechanism)
* 4:09 2018-10-02: url downloader issues reported by Icinga
* 5:00 2018-10-02: issue spotted by Joe
* 5:08 2018-10-02: overflowing log removed from actinium
* 5:15 2018-10-02: kartotherian stopped on maps1004
* 5:22 2018-10-02: tilerator stopped on maps1004
* 5:36 2018-10-02: recovery of restbase and citoid
== Conclusions ==
* Maps was wrongly configured to use url-downloader.{codfw|eqid}.wikimedia.org as a proxy for all connection since the migration to scap3 for deployment.
* maps1004 is depooled and freshly reimaged to stretch, initial data import was completed and initial tile generation is running
* active invalidation of tiles being regenerated was deployed to maps1004 , this sends invalidation messages through event bus
High tile invalidation load during initial tile generation, combined with wrong proxy configuration lead to an overload of url-downloader proxies, triggering cascading failures to other services using those proxies.
Open questions:
* My understanding is that only url-downloader in eqiad (actinium) should have been impacted. Looking at Icinga alerts, services in codfw have been failing as well (citoid endpoints health on scb200[1-6]).
* How do we prevent the cascading failure in case of one service overloads the proxies (akosiaris seems to have a few idea).
== Links to relevant documentation ==
Where is the documentation that someone responding to this alert should have (cookbook / runbook). If that documentation does not exist, there should be an action item to create it.
== Actionables ==
* remove proxy configuration from kartotherian / tilerator gerrit:463893 / gerrit:463892
* we validated that the expected tile invalidation rate was reasonable for varnish, but we have not checked that this rate is reasonable on eventbus phab:T186732
* document that tile invalidation should be disabled during initial tile generation
* Audit all services and remove usage of url downloader for internal IPs. Then update urldownloader configuration to disallow connecting to internal IPs to avoid this in the future
Category:Maps outages, 2018