nyuuzyou/wikis huggingface dataset:
Incidents/20141126-oauth

== Summary ==
After the deploy of HHVM to the general public, some users reported OAuth issues. Those were linked to the failure of apache_request_headers() to get the correct OAuth headers when apache uses mod_proxy_fcgi. A few defines in the general apache config solved the problem.
== Timeline ==
* since 2014-11-21 a significant portion of the API traffic (~ 25%) was being served by HHVM.
* Around 16:00Z on 2014-11-25 the HHVM and the Zend appserver pools have been merged, and from that moment, one third of the requests to the backend were served by HHVM
* Users complained on IRC and bug https://phabricator.wikimedia.org/T75968 was filled at 05:24Z on 2014-11-26
* The culprit was identified in the fact that the Authorization header was not available to HHVM via apache_request_headers(). Further research showed that this is not HHVM's fault, but rather an apache mod_proxy_fcgi bug.
* Consequently, a patch was created to re-inject the header (https://gerrit.wikimedia.org/r/#/c/175952/); this was merged at 7:14Z, it propagated to the whole cluster in the next 20 minutes, which made oauth login in phabricator unbroken; The ticket was thus closed
* Around 10:20Z an editor reported a malfunction still persisting in flickr2commons due to OAuth failures
* After some further debug data was available, an inspection of the code showed another missing header that was later used by the oauth library, Content-Type. This was fixed with patch https://gerrit.wikimedia.org/r/#/c/175975/ which was merged at 11:40Z, and was propagated to the whole cluster in 20 minutes.
* No further report of errors from users, and the "Hello world" OAuth app magically got unbroken (I guess it was broken since we moved testwiki to HHVM some months back)
== Conclusions ==
Apache mod_proxy_fcgi is not the best part of the webserver, but the problem we encountered was in fact sneaky enough. One thing we might want to look at is why we didn't have any report of OAuth failures in beta, where the problem should've showed up earlier. 
== Actionables ==
No real actionable for this item
Category:Incident documentation
Incidents/20141126-ocg

== Summary ==
OCG (offline content generator) was not able to serve user requests (e.g. PDF versions of pages)
== Timeline ==
* 20141128T1140 user report on #wikimedia-operations about PDF generation not working, investigation begins
* 20141128T1147 disk space is suspected to be the root cause, investigation begins on that
* 20141128T1200 older (14d) PDFs are removed from ocg100* servers, ocg doesn't recover
* 20141128T1200 ocg logs on logstash indicate failure while talking to redis, investigation proceeds on that
 Nov 25 15:37:43 ocg1002 ganglia-ocg[15741]: ocg_job_status_queue 503449
 Nov 25 15:38:19 ocg1002 ganglia-ocg[25920]: ocg_job_status_queue 0
* 20141128T1220 it is discovered that ocg configuration ships with a blank password
* 20141128T1228 the impacting configuration change is fixed
== Conclusions ==
* There was user impact on the PDF generation starting 20141125T1537, no pages were issued
* The alarm for "ocg.svc.eqiad.wmnet" was silenced, and thus didn't fire pages
* The icinga OCG health check issue WARNING even for CRITICAL issues (e.g. returning HTTP 500, connection refused, etc)
* OCG disks were almost full, at >90% utilization
== Actionables ==
* Permanent silencing alarms for production services is discouraged, if silencing is desired for a given service the "downtime" facility is to be preferred. Downtime will auto-expire after the chosen period and thus lessen these problems.
* OCG icinga health checks should correctly report CRITICAL vs WARNING conditions
* OCG service excessive disk utilization should be checked and automatically reclaimed (e.g. utilization thresholds or date thresholds)
Category:Incident documentation
Incidents/20150107-EventLogging

== Summary ==
For about 12 hours 70% of event logging events did not include an schema field and thus did not validate
Issue affected client side events, which are the majority of events sent. Server side events and events sent via client apps were unaffected.
== Timeline ==
* Changeset gets merged on January 6th
* Changeset gets deployed at 2 am UTC on January 7th to mediawiki
* Alarms are raised on throughput of events
* Look at logs, establish cause and send patchset to fix it
* Fix patch is deployed on the UTC evening (morning on the west coast)
== Conclusions ==
Deployments should be tested on Beta Cluster, we should avoid going directly to production
== Actionables ==
We actually learned about this issue via the alarms so, while we could have alarms on the database, which we do not, the alarms on throughput did their job
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150205-EventLogging

== Summary ==
Accidentally deployed EventLogging code broke EventLogging's validation of client side events between 2015-02-04 11:45 PST (2015-02-05 07:45 UTC) -- 2015-02-05 08:15 PST (2015-02-05 16:15 UTC)
Server side events were not affected but they only constitute about 30% of the pipeline. Client side events were not entered in the database at all during this outage.
== Timeline ==
;2015-02-04 11:45 PST
:Reviewed but untested EventLogging code got deployed unintentionally.
:EventLogging's processor could not validate client side events as there were two items on the capsule that were not expected:
'clientValidated' and 'isTruncated'. Clients did not have the latest version of this code where these two fields are not being sent.
:Icinga alarms regarding validation were triggered
Procedure for removal of optional fields is normally to deploy changes to clients first and once changes are propagated to all clients 
we deploy the removal of changes to the server thus making the change backwards compatible for the period of the transition.
;2015-02-05 08:00 PST (2015-02-05 07:45 UTC)
:Team wakes up, figures out what happened and reverts code
;2015-02 05 08:15 PST (2015-02-05 16:15 UTC)
:Validation proceeds normally
== Conclusions ==
* EventLogging deployments are anonymous. But as (regardless of who deployed) only an existing, reviewed commit from gerrit and not rogue code got deployed, it seems the deployment happened accidentally by trusted people, and was not ill-intentioned. We are making sure all team members are aware that before code gets deployed to production it needs to go through Beta Cluster and be tested there.
== Actionables ==
* Make sure that log files exist for the affected period, so we can backfill.
* Backfill the data https://phabricator.wikimedia.org/T88692
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150206-EventLogging

== Summary ==
EventlLogging code was dropping events sparsely from 2015-02-06 to 2015-02-10
== Timeline ==
;2015-02-05 08:15 PST
:El code got deployed from mainline (and not logged in SAL) to fix issues with incident on 
20150205
Code seems to be working normally.
;2015-02-06/2015-02-09
Alarms regarding throughput get trigger.
;2015-02-11
:Developers researchs events on db versus validated events on log and finds discrepancies. Those should agree not 100% but about 99%. (There are valid events that do not get inserted due to encoding issues and other errors)
 mysql --defaults-extra-file=/etc/mysql/conf.d/research-client.cnf --host dbstore1002.eqiad.wmnet -e "select left(timestamp,8) ts , 
 COUNT(*) from log.ServerSideAccountCreation_5487345 where left(timestamp,8) >= '20150128' group by ts order by ts;"
 ts COUNT(*)
 20150128 18237
 20150129 17546 
 20150130 16556
 20150131 15814
 20150201 17079
 20150202 17387
 20150203 17888
 20150204 11496
 20150205 6640
 20150206 11159
 20150207 10307
 20150208 10095
 20150209 10375
== Conclusions ==
* We would benefit from looking at alarms right away and not wait several days
* More precise alarms as to what is going on wouldn't hurt 
== Actionables ==
* Make sure that log files exist for the affected period, so we can backfill.
* Backfill the data https://phabricator.wikimedia.org/T88692
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150320-EventLogging

== Summary ==
A storm of WikiEditor Edit schema events (several hundred per second) caused EventLogging to occasionally drop client-side events.
Server-side events were not affected, but they only constitute about 30% of the pipeline. Client-side events were not entered in the database at all during this outage.
== Timeline ==
;2015-03-19
: The new WikiEditor EventLogging instrumentation was deployed in the wmf21 train.
: There were use cases that were not anticipated by the Editing Team, for example, some special wikis were loading wikitext editor in a non-standard way.
:Changesets that solved issues related to wikitext and instrumentation: 
:* gerrit:197810
:* gerrit:198095 (backported and deployed to wmf21, backported but not yet deployed to wmf22),
:* gerrit:198134
Main Phabricator ticket: phab:T93242
;2015-03-21
: After deployment of changesets, event inflow diminished and system went back to normal.
== Conclusions ==
* We need throttling in eventlogging so in cases like this we can "shed" a stream of events. We have had this item in our backlog for a while: phab:T69470
== Actionables ==
* Backfill the data: phab:T93602
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150324-Eqiad-Rack-C4

== Summary ==
At around 09:30 UTC on March 24, the network switch for Rack 4 in Row C in our eqiad datacenter went offline. This immediately stopped network traffic to all machines in this rack. None of the affected services were set up for paging ops as critical services, so no pages went out. The switch came back online and the affected machines were connected to the network again at about 09:50 UTC, for a total outage time window of about 20 minutes. Notable affected higher-level services included Phabricator and Graphite.
== Timeline ==
* 09:30 UTC - eqiad asw-c4 switch goes offline; Icinga alerts for hosts down. Immediately ets noticed by staff (Filippo, Giuseppe, Gage)
* 09:33 UTC - Cause determined to be asw-c4
* 09:43 UTC - Filippo calls Faidon
* 09:47 UTC - Faidon logs in, connects to serial console of asw-c4-eqiad via the scs service and presses enter. The switch emits "Rebooting...", reboots itself and ultimately comes back up online.
* 09:50 UTC - Icinga host UP alerts.
== Conclusions ==
# Not all opsens are familiar with layouts & processes regarding out of band console access and how to connect to otherwise unreachable equipment. While the switch stack (sans C4) was accessed and there was sufficient networking awareness to identify the issue, there was confusion on how to connect to the crashed switch via its console.
# This is the second time the same symptoms are encountered on the exact same switch (but no others), see November 30th's incident. This is likely a hardware fault and will need to be addressed by a hardware swap.
== Actionables ==
Being tracked in Phabricator at https://phabricator.wikimedia.org/T93730
Category:Incident documentation
Incidents/20150331-LabsNFS-Overload

== Summary ==
Almost all labs instances suffered super high load / kernel stalls, to a point where they were basically dead. rsyncs running as a result of previous incident were basically eating up all io on labstore1001, causing knock on effects on all instances. Killing the rsyncs restored service, but ultimate cause is unknown since the rsync was running for hours without issue, and was resumed later and has completed in a few hours without issue. 
== Timeline ==
05:37 Shinken sends email of beta cluster being down
05:43 Shinken sends email of toollabs being down. Several users also report th is on IRC and labs-l around this time
06:00 Yuvi gets to a computer, starts investigating. Looks like several instances were 'freezing' frequently, and not responding at all
06:20 Yuvi assumes that the problem is with the underlying Virt* hosts load since that was the problem earlier, and fixes the one host with load (virt1002) by turning off a few VMs that were causing load there.. No help. Can not debug properly since ssh to most hosts I tried fails pretty badly, and am debugging blind. More vague re-arranging of chairs
07:08 _joe_ manages to ssh in, finds out that instances are stalling due to labstore issues with messages like
 [Tue Mar 31 07:07:16 2015] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
 [Tue Mar 31 07:07:29 2015] nfs: server labstore.svc.eqiad.wmnet OK
07:12: Labstore load abnormally high. 3 rsync processes running on a screen (one rsync job).
07:16: All rsyncs are killed, things start returning to normal
== Conclusions ==
labstore being unavailable / overloaded basically kills all of labs, and is a SPOF. Our NFS setup should be modernized and lots more alerting / monitoring set up. 
== Actionables ==
# Thorough monitoring of our NFS setup https://phabricator.wikimedia.org/T94606
# Spread knowledge of our NFS setup amongst our opsen better (Coren is doing a tech talk soon, I believe) 
# labstore1002 runs jessie, but switchover is not yet tested. Do switchover after a lot of testing, and re-install labstore1001 https://phabricator.wikimedia.org/T94607 and https://phabricator.wikimedia.org/T94609
Category:Incident documentation
Incidents/20150406-EventLogging

== Summary ==
Eventlogging was moved from vanadium.eqiad.wmnet to eventlog1001.eqiad.wmnet, the move did not introduced any outage per-se. The issue that caused the outage was the disk filling up on 
eventlog1001.eqiad.wmnet due to eventlogging normal functions.
We noticed that the service was logging every single event to the application logs (/var/log/eventlogging) and also to the upstart logs (/var/log/upstart). This had been happening on vanadium too but 
the partition to which upstart logs were stored was a lot bigger in vanadium and the big logs were unnoticed.
There were also two additional issues with alarming:
1. Analytics team was not receiving "disk full" alarms, only eventlogging process alarms.
2. A prior deployment of EL had broken graphite counts on hafnium that are also used for alarming
== Timeline ==
===Mon Apr 6 11:15:10 UTC 2015 ===
icinga alarms regarding eventlogging processes are raised, teams looks at issue on disk, creates space and re-starts. 
Disk fills short after again.
Fixing disk alarms so analytics team receives them:
:* gerrit:202048
Fixing eventlogging so application data (events) is not logged to /var/log/upstart
:*gerrit:202043
Fixing graphite counts in hafnium:
:* gerrit:202070
===Mon Apr 6 18:32 UTC 2015===
: After deployment of changeset: gerrit:202043 events are no longer logged to /var/log/upstart and logs are not growing as fast
== Conclusions ==
We can see no way to anticipate an issue like this one and all things consider alarming and fixing worked well so team was able to respond promptly.
== Actionables ==
Please note that while disk was full events were not logged properly (none of them) thus backfilling will be at best incomplete and analytics team will not be backfilling events for this outage.
* Fix alarms so analytics team gets disk full alarms.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150408-s2-SiteStatsInit

== Summary ==
At around 2015-04-08 12:25:34 UTC, S2 slaves encountered https://phabricator.wikimedia.org/T95426 (SiteStatsInit::edits and SiteStatsInit::articles query storm) and S2 wikis started returning errors. The slaves' event schedulers started killing slow queries after 5min and by 2015-04-08 12:35:34 things returned to normal.
== Conclusions ==
This happened a year or so ago and led to https://gerrit.wikimedia.org/r/#/c/114994/ . Something else has since broken allowing these to run as wikiuser from the web, and not a maintenance script or job.
== Actionables ==
* phabricator:T95426: Move SiteStats queries to vslow hosts.
Category:Incident documentation
Incidents/20150409-EventLogging

== Summary ==
EventLogging events were not written to the database during some time spans, starting on March 22nd, 2015.
Related Phabricator task: https://phabricator.wikimedia.org/T96082
The data loss affected all schema tables. The actual intervals are (UTC):
* 2015-03-22 18:50:00 to 2015-03-22 20:20:00*
* 2015-03-24 00:30:00 to 2015-03-24 01:50:00*
* 2015-03-24 20:50:00 to 2015-03-24 21:30:00*
* 2015-03-29 17:40:00 to 2015-03-29 19:00:00*
* 2015-04-01 21:10:00 to 2015-04-01 21:20:00*
* 2015-04-06 09:38:03 to 2015-04-06 11:31:23
* 2015-04-06 14:08:10 to 2015-04-06 14:54:57
* 2015-04-06 15:36:38 to 2015-04-06 15:52:16
* 2015-04-06 23:00:38 to 2015-04-07 01:23:56
* 2015-04-08 18:45:18 to 2015-04-08 21:20:36
* 2015-04-09 17:12:48 to 2015-04-09 18:49:48
* 2015-04-11 03:30:25 to 2015-04-11 05:20:30
* 2015-04-11 13:59:42 to 2015-04-11 15:32:42
* 2015-04-11 18:53:40 to 2015-04-11 20:11:02
* 2015-04-12 14:25:16 to 2015-04-12 15:52:38
* 2015-04-13 11:31:46 to 2015-04-13 12:41:49
* 2015-04-13 16:23:46 to 2015-04-13 17:49:34
* 2015-04-14 14:15:46 to 2015-04-14 16:16:06
* 2015-04-14 19:22:31 to 2015-04-14 21:27:15
* 2015-04-15 16:12:54 to 2015-04-15 16:43:47
* 2015-04-19 10:07:21 to 2015-04-19 12:36:40
* 2015-04-20 07:00:48 to 2015-04-19 10:00:39
* 2015-04-20 17:40:00 to 2015-04-20 20:19:00
* 2015-04-21 14:30:00 to 2015-04-21 16:59:00
* 2015-04-22 01:50:00 to 2015-04-22 03:49:00
* 2015-04-22 12:30:00 to 2015-04-22 13:49:00
* 2015-04-22 15:30:00 to 2015-04-22 16:39:00
* 2015-04-22 18:00:00 to 2015-04-22 18:59:00
* 2015-04-22 20:20:00 to 2015-04-22 21:19:00
(*) Approximate
The problem continues to persist as of April 20, 2015. Backfilling will be carried out during this week (April 20 - 27, 2015), at least for the intervals not marked with (*).
== Timeline ==
===Thursday Apr 9, 2015 ===
The problem is observed, studied and presented to the Analytics team.
===Friday Apr 10 - Friday Apr 17, 2015 ===
The issue is researched and scoped. Root cause is found.
===Monday Apr 20, 2015 ===
Backfilling starts.
== Conclusions ==
The rate in which events are being inserted to the database is slower that the rate in which events come to the EventLogging server. This overhead accumulates events inside a buffer in the EL consumer, which eventually gets too big in memory, and gets killed by the system. All the events that were in the buffer at the time of the process kill are not inserted in the database.
Some weeks ago, this problem happened in the DB layer. Changes were made that optimized the event insertion with success. However, in the last weeks EL throughput has increased significantly, which made the problem surface again, this time in the EL consumer layer. 
== Actionables ==
* Backfilling the data.
* Implementing a quick solution for the problem to cease.
* Implementing a true solution to the EL scaling problem.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150413-LabsNFS

== Summary ==
Labstore1001 became unresponsive to NFS because its block device subsystem was overloaded. A combination of factors caused a cascade failure resulting in complete resource starvation for NFS.
== Timeline ==
17:30 Marc noticed NFS is slow during a fairly large copy, thinks the copy might be the cause
17:34 Initial diagnostic on labstore1001 show NFS is starved out of disk bandwidth
17:41 labstore1001 kernel stuck on kworker and bdflush processes, starved for IO
17:50 increase IO priority of NFS process hoping to improve interactive performance
18:00 minor improvements visible, some NFS service restored, but not much
18:17 _joe_ joins investigation
18:20 one of the shelves is rebuilding raid6, suspected of being an issue but reducing bandwidth does not visibly help
18:29 stoping NFS and unmounting filesystems to cold start labstore1001
18:33 cold start of labstore1001, go into MD800 BIOS to check diagnostics
18:36 No failures in the hardware reported, proceeding with boot
18:44 NFS back up, 
18:53 NFS spotty at best, very high iowait still noticable with disk usage pegged
19:02 bblack joins investigation
19:33 Another cold start to do the bootstrap manually, trying to isolate the problem component
19:38 attempt to let the raid resync proceed, would take 20h at current (overly slow) rate
20:05 mark suggest tuning stripe_cache_size to increase rebuild speed. Increases efficiency tenfold.
20:08 noted how raid6 is bound to a single CPU in labstore1001's older kernel, no further improvement in raid6 speed possible
20:12 reduce rebuild speed to leave some IO bandwidth, restart NFS
20:22 NFS returns to reasonable working order, with some intermitent sluggishness
20:47 Most things return to working order, while Coren and Yuvi restore some services that did not survive the outage
21:12 All services back to normal, but iowait remains high
01:34 iowait on labstore1001 returns to normal patterns. Cause unknown as rebuild still in progress.
== Conclusions ==
There seems to be no single, isolable cause to the outage. Rather, a combination of factors seem to have resulted in the demand on disk bandwidth to exceed the capacity of the system to the point where cascading failure was reached. The kernel stripe_cache_size being set to the default (too small) kernel value amplified the drain on resources caused by the raid resync to the point where a buffer flush initiated by the kernel ended up starving all processes out of disk bandwidth. In addition, the older kernel (from Precise) has a bottleneck on raid6 checksum calculation being single-CPU bound that aggravated matters.
== Actionables ==
* Move to a more modern kernel as swiftly as practical https://phabricator.wikimedia.org/T94609
* make certain the stripe_cache_size setting is puppetized and applied at boot https://phabricator.wikimedia.org/T96045
* Formulate plans for getting off of raid6 for labs NFS storage. https://phabricator.wikimedia.org/T96063
* Formulate plans for reducing unnecessary NFS I/O by pushing projects to use local storage for heavy i/o traffic. https://phabricator.wikimedia.org/T96065
Category:Incident documentation
Incidents/20150428-EventLogging

== Summary ==
A change to the mediawiki-core api caused EventLogging schemas to be returned with "required": "" instead of "required": true. This caused a schema validation error and stopped all events from validating. While debugging, we lost about 2 hours of data. We can not backfill the lost data.
== Timeline ==
=== Tuesday, April 28, 20:52 ===
Noticed that EL was crashing and throwing an error on every event. After help from Bryan Davis and Kunal Mehta we found out this was due to a mediawiki-core change that had just been deployed. They started working on it with Unbreak Now priority (thanks guys).
=== Tuesday, April 28, 22:44 ===
This patch to core resolved the issue: https://gerrit.wikimedia.org/r/#/c/207297/ and this bug was filed to track the problem from the analytics point of view: https://phabricator.wikimedia.org/T97487.
=== Tuesday, April 28, 23:03 ===
Event Logging recovered in Icinga. All is well again.
== Conclusions ==
We can't backfill the lost data because we had to stop the processes that create the backup logs in order to debug. A fairly quick fix that's available right now is to start sending events to the kafka pipeline. This would give us a buffer and let us turn off the single point of failure Event Logging server to do maintenance, debugging, etc.
* Fix to API so EventLogging schemas are served as expected.
Category:Incident documentation
Category:EventLogging/Incident documentation