nyuuzyou/wikis huggingface dataset:
Incidents/20140910-BetaCluster

Summary from Jeremy B
superm401/mattflaschen noticed the beta bits 503s ~01h UTC. (#wikimedia-labs)
in between Matt and Krinkle, puppetmaster on deployment-salt killed by oom-killer:
> Sep 10 05:51:58 deployment-salt kernel: [8255978.329157] Out of memory: Kill
> process 8394 (puppet) score 371 or sacrifice child
> Sep 10 05:51:58 deployment-salt kernel: [8255978.330110] Killed process 8394
> (puppet) total-vm:1647904kB, anon-rss:1487172kB, file-rss:2416kB
I noticed it when Krinkle replied to Matt ~7 hours later (~08h UTC) and then I started investigating.
* looked at `varnishadm health.debug`. said both backends were sick but I misread it.
* looked at `varnishlog` and immediately saw the 404 responses for health checks
* did my own curl tests against both backends. =404
* logged into one backend, looked at the apache conf (arbitrary file in sites-enabled) for one of them and saw srv/mediawiki/... as docroot. the path in conf did not exist on fileystem
sometime around now, tried steps from bug 70597. for some steps I didn't get an error message but also couldn't tell for sure if it worked. I still don't know if there's steps in there that require advanced jenkins privs. (e.g. to mark offline/online)
also, looked at puppet logs/tried puppet run/investigated puppetmaster state. normally might have hesitated about some of that on a new+unfamiliar system but given I knew the change to /srv/mediawiki was done recently and the boxes were so completely broken i figured it couldn't do too much damage.
[10:14:53] <jeremyb> !log deployment-bastion killed puppet lock (file)
[10:15:09] <jeremyb> !log deployment-salt started puppetmaster && puppet run
[10:15:27] <jeremyb> !log deployment-mediawiki0[12] both had good puppet runs
[10:16:34] <jeremyb> !log deployment-salt had an oom-kill recently.
and some box (maybe master, maybe client?) had a disk fill up
^^ same one quoted above
[10:17:07] <jeremyb> !log deployment-bastion good puppet run
[10:20:02] <jeremyb> !log deployment-bastion /var at 97%, freed up ~500MB. apt-get clean && rm -rv /var/log/account/pacct*
addressed some puppet run problems (not sure if these were before or after the !logs above). some quick hacks to get stuff working (e.g. kill the duplicate resource definition) also, a change which I thought was right but then it turns out we're also moving off of that path *too*.
fixed a few other things like deployment-jobrunner01 was set in ldap (puppetVar via wikitech interface) to use a non-existant hostname as trebuchet master.
* deployment_server_override=deployment-scap.eqiad.wmflabs.
* deployment-videoscaler01.eqiad.wmflabs still has that (old?) setting
but i'm not worrying about it tonight. audited the rest of the project, they all have the same trebuchet master
== Conclusions ==
What weakness did we learn about and how can we address them?
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Bugzilla bugs or RT tickets linked for every step.
put them here
Incidents/20140910-swift-syslog

== Summary ==
swift frontends were maxed out on CPU following an rsyslog configuration
change, impacting the image scaler cluster and regular swift traffic
== Timeline ==
* 2014-09-10T08:52 https://gerrit.wikimedia.org/r/#/c/159348/ is submitted and merged, changing rsyslog and swift proxy configuration
* 2014-09-10T08:58 first alarm, HTTP timeout on ms-fe1002
* 2014-09-10T08:59 impact seen on image scalers, LVS alarm for rendering.svc.eqiad.wmnet
* 2014-09-10T09:01 HTTP 5xx alarm
* 2014-09-10T09:11 rolling restart of swift frontends
* 2014-09-10T09:12 LVS recover for ms-fe.eqiad.wmnet
* 2014-09-10T09:14 LVS recover for rendering.svc.eqiad.wmnet
== Conclusions ==
This was an instance of Incident_documentation/20131205-Swift in which
swift busyloops when the syslog socket goes away. The issue was thought to be
fixed by latest swift upstream and confirmed during testing. The testing has
proven to not replicate the exact conditions for reoccurrence however, as this
incident demonstrates. Extra care should be put when deploying rsyslog
configuration changes that restart rsyslog as a side effect.
Related graphs:
* ganglia CPU http://ganglia.wikimedia.org/latest/?r=custom&cs=9%2F10%2F2014+8%3A51&ce=9%2F10%2F2014+9%3A20&m=cpu_report&s=by+name&c=Swift+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&hide-hf=false&sh=1&z=small&hc=4
* imagescaler apache workers sending replies http://ganglia.wikimedia.org/latest/?r=custom&cs=9%2F10%2F2014+8%3A51&ce=9%2F10%2F2014+9%3A20&m=ap_sending_reply&s=by+name&c=Image+scalers+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&hide-hf=false&sh=1&z=small&hc=4
== Actionables ==
See related actions for Incident_documentation/20131205-Swift
Category:Incident documentation
Incidents/20140919-s1

== Summary ==
At approx 2014-09-19 08:50:00 enwiki experienced a site outage. The apparent order of events was:
1. Three edits made to a template:
https://en.wikipedia.org/w/index.php?title=Template:Redirect_template&action=history
2. Jobrunner write activity[1] from wikiadmin user on enwiki master increased substantially, with cirrus in the spotlight:
https://logstash.wikimedia.org/#/dashboard/elasticsearch/LinksUpdate%20issues
Binlog showed substantial LinksUpdate hits (it is often in the top 10, but blends in with similar numbers to other traffic):
 905809 LinksUpdate::incrTableUpdate
 359301 LinksUpdate::updateLinksTimestamp
 9372 Title::invalidateCache
 8728 FlaggableWikiPage::clearStableVersion
 8099 User::invalidateCache
 5753 Revision::insertOn
 4692 ArticleCompileProcessor::save
 4607 `heartbeat`.`heartbeat`
 4404 FlaggedRevs::clearStableOnlyDeps
 3261 CheckUserHooks::updateCheckUserData
 3257 SiteStatsUpdate::tryDBUpdateInternal
 3185 RecentChange::save
The job activity occurred in waves with periods of very heavy writes, then minutes of nothing.
3. enwiki slaves started to experience intermittent replication lag. The main offender was:
 DELETE /* LinksUpdate::incrTableUpdate
4. Surges of wikiuser DB connections to slaves began appearing after each write surge above in #2. These hit max_connections on all slaves simultaneously, and apaches went critical. Note that there were no slow queries involved; just an order of magnitude more connections and queries than normal.
5. Significant numbers of wikiadmin connections sat in SELECT MASTER_POS_WAIT due to #3, which reduced the available connections for #4.
6. We killed masses of wikiadmin and wikiuser sleeping connections to make way for new ones.
7. We stopped jobrunners.
8. Things recovered.
== Observations and questions: ==
1. Batching the LinksUpdate DELETE and UPDATE queries would help with replag.
2. The storm of wikiuser traffic after the jobs was due to cache invalidation and presumably a lot of duplicated effort? Could that be mitigated in another layer above the DBs?
3. Can we throttle jobrunners more, or make them smarter in these situations?
== Actionables ==
* https://gerrit.wikimedia.org/r/161473
* https://gerrit.wikimedia.org/r/161500
* https://gerrit.wikimedia.org/r/161577
* https://gerrit.wikimedia.org/r/161603
* https://gerrit.wikimedia.org/r/161618
* https://gerrit.wikimedia.org/r/161749
</onlyinclude>
Category:Incident documentation
Incidents/20141027-CirrusSearch

== Summary ==
=== First Time ===
CirrusSearch fell over for a few hours. We mitigated the problem by falling back to lsearchd during most of the outage. It isn't clear what caused the outage at this point but we've added instrumentation so if whatever caused the outage happens again we'll learn more.
=== Second Time ===
A few days later the same thing happened and we were able to learn more
== Timeline ==
* 2014-10-27 12:26:43 utc - Queries started to fail on the Elasticsearch side.
* 2014-10-27 ~13:00 utc - logs in for the day and learns about the issue. The cause of the problem is determined to be huge spikes in Java heap usage to the point where the JVM can't free enough heap to take any actions including dumping the heap for analysis.
* 2014-10-27 13:47 utc - We fall back to lsearchd after determining that we don't know what is causing the high heap usage.
* 2014-10-27 14:54 utc - After trying for a while to get heap dumps for analysis without success we give up and reboot all the frozen machines and Elasticsearch immediately becomes healthy again. Nik goes to get coffee.
* 2014-10-27 15:32 utc - After finding out that falling back to lsearchd turned off the Geo extension we reenable cirrus as a secondary search engine to fix geo.
* 2014-10-27 ~17:00 utc - Elasticsearch cluster locks up again with the same symptoms. We spend more time trying to get heap dumps. We're still on lsearchd at this point so we have time.
* 2014-10-27 ~18:00 utc - We make configuration changes to Elasticsearch such that if the crash happens again again we'll get a heap dump we can use and we restart Elasticsearch.
* 2014-10-28 1:01 utc - After Elasticsearch cluster has stabilized bring reenable Cirrus everywhere.
* 2014-10-31 10:10 utc - Elasticsearch starts failing again with totally full heap
* 2014-10-31 11:25 utc - Elasticsearch fully healed again 
== Conclusions ==
The outage is caused by some nasty regex queries that Lucene tries to compile using a O(2^n) algorithm. Not cool dude. 
== Actionables ==
* Add JVM settings to dump heap on out of memory. If the problem happens again we'll be able to figure out what it is.
* Temporarily disable regex queries.
* Add circuit breaker to prevent heap exhaustion on regex compilation.
* Deploy circuit breaker and reenable regex queries.
Category:Incident documentation
Incidents/20141106-Labstore

== Summary ==
The primary storage server for Labs (labstore1001.eqiad.wmnet) went down, preventing access to shared Labs filesystem and therefore stalling user logins and many processes in Labs.
== Timeline ==
* 2014-11-06 16:22 UTC - Icinga reports labstore1001 goes down, pretty much simultaneously with user reports of issues.
* 2014-11-06 16:36 UTC - Coren begins preleminary diagnostic; box has gone down hard and requires power cycle
* 2014-11-06 16:43 UTC - Diagnostics show two of the disk shelves as invisible from the server; suspect hardware issue
* 2014-11-06 16:54 UTC - cmjohnson on site, begins to help diagnostics by kicking the shelves
* 2014-11-06 18:53 UTC - after a lot of hardware swaps, finally isolate the fault to cabling
* 2014-11-06 19:08 UTC - swapping cabling made shelves visible again; restarting labstore1001 in stages to watch for corruption
* 2014-11-06 19:26 UTC - all is now well, service goes back up
== Conclusions ==
Having the hardware mirrored turns out to have been a wise move (because it made fault isolation possible without having to involve vendor support); and once we had determined the actual disk enclosures worked properly we had the option to bring up the backup server - which we would have done within the next half-hour or so had we not gotten a reasonable probability of resolution. During the diagnostic steps however, we realized there remains one piece of hardware that does not have an available spare at hand and which would have prevented resolution: a disk enclosure. They have redundant power supplies, redundant controllers and redundant data paths - making complete failure improbable - but not having a shelf we could swap the drives to means that if the failure was located there we would have had to wait for vendor replacement before being able to restore service.
It also appears that Icinga alerts of labstore1001 did not lead to opsen being paged; this did not have consequences this time because the failure occured during normal North-American working hours and staff were at hand but needs to to be verified and fixed.
== Actionables ==
https://phabricator.wikimedia.org/project/view/893/
* Ensure that opsen are paged on failure of labstore1001's NFS service
* Order (and rack?) a spare enclosure for labstore1001 to allow migration of the disks away from a failed shelf. (RT:8827)
Category:Incident documentation
Incidents/20141113-EventLogging

== Summary ==
Restarting consumers after temporary switch of m2-master CNAME caused ~20 minute outage of writing events to the database between 2014-11-13T00:59 -- 2014-11-13T01:21.
(The data did not get lost, as the data is available in log files.)
== Timeline ==
;2014-11-13T00:34
: Change got merged and updated m2-master to point to dbproxy1002 to test db proxying.
;2014-11-13T00:59
: EventLogging's mysql-m2-master consumer got restarted by hand to pick up the above dns change.
: The consumer failed to connect to m2-master (which now pointed to dbproxy1002).
: So from this time on, no EventLogging events could get written to the database.
: (But the other consumers (like the plain log file writer) continued to work as expected.)
;2014-11-13T01:19
: The above dns change got reverted by change .
;2014-11-13T01:21
: EventLogging's mysql-m2-master got restarted by hand to pick up the above dns change.
: The consumer could again connect to m2-master (which now pointed to db1020 again).
: EventLogging events started to get written again to the database.
== Conclusions ==
* The fact that EventLogging is writing data synchronously and not buffering up database writes (hence cannot handle database connection issues nicely) is not known visible enough throughout the foundation.
* Regardless of the testing done in labs, production firewalls get in the way.
== Actionables ==
* Make sure that log files exist for the affected period, so we can backfill.
: Logs exist (on vanadium), not yet on stat1002, or stat1003 and look good.
* Backfill the data.
* Make sure vanadium can connect to dbproxy1002 (RT8863).
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20141114-EventLogging

== Summary ==
Event volume grew beyond what EventLogging's database writer could handle, and not all events could get written to the database between 2014-11-14T01:00 and 2014-11-21T21:00
== Timeline ==
;2014-11-14T01:00
:With the lightning deploy at that hour, changes got merged that increase the total count of messages from ~140 msg/s to ~220 msg/s during busy hours.
:From now on, during hours with higher traffic (~10:00--24:00), the EventLogging database writer failed to bring all events to the database. Extrapolating from a few high volume schemas, it seems that during peak hours only about 70% of the events made it to the database.
:Throughput monitoring on the EventLogging processors did not go off as it was configured for the event validation bottle-neck (which can handle more messages than the database bottle-neck).
;2014-11-19T11:41
:Gilles sent notice to the analytics mailing list about a suspicios decrease in MediaViewer actions.
;2014-11-20T21:00
:Ori deployed a fix for EventLogging's database writer.
== Conclusions ==
* Having multiple EventLogging issues at the same time does not help when trying to find root causes :-(
* Teams are not to trust when asking whether or not an increase in numbers is expected. We need to be able to walk the deployment lists on our own and find out what code the teams deployed when and what numbers they should expect. This is frustrating, as it explodes the code base we need to understand and be able to cover.
* Monitoring throughput only at EventLogging's “front-end” processors is not sufficient. We need monitoring that checks whether all events could get written to the database during peak hours.
* We need to re-evaluate the monitoring thresholds for EventLogging's throughput monitoring. They are tuned to the volume that event validation can take. But here the bottle-neck of the database writer became relevant before the event validation bottle-neck became relevant.
== Actionables ==
* Make sure that log files exist for the affected period, so we can backfill.
* Backfill the data.
* Implement monitoring that makes sure that the expected volume of events end up in the database.
* Re-evaluate the throughput monitoring's thresholds.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20141118-EventLogging

== Summary ==
Accidentally deployed EventLogging code (for which the database writer failed after some time) rendered EventLogging's writing of events to the database unreliably between 2014-11-18T00:59 -- 2014-11-13T01:21.
== Timeline ==
;2014-11-18T02:10
:Reviewed but untested EventLogging code got deployed unintentionally.
;2014-11-18T02:12
:EventLogging's database writer failed.
;2014-11-18T02:37
:EventLogging's database writer recovered. Events got written to the database again.
;2014-11-18T07:26
:EventLogging's database writer failed.
;2014-11-18T11:43
:EventLogging's database writer recovered. Events got written to the database again.
;2014-11-18T13:44
:EventLogging's database writer failed.
;2014-11-18T15:02
:EventLogging's database writer recovered. Events got written to the database again.
;2014-11-18T18:16
:EventLogging's database writer failed.
;2014-11-18T19:48
:Ryan sent a notice to the analytics list about EventLogging events do not showing up in the database.
;2014-11-18T21:05
:EventLogging's database writer recovered. Events got written to the database again.
;2014-11-18T23:19
:EventLogging's database writer failed.
;2014-11-18T23:55
:QChris saw Ryan's message to the mailing list.
;2014-11-19T00:07
:EventLogging's database writer got restarted manually. Events got written to the database again.
;2014-11-19T00:36
:Ori deployed a working EventLogging version.
== Conclusions ==
* EventLogging deployments are anonymous. But as (regardless of who deployed) only an existing, reviewed commit from gerrit and not rogue code got deployed, it seems the deployment happened accidentally by trusted people, and was not ill-intentioned. Also, this is the first time it happened for EventLogging. So no need to take measures against it.
* Monitoring of EventLogging processes is not sufficient. We need monitoring that checks if events actually end up in the database. That's on our to-do list anyways.
== Actionables ==
* Make sure that log files exist for the affected period, so we can backfill.
* Backfill the data.
* Implement monitoring that makes sure that the events end up in the database.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20141118-m2

== Summary ==
At 2014-11-18 14:42 m2-master (db1020) mysqld process aborted (sig 6) after an InnoDB assertion failure during a long semaphore wait. Same as https://mariadb.atlassian.net/browse/MDEV-6440 but fairly unhelpful as we didn't get a stack trace either.
The affected services were:
* gerrit
* eventlogging
* otrs
* reviewdb
* scholarships
The m2-slave (db1046) showed replication was in sync -- we use semi-synchronous replication plugin -- so the simplest, safest, and fastest response was to fail over to the slave and leave the master alone for investigation. My worry is that SIGABRT from InnoDB often indicates some sort of corruption somewhere in the stack, either directly or indirectly. I would want to reload the data regardless.
Total downtime was 13 minutes.
Amusingly, dbproxy1002 correctly went though fail over motions by itself, though it was not yet handling m2 traffic. So there's that.
== Timeline ==
* ~14:42 m2-master aborted
* ~14:55 m2-slave promoted manually
== Conclusions ==
* The dbproxy1002 is needed.
== Actionables ==
* Get dbproxy1002 rolled out.
* Identify the upstream bug, and/or arrange to collect a stack trace or core file. Complicated in that MariaDB 10.0.15 has been released and may mean we never see this again.
Category:Incident documentation
Incidents/20141122-m2

== Summary ==
db1046 (m2-master) encountered a thread pool lockup of some sort and started denying new connections:
141122 20:10:53 [ERROR] Threadpool could not create additional thread to handle queries, because the number of allowed threads was reached.
141122 20:10:53 [Note] Threadpool has been blocked for 30 seconds.
Existing persistent connections for otrs and eventlogging were apparently still working, but gerrit could not connect. It wasn't immediately obvious which thread was doing the blocking, and a gdb backtrace suggested that a low-level buffer pool mutex deadlock may have occurred, so I opted to restart mysqld. That returned things to normal.
Further investigation needed. Ori wondered on IRC if the recent eventlogging batching changes were related; unlikely IMO, since EL isn't doing anything more unusual than batching inserts. Most likely this is an upstream bug.
== Timeline ==
* ~20:10 thread pool lockup
* ~20:30 mysqld restart 
== Actionables ==
* Upgrade to MariaDB 10.0.15.
Category:Incident documentation
Incidents/20141125-EventLogging

== Summary ==
EventLogging's database writer service failed to write events to the database between ~2014-11-25T03:09 and 2014-11-26T00:03.
== Timeline ==
;2014-11-25T03:09
:One of the threads of EventLogging's database writer died.
:The other threads of EventLogging's database writer did not exit but stayed alive. Hence, the whole process (from upstart perspective) was still up and running. However, the process was not able to bring events to the database.
;2014-11-25T22:41
:Deskana came into the analytics channel and reported issues with events for a new schema not showing up in the database.
;2014-11-26T00:03
:EventLogging's database writer was restarted manually, and events get written to the database again.
== Conclusions ==
* The many meetings/discussions around recent EventLoggings issues ground us to a halt. We had a fix for the issue in gerrit (since 2014-11-23, before the issue happened), but the meetings/discussions around last week's EventLogging outages consumed too much time, and the fix did not get reviewed/deployed before the thread synchronization issue struck us.
* Monitoring does not buy us anything, if we cannot grab the time to work on the alerts. We had some internal monitoring alert about the issue twice before Deskana escalated with us. But with the backlog of alerts from previous week, we did not yet get to them.
== Actionables ==
* Make sure that log files exist for the affected period, so we can backfill.
* Backfill the data.
* Implement monitoring that makes sure that the expected volume of events end up in the database.
* Get the thread synchronization fix reviewed and deployed
Category:Incident documentation
Category:EventLogging/Incident documentation