nyuuzyou/wikis huggingface dataset:
Incidents/20140607-Elasticsearch

== Summary ==
Search requests powered by CirrusSearch slowed down and sometimes reported "temporary errors". Cirrus provides search for most wikis but not most large wikipedias and commons. It does provide search for Italian wikipedia, Catalan wikipedia, all wiktionaries, wikisources, wikivoyages, etc. 
== Timeline ==
2014-06-06 ~1900 UTC (Friday afternoon (not a good start to the story)) we added elastic1017 to the Elasticsearch cluster - its an old geo box. It immediately came under reasonably high io wait which is somewhat expected because it was restoring shards to itself. I didn't think it was a problem.
<Intervening time> As shards trasferred to it it started serving requests more slowly.
2014-06-07 ~1930 UTC I received a voicemail about searches complaining of "temporary problems".
2014-06-07 ~2000 UTC I got home and discovered that elastic1017 was thrashing its disks. I shut down some long running processes which didn't help. Then I removed the node from the Elasticsearch cluster. The slow queries immediately stopped.
Sorry for the inexact times!
== Conclusions ==
* Cirrus was using SSDs!
** Why didn't we know? That's not clear. Obviously some communication breakdown.
== Actionables ==
* - Don't ever deploy stuff on Friday afternoon.
* - Cirrus was using SSDs!
** Why didn't we know? That's not clear. Obviously some communication breakdown.
** Short term: we're going to continue using SSDs.
** Long term: we don't actually look like we're doing much IO at once at steady state. When shards move around we obviously need more. We might be able to avoid needing expensive ssds. OTOH, it might not be worth the time. Not sure.
* - We're going to have to file a bug upstream to do something about nodes that are *broken* like this one. We've had this issue before when we built a node from puppet and didn't set the memory correctly and it ran out of heap and started running really really slow. The timeouts we added then helped search limp along with the broken node, but it'd be nice to do something automatically to the node that is obviously sick.
** filed
Category:Incident documentation
Incidents/20140608-Kafka

== Summary ==
A single disk on one of the two analytics Kafka brokers failed. The second broker took over as leader for all partitions. Since we currently send more data to Kafka than a single broker can handle (bits, upload, text), messages were dropped.
== Timeline ==
At 2014-06-07T23:37:04, kafka on analytics1021 shut down with a storage exception due to 'Read-only file system'. syslog at this time shows
 Jun 7 23:37:04 analytics1021 kernel: [10635272.839586] lost page write due to I/O error on sdf1
Sean was the first to respond and notice. The disk was not fixable remotely, so he disabled puppet and shut down Kafka. As far as I know, no pages for this were sent. Otto or Gage couldn't have fixed the problem remotely anyway, but others should not have been left wondering how to react.
Monday morning on June 9th, Otto checked email and learned of the problem. Analytics team then met and deliberated on how to proceed.
It has been known that a single broker was not enough to handle the data we are currently sending through it. Ever since we added text back in May, a single broker has not been enough. There are plans in the work to provision more Kafka brokers, but these have not yet come through. We plan to use some of the existing Hadoop DataNodes as additional Kafka brokers, and to order more new replacement DataNodes. We had planned on waiting for these new DataNodes to come in before repurposing existent DataNodes as brokers, but this past weekend's downtime has pushed that schedule forward.
We plan to take DataNodes from the existing Hadoop cluster ASAP to provision as new Kafka Brokers. This will give us some time to do more failover and load testing with more brokers and more traffic, before we start relying on this service more heavily.
== Actionables ==
* - Decommission 2 or 3 Hadoop DataNodes and provision as Kafka Brokers.
* - Create partman recipe for new Kafka brokers. Too complicated for partman :/
** WIP: https://gerrit.wikimedia.org/r/#/c/138451/
* - Replace sdf on analytics1021:
** https://rt.wikimedia.org/Ticket/Display.html?id=7647
* - failover and load tests of Kafka Brokers with all varnish log traffic.
* - fix or replace this alert: https://gerrit.wikimedia.org/r/#/c/138302/
** Faidon disabled this as it was flapping too much during the downtime.
*** https://rt.wikimedia.org/Ticket/Display.html?id=7828
*** Done here: https://gerrit.wikimedia.org/r/#/c/150010/
* - fix (?) pages for Kafka services.
** https://rt.wikimedia.org/Ticket/Display.html?id=7829
* - See if we can tune a single broker to handle more traffic.
** I.e. why is one broker not able to handle all traffic? Just curious!
Category:Incident documentation
Incidents/20140613-Videoscalers

== Summary ==
Due to a change to the way the job-loop script works, that was not syncronized on the videoscalers, processing of videos stopped effectively in the morning of Friday June 13th 2014. Completing the change on the video scalers puppet manifests resolved the issue and practically restarted the jobs.
== Timeline ==
On Friday June 13th, 10:26 UTC a series of changes to the job-loop bash script that is in charge of running our videoscalers has been merged in the repository (https://gerrit.wikimedia.org/r/#/c/139208/) As this change forgot to remove the extra unused argument from the videoscalers, processing of videos effectively stopped at that point (any change to jobloop triggers a restart of the service in puppet).
Brian Wolff was notified that some uploaded video on commons were not being converted, so he opened an RT ticket (#7693) on Saturday June 14th, 22:17 UTC. I've seen the ticket on Sunday afternoon, and upon investigation (with the notable help of Reedy), we were able to pin the problem to this change. As soon as we've merged https://gerrit.wikimedia.org/r/#/c/139682/ the video scalers instantly began converting videos again. So the incident closed at 14:26 UTC on Sunday June 15th, 2014.
== Conclusions ==
While this is a very honest error in a merge, which could happen again and again, we've got some food for thoughts:
* Job-loop does *not* log or control execution of the jobs it launches. Worse, there is no log of the program running anywhere. The only way to understand if the jobs were running correctly was basically grepping on fluorine for output. The non-working state was completely silent
* In general jobs are poorly monitored and poorly monitorable
So while these kind of errors are pretty normal and could keep happening, we should at least be notified when we do so. This, together with the need for a better job control/logging, make job-loop the weak link here. 
== Actionables ==
* - Add monitoring for individual job types on single machines.
* - Make job-loop aware of the status of launched jobs, or rethink and rewrite it completely. 
** https://gerrit.wikimedia.org/r/#/c/144612/ is going live before HHVM (hopefully)
Category:Incident documentation
Incidents/20140616-LuceneSearch

== Summary ==
Lsearchd-based search is unstable, and seems to have issues from time to time that fall in the "known problem" field, so the long and short term solution seems to be to migrate to ElasticSearch.
== Timeline ==
On monday Juner 16th, 08:31 UTC we got an alarm for the search pool 4 that was practically down. Upon inspection, both members of the pool were basically offline, as reported correctly by pybal. From logs, it appeared as some deadlock condition was met. Solution was simply to restart one server at a time, which resulted in a ~ 30 mins recovery cycle, which lasted around 09:30 UTC. Some users of hewiki reported they got empty search results and/or errors when searching during this time lapse.
== Conclusions ==
The problem with lsearchd-based search systems are multiple:
* lsearchd is unstable and has frequent crashes, that are usually slow to recover from.
* doc is scarce and outdated
* Pybal is probably the only thing monitoring the real health of these systems
== Actionables ==
* - Move search to ElasticSearch, decom lsearchd.
Category:Incident documentation
Incidents/20140618-Wikitech

== Summary ==
Wikitech failed for about 10 minutes. The proximate cause was fixing puppet; the ultimate cause was a bug in Apache puppet refactoring.
== Timeline ==
* Andrew B notices that puppet runs are failing on wikitech, fixes duplicate definition problem
* Puppet is fixed, thus breaking Apache
* Andrew B fixes hotfixes apache
* Proper apache puppet fix is merged
Rough outline:
Earlier in afternoon I discovered that puppet was failing to run on virt1000 (aka wikitech) along with the other puppet servers (e.g. palladium). This was due to a duplicate package definition (libapache2-mod-passenger) that crept in somewhere around the 'Kill apache_module' patch [1].
Despite this being a catalog compilation failure, icinga failed to notify us about puppet failures.
I fixed the duplicate package [2] and forced a puppet update on palladium. Everything went fine, puppet ran cleanly and palladium's clients continued happily. So, I forced a puppet run on virt1000, which also ran cleanly. Alas, during the intervening puppet outage, virt1000 had accumulated a backlog of unapplied patches. Another Apache refactor patch [3] added a new (spurious) sites-enabled file to wikitech with some settings that conflicted with the proper wikitech virthost. When apache2 restarted, it just chose one, and chose poorly, such that the only functional vhost on wikitech redirected to virt1000. This worked poorly.
I fixed the accidental renaming with a couple of hurried patches [4][5]. After a bit of fumbling, wikitech is happy again.
[1] https://gerrit.wikimedia.org/r/#/c/140211/
[2] https://gerrit.wikimedia.org/r/#/c/140571/
[3] https://gerrit.wikimedia.org/r/#/c/140218/
[4] https://gerrit.wikimedia.org/r/140576
[5] https://gerrit.wikimedia.org/r/140582 
== Conclusions ==
What weakness did we learn about and how can we address them?
# We need to improve puppet monitoring so that puppet breakages produce some sort of notification.
== Proposals ==
# Add a reporter to our puppetmaster that registers an icinga check when a puppet run has errors.
== Actionables ==
* - Add a puppet success check to icinga
** RT 7842
Category:Incident documentation
Incidents/20140622-es1006

== Summary ==
There was a 50-second flash of 5xx responses which corresponds to a spike of "Too many connections" errors for es1006 in dberror.log. 
== Timeline ==
 First error: Sun Jun 22 8:01:37 UTC 2014
 Last error: Sun Jun 22 8:02:27 UTC 2014
Mean CPU utilization since the 12th is up around 90% compared to the previous ten day period:
http://graphite.wikimedia.org/render/?target=servers.es1006.cpu.total.user.value&from=00%3A00_20140601&until=23%3A59_20140622&width=600&height=300
In fact it is a general load jump for external storage that has been causing similar glitches for some days. There is a corresponding jump also starting on the 12th on the S5 slaves (dewiki, wikidatawiki). None of the other shards show the pattern.
During IRC discussion a probable spike in Wikidata traffic was identified; mostly Wikibase\Lib\Store\WikiPageEntityLookup::selectRevisionRow which would also hit ES. Aude and Hoo investigated and found a latent Wikidata caching bug.
== Conclusions ==
Traffic increased on ES and S5. Probable cause was a latent Wikidata bug.
== Actionables ==
* An additional S5 slave has been deployed.
* DB traffic sampling has been deployed to S5.
* Aude and Hoo deployed https://gerrit.wikimedia.org/r/#/c/141997/
Category:Incident documentation
Incidents/20140622-imagescaler

== Summary ==
Starting on June 22th UTC morning the imagescaler fleet ran out of available workers due to increased load. Users requesting thumbnails that needed scaling might have incurred in timeouts/errors.
== Timeline ==
* 2014-06-22 08:58 operations start getting alarms for rendering.svc.eqiad not responding
* 2014-06-22 09:30 initial investigation by , increased load on imagescalers and more objects making their way to swift. Looks like genuine load.
* 2014-06-22 10:30 a bulk image upload to commonswiki is suspected as the root cause and evidence found and the user is asked to stop
* 2014-06-22 10:30 a bulk image upload to commonswiki is suspected as the root cause and evidence found
* 2014-06-22 17:40 a new alarm for rendering.svc.eqiad is raised and investigation continues, no abnormal upload counts are seen
* 2014-06-22 21:00 points out that thumbnails are being generated with unusual sizes and a user-agent is suspected (WikiLinks)
* 2014-06-23 10:00 notes that WikiLinks has been offered free the previous day, above normal load on the image scalers is seen but not at critical levels
== Conclusions ==
The imagescaler cluster can sustain normal churn of thumbnails but
not spikes while generating uncommon thumbnail sizes. Swift frontend bandwidth
was nearly at capacity due to the increased load.
== Actionables ==
* - Upgrade swift frontend bandwidth to 10G
** RT 7728
* - Number of uploads should be a metric in graphite
** 
* - Increase the number of imagescaler workers once swift bandwith can withstand it
** RT 7823
Category:Incident documentation
Incidents/20140625-CirrusSearch

== Summary ==
All wikis using CirrusSearch lost search for 30 minutes. Users opted into the CirrusSearch BetaFeature lost it for an additional 20 minutes.
== Timeline ==
* 2014-06-24 19:39 attempted to sync out a config change for Cirrus but instead just pulled the configuration change to the staging server and didn't sync anything.
* 2014-06-25 16:16 sync a different config change for Cirrus - this one makes CirrusSearch the primary search backend on all but 11 wikis. This half synced out the config change by which caused search to totally break.
* 2014-06-25 16:18 and realize that something is wrong with the config change. Independently of the broken sync from 2014-06-24 this change caused a small load spike while search caches warmed and caused some searches to temporarily fail. We monitor the situation thinking that cache will warm and everything will be OK.
* 2014-06-25 16:21 Everything is not yet ok even though it looks like the caches are plenty warm.
* 2014-06-25 16:23 rolls back the config change from 2014-06-25 16:16 thinking something is wrong with the change. It doesn't help. We do more investigating.
* 2014-06-25 16:42 fails all wikis back over to lsearchd. Only users that have opted into the CirrusSearch BetaFeature still have broken search.
* 2014-06-25 17:06 finds the half synced configuration and performs another configuration sync, this time pushing all the remaining half synced changes. CirrusSearch is now fixed.
* 2014-06-25 17:41 reenables CirrusSearch as the primary search backend for all the wikis that it was serving before the 2014-06-25 16:16 configuration change.
== Conclusions ==
* Don't forget any of the steps in the deployment process. Recheck after the deploy to verify that it made it. Common sense but when you deploy all the time you get cavalier about it.
== Actionables ==
* - Be more careful next time.
* - If sync-dir/sync-file/scap don't sync any files then we need to log something about it because its weird. Warn the operator that the sync they just performed was a noop.
** 
* - Add automatic cache warming to CirrusSearch to prevent load spikes when loading cold caches.
** 
* - Improve CirrusSearch error handling, it's very broken.
** 
Category:Incident documentation
Incidents/20140714-Lists

== Summary ==
Due (presumably) to a problem with a puppet change and incompatibilities between puppet versions, the server hosting lists.wikimedia.org crashed. This resulted in a corruption of the mailman files, which required restoring from the backup. This, again, caused the root partition of the server to exhaust all inodes and to become unable to write to disk. As a result, mailing lists were unavailable for about 16 hours.
== Timeline ==
=== 07/14/2014 === 
* 14:26 The change is merged into the puppet repository
* 14:29 Amongst a lot of puppet failures, nescio goes down. Later inspection shows it died while applying the puppet catalog
* 14:30 Sodium goes down as well; later inspection will show it died while applying the puppet catalog. Sodium and nescio both still run puppet 2.7 as they're on an elder distribution. This resulted in the generation of a file called /etc/init/.conf and possibly in an attempted execution of the script via upstart. We are still not 100% sure this is the reason of the crash, though.
* 14:33 Sodium is rebooted, all checks on icinga are green.
* 16:57 Someone notices lists.wikimedia.org is not working and reports that to us
* 17:05 Mailman is started on sodium manually (see Server_admin_log). Upon restart, it is found that the crash corrupted the config files for a few lists. The configs are recovered from the backups.
* 19:03 Files have been restored from backup, mailman is started again and starts to serve mailing lists again.
* 19:39 Due to the large amount of files recovered from backup and some mail lying in the input queue, the root filesystem of sodium exhausted its inodes, thus practically preventing any correct functionality for sodium
* 23:29 Puppet failures trigger an icinga alarm for puppet failing runs
=== 07/15/2014 ===
* 06:55 One member of the ops team is notified in IRC that mailman is still not sending emails.
* 07:11 After a first inspection and some cleanup, email start being sent again out. Contrary to what I originally reported, no email accepted by the system should have been lost, but the queue of messages to send is large, and as soon as the mailserver starts accepting mail again, most of the messages sent in the last few hours get submitted again, thus creating a large queue.
* 07:22 After removing the stale backup files, we decide to perform one more restart of mailman
* 08:50 Before realizing the queue of outgoing messages includes more than 10K messages, we try to understand why messages apparently continue not to flow through.
* 13:30 As of now, 2600 outgoing emails are still in the queue, which is slowly getting to zero.
== Conclusions ==
This episode showed a few things we should improve our monitoring of both lists and disk, and that we're better off with all servers on an uniform platform (distribution/puppet/etc). The impact of this was that internally managed mailing lists were down for about 16 hours.
== Actionables ==
* - Monitor mailman processes, queue sizes, etc...
** RT 7878
* - Alarm on disk inode excessive usage, and not only on space use
** RT 7879
Category:Incident documentation
Incidents/20140715 CirrusSearch

== Summary ==
All wikis using CirrusSearch experienced a significant increase in search failures. I haven't dug into exactly how many, but thousands an hour.
== Timeline ==
Sorry about not being as accurate as I normally am. I don't have great times on this one:
We got a bit over aggressive about pushing Cirrus as the primary
search backend for bigger wikis and pushed ourselves over the edge but in
slow motion. Things started breaking down during Europe's peak time on
Tuesday. I wrestled with the production system all day trying get an
accurate fix on exactly how we were failing and to stem the tide. I
thought I had it by the end of my day on Tuesday. On my Wednesday morning
(Europe's afternoon) I woke to see us slipping again. So I rolled back all
the recent deploys making Cirrus primary all the way back to the commons deploy.
== Conclusions ==
* Cirrus is just too slow as is. We need to make it faster.
== Actionables ==
* - Cirrus needs to decrease the working set size required to usefully serve traffic.
** Started by producing two weighted fields that we can query instead of 16 different fields. merged but not deployed. Production and testing must come before actual use as well.
** Other performance bugs: Filing performance bugs for Cirrus
Category:Incident documentation
Incidents/20140730-Icinga

== Summary ==
This is a short (<= 1 paragraph) of what happened.
Please ensure to remove private information.
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
Please ensure to remove private information.
== Conclusions ==
What weakness did we learn about and how can we address them?
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Bugzilla bugs or RT tickets linked for every step.
put them here
Category:Incident documentation
Incidents/20140820-SwiftMemcache

== Summary ==
Starting 2014-08-19 multiple users reported intermittent failures when doing operations on files (esp. deletes in commons) resulting in bugzilla:69760 to be opened. Investigation into swift later revealed several timeouts while talking to memcache.
== Timeline ==
* 2014-08-19T20:54:57Z bugzilla:69760 filed by users reporting errors
* 2014-08-20T13:50:15Z bug escalated and investigated. An XFS issue on ms-be1003 was found as probably root cause and the machine rebooted, seemingly fixing the problem.
* 2014-08-20T17:45:59Z more issues of the same type reported on the bug, however the failures were intermittent with users usually succeeding upon retry.
* 2014-08-21T20:54:55Z after further investigation into the swift logs several timeouts from swift proxy talking to memcache have been observed
* 2014-08-21T21:33:55Z https://gerrit.wikimedia.org/r/#/c/155629/ is deployed to increase maximum connections to memcache, thus fixing the frequent timeouts
== Conclusions ==
* The interactions between mw and swift have been historically troublesome and opaque to users, this has been no different.
* The root cause was thought to be fixed by restarting a machine in trouble, however that was a red herring.
* The real resolution was slowed down in part by the amount of logging swift does as part of its operation, masking potential problems. Another contributing factor was lack of deep insight into how mw and swift interact and what mw metrics are related to that.
* Upon inspection the swift code involved in talking to memcache doesn't seem to provide any instrumentation and thus no metrics.
* The default connection limit of 2 connections per worker was obviously not adequate with 4 memcache machines, however this doesn't seem to have caused such widespread problems in the past (compounded by the lack of error visibility too)
== Actionables ==
* - separate swift request logging from error logging RT #8181
* - instrument swift memcache code to report operational metrics RT #8182
* - improve visibility on the interactions between mw and swift and what metrics are affected
Category:Incident documentation