nyuuzyou/wikis huggingface dataset:
Incidents/20150506-EventLogging

== Summary ==
On 2015-05-05 22:20:00 UTC, the schema MobileWebSearch started to send lots of events to EventLogging, in the range of 500 events per second. Current EL capacity is far below that, so the system stagnated.
On 2015-05-06 20:00:00 UTC, the Mobile team deactivated events for that schema and EventLogging went back to normal.
During that period (22 hours), 40% of events were lost (probably for some data loss in the zmq level, caused by its buffer overflowing) and 4 data gaps appeared due to EL consumer process being killed for too much memory consumption.
The alarms for the problem were raised only on 2015-05-06 13:30:00 UTC, after 15 hours of the beginning of the issue, presumably because of the lower throughput in the night hours and because the EL buffers acted as cushion to the problem until they got full.
== Conclusions ==
Alarms: The fact that we are using fixed thresholds to trigger EL volume alarms is a limitation that prevents us to catch problems like this. We could either go for anomaly detection or different thresholds depending on the hour of the day.
Buffers: We are already working on migrating EventLogging to use Kafka as a more robust transport layer. This way we'll avoid buffer overflow issues.
== Actionables ==
* Backfill lost events.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150519-LabsOutage

== Summary ==
Labvirt1006 failed at around 18:20 on May 19th. All hosted instances become unresponsive. The system was rebooted and all instances restarted; normal service was resumed by 18:40.
== Timeline ==
* [ 16:00 ] Andrew restarts a script that suspends/resumes instances affected by the Venom issue. This was also happening when Incident_documentation/20150518-LabsOutage happened.
* [ 18:20 ] Shinken sent 'host down' alerts for a couple of tools instances. Yuvi notes that all of them are on on labvirt1006 and this is probably a repeat of Incident_documentation/20150518-LabsOutage 
* [ 18:31 ] Yuvi reboots labvirt1006
* [ 18:36 ] labvirt1006 comes up after POST
* [ 18:40 ] Yuvi runs a scripted 'start' of each instance formerly running on labvirt1001
* [ 18:45 ] All instances have resumed normal operation
== Aftermath ==
bblack noted that this is probably a kernel issue - there was a GPF in the kernel log related to XFS and one about Virtual Memory (log in /home/yuvipanda/kernlog-20150519-outage on labvirt1006). Apparently the Virtual Memory subsystems are kind of terrible in kernel series until 3.19, so this might be related. 
== Action items ==
* Make Tools Redis (toollabs redis was the only affected service) hot swappable as well - https://phabricator.wikimedia.org/T99737
* Investigate the kernel issues and find a potential solution https://phabricator.wikimedia.org/T99738
Incidents/20150519-Mailman

2015-05-19 Mailman outage
== Summary ==
The mailing lists underwent maintainance on 2015-05-19 @ 17:00 UTC. The window was expected to last until 19:00 UTC. Due to unexpected issues, the mailing list server was offline and experiencing errors until roughly 21:00. This outage was contributed towards by a mailman configuration patch (merged month+ ago) causing an unexpected issue at time of mailman service restart. Tracking down and fixing this issue was a team effort with Rob, JohnFLewis (volunteer), & Daniel.
The 'outage' was actually large scale moderation of all messages sent to lists, even by list members. All messages should be able to be sent onward by list moderators.
== Timeline ==
* 17:01 (April 15): Mailman configuration patch is merged and deployed but mailman is not restarted.
* 17:04 Mailman maintainance window started per T99098
* ~18:00: Rob finishes first of two changes, notices some odd errors on mailman restart (should have looked closer.)
* 19:00 Changes per T95195 & T99136 are completed. Testing of changes results in discovery of some/all mailing list messages being held for moderation (even when sender is member of list.)
* 19:00-21:00: Troubleshooting of all steps taken during maintainance window. John discovers old patchset, and livehack testing determines solution. Patchset reverted and pushed live.
== Conclusions ==
* lack of central logging (only roots can troubleshoot logs for sodium); should mailman logs route to central logging?
* configuration changes need to have mailman restarts at time of change
* unrelated: someone hacked up the mbox file on wiki-research-l and then didnt rebuild the archives. Once we did so today, it resulted in renumbering, which would have been best caught at the time of the introduction of said renumbering.
== Actionables ==
* Ensure ALL configuration changes are tested and production service is restarted at time of configuration change.
:* DONE via 
:* This may be a good case for a mailman-admins group, similar to other service groups.
* Either push logs to central logging, or setup a user group for non-opsen to access mailman logs: https://phabricator.wikimedia.org/T99734
Category:Incident documentation
Incidents/20150519-RESTBase

== Summary ==
The deployment of https://gerrit.wikimedia.org/r/#/c/198433/ brought down the RESTBase service for around 30 minutes, due to the parallel execution of Puppet agents across the RESTBase eqiad cluster. During this period, all of RESTBase's users were affected - VE, OCG, CXServer.
== Timeline ==
* 2015-05-19 13:08: mobrovac thinks he disabled Puppet on restbase100x
* 2015-05-19 13:09: godog merges https://gerrit.wikimedia.org/r/#/c/198433/
* 2015-05-19 13:11: mobrovac notices RESTBase cannot start up on restbase1001 due to Cassandra time-outs in select queries
* 2015-05-19 13:16: mobrovac retries with a modified config (with simplified consistency), but no change
* 2015-05-19 13:20: icinga reports RESTBase's unavailability
* 2015-05-19 13:25: _joe_ joins the investigation
* 2015-05-19 13:36: mobrovac realises he didn't really disable Puppet, so all nodes were fighting to create the new keyspaces
* 2015-05-19 13:39: godog reverts https://gerrit.wikimedia.org/r/#/c/198433/ (in https://gerrit.wikimedia.org/r/211982)
* 2015-05-19 13:42: _joe_ runs and then disables Puppet on the nodes, RESTBase is now up
* 2015-05-19 14:11: mobrovac removes all of the offending keyspaces manually from Cassandra
* 2015-05-19 14:15: mobrovac starts one RESTBase instance manually which creates the keyspaces correctly
* 2015-05-19 14:33: _joe_ re-merges the patch (as https://gerrit.wikimedia.org/r/211993)
* 2015-05-19 14:36: _joe_ disables Puppet and runs it sequentially on each node, _joe_ and mobrovac monitor the progress
* 2015-05-19 14:43: the new configuration is applied on all nodes and RESTBase and Cassandra are up and running this new version
== Conclusions ==
This particular patch required the creation of new keyspaces in Cassandra, which does not handle parallel keyspace and table creation well. As all of the RESTBase services were restarted automatically by Puppet almost at the same time, they were fighting for this creation process, effectively rendering the cluster unavailable. Thus, when there is such a code/config change, gradual application of changes, rolling restarts and periodic checks of the previous steps are imperative.
== Actionables ==
* Short-term:
** Do not allow Puppet to automatically restart RESTBase (https://gerrit.wikimedia.org/r/#/c/212016/)
** On a config-change deploy, wait for Puppet to install the new version, and then run a second RESTBase instance which brings Cassandra to the desired state
* Long-term:
** Streamline service deployment, specifically automating the rolling change application and restart ()
Category:Incident documentation
Incidents/20150529-Tagging

== Summary ==
Changes deployed on an emergency basis Friday, 2015-05-22, caused some tagging data to no longer be captured, causing an unrecoverable loss of data. This was feared to invalidate the Visual Editor A/B Pilot test in progress; however, that test used a different mechanism to gather data and was not affected.
Relevant: , 
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
Please ensure to remove private information.
== Conclusions ==
What weakness did we learn about and how can we address them?
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
Category:Incident documentation
Incidents/20150605-redis

== Summary ==
redis slaves rdb1002 and rdb1004 were unable to fully sync from master after a restart from 15.30 until 20.00 UTC. OCG, job queue and other services have seen impact whenever trying to read from slaves
== Timeline ==
* 15:30 moritzm upgrades redis on rdb1002/rdb1004 (slaves) and rdb1001/rdb1003 (masters) followed by a restart
* 18:22 OCG throws sporadic 500s which show up in icinga but not enough to trigger a notification, investigation begins
* 18:40 OCG failures correlated to redis restarts and redis network spikes
* 19:07 redis logs indicate periodic slave failures to sync from master, root cause being output buffers limits for slave clients
* 19:17 rdb1003 is restarted with new buffer limits, full sync is achieved
* 19:50 new settings applied to all rdb100[1-4], full sync and full recovery (https://gerrit.wikimedia.org/r/216293)
== Conclusions ==
redis is an important service that can have a visible impact on users, even though not complete failures like http 500.
Also the job queue, ocg and other services are impacted if redis is malfunctioning. There are no alarms if a redis slave is behind from the master or fails to sync, also operational procedures for a full redis restart and validation are missing.
In addition to the failures, the redis processes were running since 2013 and the running theory is that the dataset grew in the meantime, so a full slave sync was never needed before.
== Actionables ==
* improve redis master/slave monitoring ()
* document redis upgrade/restart procedures ()
Category:Incident documentation
Incidents/20150616-Elasticsearch

== Summary ==
Elasticsearch service (on elastic*.eqiad.wmnet nodes) backing the search functionality on all wikis was partially down for approx. 85 minutes (starting at 4:35 UTC) and fully unavailable for half an hour (starting 5:37).
== Timeline ==
* 04:35: Icinga starts to report the first failures on elastic1029 and shortly after several others (14 in total by 4:48)
* Initial investigation reveals the same outage type as in https://wikitech.wikimedia.org/w/index.php?title=Incident_documentation/20150615 (but with the master elastic1000 still working)
* 05:11: Filippo restarts elastic1031, which it fails to rejoin the cluster with a MasterNotDiscovered exception being thrown
* 05:14: Greg calls Nik, who appears shortly after
* 05:37: Filippo disables search im wmf-config via poolcounters (https://gerrit.wikimedia.org/r/#/c/218589/)
* 05:45: Nik restarts all cluster nodes
* 06:02: Search nodes are restored
* 06:08: Search is restored in wmf-config (https://gerrit.wikimedia.org/r/218591)
== Conclusions ==
* Same as https://wikitech.wikimedia.org/w/index.php?title=Incident_documentation/20150615
== Actionables ==
* Same as https://wikitech.wikimedia.org/w/index.php?title=Incident_documentation/20150615
Category:Incident documentation
Incidents/20150622-EventLogging

== Summary ==
Between Mon, 22 Jun 2015 16:10:29 UTC and Mon, 22 Jun 2015 16:43:17 UTC server side events were dropped.
We did a deploy for which forwarder (and other) config files needed to be changed. In doing so we forgot to add the --count flag to the server side forwarder config. During this period, server side events were not processed.
== Actionables ==
* Backfill lost events.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20150625-LabsNFSOutage

== Summary ==
There was a 12 minute Labs NFS outage (and subsequently Tools outage) caused by human error (reboot entered on the wrong terminal accidentally). The system came back up, the arrays were assembled and NFS was back up in 12 minutes. 
== Timeline ==
16:35: Yuvi accidentally types reboot into labstore1002 instead of a labs instance awaiting NFS removal.
16:38: Machine comes back up, Coren starts assembling arrays and bringing the NFS server back up
16:47: Instances all recovered from NFS stall, everything working ok
== Actionables ==
* Feed Yuvi to a Moose Yuvi, we love you!
* Install the molly-guard package on production hosts https://phabricator.wikimedia.org/T103873
Incidents/20150702-RESTBase

== Summary ==
On 2015-07-02, around approximately 10:18 UTC RESTBase started refusing connections because its workers were stuck due to a time-out error in executing Cassandra queries (stack trace), which were themselves caused by the Cassandra cluster's high load. In addition, RESTBase's automatic worker restart did not fire off.
== Timeline ==
* 2015-07-02 10:18: Connections start getting refused on RESTBase's LVS
* 2015-07-02 10:19: akosiaris notifies mobrovac
* 2015-07-02 10:21: mobrovac verifies the ports, which present themselves as up
* 2015-07-02 10:22: _joe_ discovers failed fetches for individual nodes
* 2015-07-02 10:24: mobrovac restarts one node, which brings it back to life
* 2015-07-02 10:26: mobrovac rolling-restarts the rest of the nodes, normal operation resumes
== Conclusions ==
Due to limited monitoring checks (only an HTTP check on LVS), we were confused as to what was going on since the processes on the nodes were seemingly up and bound to the usual port. Moreover, statistics and logs indicate that RESTBase workers were dying off slowly over time (from approximately 2015-07-02 07:30 UTC onwards), which means that the incident had been undergoing for 2:30 before we received a notification.
Furthermore, it was not clear to us why RESTBase workers were not automatically restarted, given that RESTBase was supposed to have automatic worker restarts.
== Actionables ==
* Include HTTP checks for each RESTBase back-end
* Investigate the failure of RESTBase to restart workers
* Investigate why some workers did not actually die but were left hanging
Category:Incident documentation
Incidents/20150709-poolcounter

== Summary ==
There was an 8 minute outage of api at eqiad, starting from 2015-07-09 17:26:35 and ending at 2015-07-09 17:34:15 caused by scheduled maintenance and an unforeseen dependency. helium was powered down for https://phabricator.wikimedia.org/T84770. helium however is also a poolcounter machine. Unfortunately mediawiki has a 0.5 sec timeout when falling back to the next poolcounter server in line which is too high.
== Timeline == 
17:22 cmjohnson1: shutting down helium for a few minutes to move within the same row
17:26:35 icinga complains about api.svc.eqiad.wmnet. Before that it had already complained about HHVM queue sizes on various mw hosts. mutante noticed it's poolcounter host
17:31 ori merges https://gerrit.wikimedia.org/r/#/c/223838/ making mw1154 a poolcounter server, effectively bypassing helium and pottasium. Recoveries start coming in
17:34 icinga declares api.svc.eqiad.wmnet OK.
== Actionables ==
* Stop a poolcounter server fail from being a SPOF for the service and the api (and the site) - T105378
* Revert https://gerrit.wikimedia.org/r/#/c/223838 after helium is deemed fine again (merge https://gerrit.wikimedia.org/r/#/c/223847/ to revert) T105379
* Remove poolcounter from mw1154 for housecleaning (the box is going to get reimaged anyway however, hopefully soon) T105380
Incidents/20150803-Kafka

== Summary ==
A gradual upgrade was planned to expand the Kafka cluster and to bring it to Jessie running Kafka 0.8.2.1 in https://phabricator.wikimedia.org/T106581.
Otto tested this upgrade plan in labs, and all was fine. However, the testing was not extensive enough, as it was not tested with varnishkafka or a regular stream of events. 
== Timeline ==
At 2015-08-03T17:21, Otto began the process of partition reassignment to balance partitions across 6 nodes, 3 of which were the original Precise with Kafka 0.8.1.1, 3 Jessie with Kafka 0.8.2.1. This seemed fine, but Otto believes that once some partitions were totally moved off of the original 0.8.1.1 brokers, things started going wrong. Otto is still not entirely sure what went wrong. He spent several hours attempting to reconcile the problems. During this time, most varnishkafkas were still producing. However, many of them periodically failed with metadata request timeouts, which caused failed produce requests.
After several hours of trying to fix this, Otto decided the faster way to get everything back to normal would be to revert to only the 4 original 0.8.1.1 Kafka brokers, by wiping the Kafka cluster metadata and data clean and starting from scratch. This was done at at 2015-08-03T23:45. After this, the status quo was restored and message production and consumption proceeded as normal.
Webrequest data experienced significant loss during this outage. Data in Hadoop between 2015-08-03T17:00 and 2015-08-04T00:00 should be considered unreliable.
See also: Analytics/Data/Webrequest
== Actionables ==
* - Rebuild 0.8.2.1 Kafka .deb package with updated Snappy dependency, and double check other dependencies in gradle.
* - Figure out what went wrong and/or do more extensive testing in labs.
* - Augment deployment plan to do this again without incident.
Category:Incident documentation
Incidents/20150806-poolcounter

20150806-poolcounter
== Summary ==
A firewall change was merged on server helium which serves as Bacula director and also as a poolcounter.
There was a failure to start the ferm service. As soon as iptables rules were loaded the conntrack table filled up.
Packets and connections to the poolcounter were dropped.
This caused an API outage (cached things were unaffected).
== Timeline ==
#~ 20:29 UTC - gerrit:229054 gets merged to complete T104996 and apply ferm rules on helium, ferm rules for both bacula and poolcounter exist
#~ 20:31 UTC - Daniel runs puppet on helium and sees ferm fails to start with error [1]
#~ 20:32 UTC - Icinga starts to report Socket timeouts for Apache HTTP and HHVM rendering 
#~ 20:33 UTC - Daniel runs manual script to flush all iptables rules that was there in case ferm fails [2], disables puppet agent
#~ 20:40 UTC - Brandon starts looking at helium, connects via mgmt and finds [3] shortly after
#~ 20:42 UTC - Daniel reverts gerrit:229054
#~ 20:43 UTC - Daniel re-enables puppet-agent, attempts run, gets fails because helium fails DNS lookups, packets are still dropped
#~ 20:43 UTC - Brandon manually rmmod's all the iptables kernel modules
#~ 20:45 UTC - Icinga RECOVERies start showing up
== Conclusions ==
- avoid the issue with the NOTRACK target in ferm
- if ferm fails and conntrack table fills up, rmmod kernel modules, flushing all tables is not enough
- poolcounter is a SPOF (T105378)
== Actionables ==
# stop a poolcounter server fail from being a SPOF (T105378)
# detect failing ferm restarts (T108303)
# fix gerrit:228137
[1] 
867 Aug 6 20:29:51 helium kernel: [2422180.176691] ip6_tables: (C) 2000-2006 Netfilter Core Team
868 Aug 6 20:29:52 helium kernel: [2422181.102840] x_tables: ip_tables: NOTRACK target: only valid in raw table, not filter
869 Aug 6 20:29:52 helium kernel: [2422181.123525] x_tables: ip6_tables: NOTRACK target: only valid in raw table, not filter
870 Aug 6 20:29:52 helium puppet-agent[10048]: (/Stage[main]/Ferm/Service[ferm]) Failed to call refresh: Could not start Service[f erm]: Execution of '/etc/init.d/ferm start' returned 1:
871 Aug 6 20:29:52 helium puppet-agent[10048]: (/Stage[main]/Ferm/Service[ferm]) Could not start Service[ferm]: Execution of '/etc /init.d/ferm start' returned 1:
[2]
#!/bin/sh
# removes all iptables rules
# https://en.wikipedia.org/wiki/Tear_down_this_wall!
echo "flushing all iptables rules.."
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
echo "done"
[3]
Aug 6 20:30:32 helium kernel: [2422220.736092] nf_conntrack: table full, dropping packet.
Category:Incident documentation
Incidents/20150814-MediaWiki

== Summary ==
On 23:40 a change was pushed out which removed and unloaded the FastStringSearch extension for HHVM. We had code in MediaWiki which checks for the presence of this extension and branches accordingly, using a fallback when the extension is not available. One of the callers of this code passes it a parameter of the wrong type, an error which the FastStringSearch extension had swallowed, but which caused a fatal error in the fallback branch (now filed as ). This bug was not caught on the beta cluster, because the code-path is exercised when converting text from one language variant to another, which does not happen frequently in that environment.
It does happen frequently in production, so the error-rate spiked. To revert, the configuration line hhvm.dynamic_extensions[fss.so] = fss.so needed to be restored to /etc/hhvm/fcgi.ini, but it needed to happen sooner than the next Puppet run. An engineer ran a command across all application servers which was meant to append the line to end of the file but which truncated the file instead. This caused HHVM to restart with a skeleton configuration file, making a bad problem worse.
At 23:54 a good copy of the configuration file was provisioned across the cluster and HHVM was restarted, at which point the site recovered.
== Timeline ==
* 23:40 bad change pushed
* 23:54 recovery starts
== Actionables ==
* ReplacementArray::replace() called with ResourceLoaderContext instead of string ()
Category:Incident documentation
Incidents/20150817-ToolLabs-WebgridOutage

== Summary ==
The grid engine exec daemon did not restart correctly on all host after reboots, causing the trusty webgrid queue to be overloaded. This in turn prevented webservice jobs from (re)starting, which caused outages for several tools on Tool Labs.
== Timeline ==
* Earlier: tools-webgrid-lighttpd-1404 and tools-webgrid-lighttpd-1406 are rebooted, and gridengine-exec does not start correctly
* 2015-08-17 16:17: Andrew disables tools-webgrid-lighttpd-1406, marking the start of the downtime
=== 2015-08-17 ===
* 21:10 Valhallasw receives complaints about webservice not starting on a private channel and investigates
* 21:20 notices job is stuck in ‘qw’ state. Notices this is due to a lack of available vmem, but initially memory, but initially blames this on the 4GB h_vmem requirement of this job
* 21:31 Valhallasw realizes the 4GB is standard for webservice jobs, and calls on Coren and Andrew in #wikimedia-labs
* andrew suggests this is due to a lack of webgrid nodes
* 21:37 Valhallasw goes to sleep.
* -22:13 Andrew builds a new webgrid node
* 22:13 Andrew asks Yuvi for advice on how to add a new tools-webgrid-lighttpd node to the grid; Yuvi is on vacation in Europe and asleep.
=== 2015-08-18 ===
* 01:00 Complaints about webservices not starting on IRC; also on labs-l
* 07:25 Valhallasw looks into the issue again, spends 1.5 on building host and adding it to the grid due to no documentation
* 08:30 during configuration, Valhallasw notices queues on tools-webgrid-lighttpd-1403.eqiad.wmflabs and tools-webgrid-lighttpd-1404.eqiad.wmflabs are also in alarm state.
* 08:36 Valhallasw brings back online tools-webgrid-lighttpd-1403.eqiad.wmflabs and tools-webgrid-lighttpd-1404.eqiad.wmflabs, as well as the new webgrid host tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs, marking the end of the downtime
Total downtime: approx. 16 hours
== Conclusions ==
* exec nodes do not always start the gridengine-exec daemon on boot, causing the node to be unavailable for jobs
* there was no documentation on building a new exec host, which made it difficult for Andrew and Valhallasw to do so
* maintenance (virt reboots) was planned in a time slot where Coren was unavailable, which meant there was no-one with SGE-specific knowledge around to investigate issues
* there is no monitoring for queue status, so we did not notice the queues were offline before issues arose
* there is no monitoring to check there are enough hosts available for current and predicted load
== Actionables ==
* Make sure gridengine-exec starts on boot ()
* Write documentation on building new exec hosts ()
* Add monitoring for queue status ()
* Add monitoring for expected load issues ()
Category:Incident documentation