nyuuzyou/wikis huggingface dataset:
Incidents/20151118-LVS-PyBal

== Summary ==
PyBal was stopped on the primary eqiad LVS servers (lvs100[123]) for maintenance, with the expectation that traffic would be unaffected and shift to the backup servers (lvs100[456]). Some services were not operating correctly on the backups, causing a partial outage. pybal was restarted ~5 minutes later on the primaries, ending the outage.
The impact during the window was limited. Most service traffic moved successfully, and only a few specific service+proto+port combinations failed, with the primary public-facing affected services being:
Service 	Protocol	Port
text-lb 	IPv4 	80
mobile-lb 	IPv6 	443
misc-web-lb	IPv4 	443
Because most of our traffic is IPv4 on port 443 (HTTPS) for all services, the impact to text and mobile services was limited (HTTP->HTTPS redirects for text, IPv6 users for mobile). misc-web was affected for most users, denying access to services such as phabricator, gerrit, racktables, etc.
Traffic graphs of the primary clusters in eqiad: https://phabricator.wikimedia.org/F2972690
== Timeline ==
* 13:56 - pybal stopped on lvs100[123] by bblack
* 14:00 - first user report on IRC: "< aude> did someone kill phabricator?"
* 14:01 - first automated report on IRC: "< icinga-wm> PROBLEM - LVS HTTP IPv4 on text-lb.eqiad.wikimedia.org is CRITICAL: Connection refused"
* 14:01 - pybal restarted on lvs100[123] by bblack
* 14:02 - traffic restored to normal
== Conclusions ==
Because the same pattern of pybal failover to the same version of software had happened successfully at 3 other datacenters the day before, confidence was too high and not enough pre-flight checking was done. More verification on the state of lvs100[456] should have been done prior to the start of maintenance. The problem issues were obvious prior to the outage in the output of "ipvsadm -Ln" as well as the pybal service logs in "journalctl" if they were examined in depth. 
The actual technical issues are due to bugs in PyBal.
== Actionables ==
* Fix PyBal ()
Category:Incident documentation
Incidents/20151123-LabsOutage

== Summary ==
As part of switching primary Labs DNS service from holmium to labservices1001, Andrew caused an outage of labs-recursor0 and labs-recursor1, thus breaking resolution of internal Labs IPs from about 05:22 to about 05:31.
== Timeline ==
* [previously] Andrew is moving Labs designate and DNS services from holmium to labservices1001, in pursuit of the ultimate renaming of holmium.
* [05:07] Andrew merge a patch which is meant, via hiera, to exchanges the assignments of labs-recursor0 and labs-recursor1. This is consistent with the 'primary service on labservices1001' goal, but ignores the fact that each labs-recursor0's IP is routed to the rack containing holmium and /not/ routed to the rack containing labservices1001, and likewise for the IP for labs-recursor1. So, this patch should have broken DNS immediately -- it did not due to a mistake in the patch which accidentally assigned the IP for labs-recursor0 is assigned to both holmium and labservices1001. Consequently labs-recursor0 still works and labs-recursor1 does not.
* [05:17] Andrew merges a second patch patch which corrects the typo. At this point labs-recursor0 is assigned to labservices1001 and labs-recursor1 to holmium, both unroutable.
* [05:22] The above patch is applied, and the first diamond alerts start showing up about DNS resolution failure.
* [05:30] Andrew realizes the source of the problem, submits a patch returning the IPs to their original homes.
* [05:32] The above patch is merged, and normal service is restored.
Incidents/20151126-graphite-grafana

== Summary ==
graphite.wikimedia.org was overwhelmed with legitimate requests, yielding 500s returned to clients
== Timeline ==
* 2015/11/25 14:48 - graphite starts throwing 500s to clients
* 2015/11/25 14:52 - alert on icinga, investigation begins
* 2015/11/25 14:57 - heavy query/dashboard suspected, uwsgi bounced
* 2015/11/25 15:11 - big influx of requests on graphite1001's apache identified as being the root cause. likely a misbehaving dashboard
* 2015/11/25 15:24 - labs-monitoring grafana dashboard change default refresh inverval from 5s to 5m
* 2015/11/25 15:53 - kafka dashboard also suspected and banned from apache
== Conclusions ==
grafana dashboards relying on intensive graphite queries can easily
overwhelm graphite itself, particularly dashboard that refresh
frequently, resulting in denial of service.
in addition, it has been observed that misc varnish retries the
request on 5xx from a backend, further contributing to thundering herd
of requests.
== Actionables ==
* Make it easier to ban misbehaving dashboards from graphite ()
* Enforce a minimum refresh period for grafana dashboards hitting graphite ()
* 500 errors from graphite shouldn't be retried by varnish ()
Category:Incident documentation
Incidents/20151127-EventLogging

== Timeline ==
On the 27th of November
1:30 am UTC sql insertion rate goes to zero, topics that are feed from outside in kafka continue to receive events, event-login-valid-mixed is receiving events but not as much as it should have
 
At the same time we see this errors in the eventlogging_processor log:
 2015-11-27 01:31:09,663 (MainThread) Could not receive response to request [0000026b0000000000a0 ... 6b69223a2022656e77696b69227d] from server <KafkaConnection host=kafka1013.eqiad.wmnet port=9092>: Kafka @ kafka1013.eqiad.wmnet:9092 went away
 2015-11-27 01:31:09,664 (MainThread) Could not receive response to request [0000038a00000000009d ... 6b69223a2022657377696b69227d] from server <KafkaConnection host=kafka1013.eqiad.wmnet port=9092>: Kafka @ kafka1013.eqiad.wmnet:9092 went away
Kafka had an outage in which only one of the brokers seems to be working:
06:50 am UTC eventlogging gets rebooted and a spike on consumption can be seen on Grafana
07:05 am UTC consumption catches up
== Conclusions ==
Eventlogging consumers get stuck when there are connection problems talking to kafka. System requires a reboot to be able to recover after kafka has been brought back.
== Actionables ==
* Investigate whether backfilling is needed
* Backfill missing data
* Make Eventlogging more resilient to kafka outages: 
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20151203-SiliconOutage

On December 3 2015 at about 14:00 UTC Silicon (the fundraising ActiveMQ server) stopped accepting connections. FR tech then received a flood of email alerts from Barium, the queue consumer. Elliott, Peter, and Jeff investigated immediately.
== Sequence of Events==
* 14:00 email alerts begin
* 14:15 Peter takes banners down
* 14:30 Jeff finds CPUs pinned at 100% on Silicon and restarts ActiveMQ with an increased heap size
* 15:00 queues start processing and successful test donations are made
* 15:10 Peter puts banners back up
== Impact ==
During the time the queue was locked up no donations could be processed, donors may have seen error messages or timeouts. Any data irregularities should sort themselves out via the audit file download and process jobs.
== Root cause ==
JVM maxed out its heap, probably due to a tuning problem:
 2015-12-03 14:30:31,496 | WARN | Transport Connection to: tcp://10.64.40.109:59670 failed: java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java heap space | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.64.40.109:59670@61613
== Recommendations ==
The problem has not reoccurred since increasing the heap size. However, reporting on Ganglia did not show the queue to be abnormally large at the time Java ran out of memory, so if this happens again further investigation into the JVM tuning on Silicon may be warranted.
Incidents/20151216-ores

== Summary ==
ORES stopped serving requests for 3 hours because Redis ran out of disk space and the Workers could no longer function.
== Timeline ==
; 1130 UTC: a report was made via IRC that ORES was returning 503 "server overloaded" errors.
; 1300 UTC: this message was seen by Halfak, the workers were restarted and came back online briefly only to go back offline again
; 1325 UTC: web proxy for ores.wmflabs.org was changed to direct traffic to the staging server (cpu usage went to 100%, but most requests were being served effectively)
; 1345 UTC: workers and logs are observed. Workers appear to come back online and the crash with an error:
:redis.exceptions.ResponseError: MISCONF Errors writing to the AOF file: No space left on device
; 1350 UTC: Halfak calls Yuvipanda for help. Steps taken:
:# hand hack redis config file to turn off AOF
:# physically move the files out of /srv/redis
:# restart redis
; 1410 UTC: web proxy for ores.wmflabs.org was changed to direct traffic back to the prod cluster. precached turned back on
; 1430 UTC: Victory declared. Yuvipanda goes back to sleep and Halfak continues monitoring
== Conclusions ==
# Our monitoring does not catch when the webservers are up but the workers are down. We need to implement monitoring for the worker nodes.
# Our LRU policy for redis did not account for the additional disk space that would be required when using the AOF (append) persistence strategy. Switch to using RDB (snapshots)
== Actionables ==
* Phab:T121658 -- Switch from AOF to RDB persistence strategy for ORES redis
* Phab:T121656 -- Add monitoring to ORES workers
Category:Incident documentation
Incidents/20151218-payments wiki

On December 18 2015 at 11:15 UTC icinga started sending alerts about payments
webserver timeouts. We found apache processes backing up, waiting for trivial 
queries to the fundraising drupal database.
== Sequence of Events==
* 11:15 icinga alerts begins, Faidon investigates
* 11:38 Jeff contacted and starts investigating
* 11:40 Peter takes banners down
* 12:20 Jeff adjusts mysql table_open_cache and flushes tables on mysql server, services promptly recover
* 12:30 Peter puts banners back up
== Impact ==
During the time the database was sluggish payment processing performance was
degraded and donors may have seen error messages or timeouts.
== Root cause ==
We appear to have overrun mysql's configured open table cache, which caused mysql to become sluggish to open and close tables for queries.
== Recommendations ==
Additional mysql tuning was performed after the outage to prevent similar outages.
Category:Incident documentation
Incidents/20151229-SiliconOutage

On December 29 2015 at about 21:57 UTC Silicon (the fundraising ActiveMQ server) stopped accepting connections. FR tech received a flood of email alerts from Barium, the queue consumer. Elliott investigated immediately, and paged Jeff.
== Sequence of Events==
* 22:00 email alerts begin
* 22:05 (correct time?) Peter takes banners down, pages Jeff
* 22:40 Jeff finds CPUs pinned at 100% on Silicon, ActiveMQ unresponsive and restarts ActiveMQ
* 22:41 queue and payments processing return to normal function
* 22:45 Peter puts banners back up
== Impact ==
During the time the queue was locked up no donations could be processed, donors may have seen error messages or timeouts. Any data irregularities should sort themselves out via the audit file download and process jobs.
== Root cause ==
ActiveMQ appears to have crashed.
== Recommendations ==
Further investigation re. potential ActiveMQ bugs.
Category:Incident documentation
Incidents/20160125-EventLogging

==Summary==
The GC logs of some Kafka nodes grew so big that filled their disks. A couple hours later, EventLogging's processors lost connection with Kafka and stopped consuming events (EL's mysql consumers were already disabled at the time of the incident for db maintenance). Operations fixed the issue removing the GC logs and setting up rotation for them (https://phabricator.wikimedia.org/T124644). Analytics restarted EventLogging and backfilled the missing data.
==Timeline==
=== 2016-01-24 00:09:18 UTC ===
 PROBLEM - Disk space on kafka1020 is CRITICAL: DISK CRITICAL - free space: / 1060 MB (3% inode=96%)
First occurrence of the issue in the wikimedia-operations channel, followed by other ones.
===2016-01-25 07:22 UTC===
The Ops team started to see the following errors in the wikimedia-operations channel:
 PROBLEM - Disk space on kafka1020 is CRITICAL: DISK CRITICAL - free space: / 1060 MB (3% inode=96%)
They quickly identified the problem, the disk was full:
 df -h:
 ...
 /dev/md0 28G 26G 960M 97% /
 ...
Due to really big GC logs 20+GB: 
 ls -lh /var/log/kafka/kafkaServer-gc.log
 -rw-r--r-- 1 kafka kafka 23G Jan 25 08:32 /var/log/kafka/kafkaServer-gc.log
This was more or less spread across all nodes.
=== 2016-01-25 09:30 UTC ===
The Ops team filed a gerrit code review to add a better rotation policy to Kafka brokers (https://gerrit.wikimedia.org/r/#/c/266203/).
=== 2016-01-25 09:56 UTC ===
The Ops team restarted the first Kafka broker (kafka1012).
===2016-01-25 10:13 UTC===
The Analytics team received the following alert in the wikimedia-analytics channel:
 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: 60.00% of data above the critical threshold [30.0]
The Kafka brokers restarts caused an expected metadata change that was not handled correctly by pykafka clients. EL's logs were stuck to Connection refused messages from Kafka brokers. 
===2016-01-25 11:00 UTC===
Analytics restarted EventLogging and processors went back to normal. The root cause of the Kafka failure was that the GC logs were not setup for rotation. The root cause for the EventLogging producers blocking is still unknown.
==Actionables==
* Investigate the root cause of the Kafka disconnects. 
* Investigate if a stricter set of alarms would have helped (triggering for example at 70/80% disk utilization rather than 96%).
* Adding Kafka and Hadoop alarms to the analytics channel for better visibility.
* Backfill missing data.
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20160128-MediaWiki-API

== Summary ==
A Kafka broker was rebooted as part of a standard upgrade. The MediaWiki API cluster failed as a result, due to an overwhelming number of piled up connection attempts to that particular Kafka broker. The broker was ultimately depooled from mediawiki-config as part of the incident response and the service was ultimately restored, 25 minutes after it first started failing.
== Timeline ==
* 12:53 Luca follows the safe broker restart process and stops Kafka on kafka1012 for a host reboot for a kernel upgrade
* 13:36 multiple mw* hosts report HHVM Rendering "Socket timeout after 10 seconds" failures, multiple opsens respond
* 13:49 Giuseppe notices a lot of HHVM threads stuck at attempting to connect, then a large number of connections to kafka1012 in a SYN_SENT state.
* 13:53 Faidon pushes a patch to remove kafka1012 from mediawiki-config and syncs it to the appserver fleet
* 13:58 File is synced across the fleet, traffic recovers
== Actionables ==
* MediaWiki monolog doesn't handle Kafka failures gracefully ()
Category:Incident documentation
Incidents/20160209-PlannedMaintenance

Between 23:00 and 23:59 UTC, February 9th 2016 there is a scheduled maintenance window that will affect some of the wikis hosted by the Wikimedia Foundation. 
The maintenance is needed in order to perform necessary hardware, operating system and database upgrades. During the upgrade, content on affected wikis will be available at all times, but edits may fail during approximately 5 minutes within that schedule (these wikis will be in "read only mode"). 
== Wikis affected ==
The following wikis will be affected:
* bg.wikipedia.org
* bg.wiktionary.org
* cs.wikipedia.org
* en.wikiquote.org
* en.wiktionary.org
* eo.wikipedia.org
* fi.wikipedia.org
* id.wikipedia.org
* it.wikipedia.org
* nl.wikipedia.org
* no.wikipedia.org
* pl.wikipedia.org
* pt.wikipedia.org
* sv.wikipedia.org
* th.wikipedia.org
* tr.wikipedia.org
* zh.wikipedia.org
All other wikis will not be affected by this maintenance.
We apologize in advance for this disruption and will try to minimize the duration of the maintenance work.
== Post-Maintenance Report ==
Upgrade seems to have gone well. Read-only mode was enabled for a bit longer than we initially expected: approximately between 23:16 and 23:30, but all within the scheduled maintenance window. Most of the time was spent monitoring that the switchover was done correctly, and that traffic was not affected after the fact. s2 master is now running a modern operating system version (Debian Jessie) and the latest version of MariaDB 10 on a new host, and no replication lag or major issues where detected after the failover. We will continue monitoring that shard and wikis for stability and performance.
If you find any further issues, please use Phabricator to communicate this to us.
Category:Incident documentation
Incidents/20160214-labsdb1002

== Summary ==
At about 22:30 UTC on 2016-02-14, labsdb1002 suffered a disk failure and unmounted the volume that hosted several tools databases. We were unable to remount the system, so the db server was depooled pending a disk replacement.
== Timeline ==
22:28: incinga reports
 mysqld processes on labsdb1002 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld
 MariaDB disk space on labsdb1002 is CRITICAL: DISK CRITICAL - /srv is not accessible: Input/output error
 Disk space on labsdb1002 is CRITICAL: DISK CRITICAL - /srv is not accessible: Input/output error
Giuseppe, Andrew, Chase, Jaime, Ariel and Alex Monk responded to the alerts. Logs include the following:
 Feb 14 22:24:55 labsdb1002 kernel: [21554975.965750] XFS (dm-1): Log I/O Error Detected. Shutting down filesystem
 Feb 14 22:24:55 labsdb1002 kernel: [21554975.965752] XFS (dm-1): Please umount the filesystem and rectify the problem(s)
 Feb 14 22:24:55 labsdb1002 kernel: [21554975.965754] XFS (dm-1): metadata I/O error: block 0xc00f2fb0 ("xlog_iodone") error 5 numblks 64
 Feb 14 22:24:55 labsdb1002 kernel: [21554975.965757] XFS (dm-1): xfs_do_force_shutdown(0x2) called from line 1170 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_log.c. Return address = 0xffffffffa02db801
22:50: Giuseppe attempts to unmount and remount the failed volume
22:59: After discussion it's agreed that the other db servers can handle the load from 1002, so Jaime submits https://gerrit.wikimedia.org/r/#/c/270650/ which directs access for affected DBs to other servers. Andrew merges the patch and applies on labservices1001.
23:05: Andrew restarts the 'replag' tool by hand, at which point it resumes normal operation. Many tools recover spontaneously, and a few others are restarted by hand by operations staff.
23:22: Andrew sends an email to labs-announce encouraging tool maintainers to restart their services.
== Conclusions ==
Affected tools replica databases:
* 'bgwiki'
* 'bgwiktionary',
* 'commonswiki'
* 'cswiki'
* 'dewiki'
* 'enwikiquote'
* 'enwiktionary'
* 'eowiki'
* 'fiwiki'
* 'idwiki'
* 'itwiki',
* 'nlwiki'
* 'nowiki'
* 'plwiki'
* 'ptwiki'
* 'svwiki'
* 'thwiki'
* 'trwiki'
* 'wikidatawiki'
* 'zhwiki'
Tools with sensible reconnect logic recovered immediately after labsdb1002 was depooled. Those without will require a manual restart, which is largely left up to tool maintainers.
== Actionables ==
* Replace the broken disk and repool labsdb1002: https://phabricator.wikimedia.org/T126946
* Consider implementing an automatic failover system for labsdb shards
Category:Incident documentation
Incidents/20160319-Ores

== Summary ==
ORES went down and responded slowly for ~2 hours today. 
== Timeline ==
* 1930 UTC: New deployment begins
* 2005 UTC: ORES begins to be overloaded
* 2025 UTC: A problem with old Jessie installs is discovered Phab:T130463 -- it turns out that it was really a pip issue with versioning https://github.com/pypa/pip/issues/214
* 2130 UTC: A new cluster is built and requests are being served at the rate that they come in
* 2300 UTC: A new cluster configuration is complete.
== Conclusions ==
# Pip does not remove old versions when installing new wheels. This will need to be done manually
# Our precaching utility will back-up during a short outage and unleash a load of requests on the service when it comes back online
== Actionables ==
* Implement version purging on top of pip see change 
* Add a web node to the cluster Phab:T130394 
Category:Incident documentation
Incidents/20160407-Mediawiki

== Summary ==
All Mediawiki servers were serving mostly HTTP 5XX for about 5 minutes at 1350 UTC
== Timeline ==
* 13:50 UTC: switch CirrusSearch traffic to codfw, with a buggy configuration (see https://gerrit.wikimedia.org/r/#/c/282163/ for the correction)
* almost immediate raise in HTTP 5XX errors to 400K errors / minute
* 13:53 UTC: rollback
* 13:55 UTC: error rate back to reasonable level
== Conclusions ==
# unit testing wmf-config is hard
# testing configuration changes related to datacenter is not possible on labs
# carefully testing this kind of change on test nodes (mw1017/mw1099/mw2017/mw2099) is the minimum required
== Actionables ==
Immediate issues have been addressed. This incident is mainly about human error (mine) and insufficient testing (me again).
* a standardized and automated canary test system would help mitigate this kind of issues, but is probably a long term action outside of the scope of a post incident action.
Category:Incident documentation
Incidents/20160503-Wikidata-Query-Service

== Summary ==
Starting from May 03 2016 around 7:30 UTC, WDQS started to have occasionally increased response time, leading to HTTP 502 errors from Varnish. At that time, WDQS was running on a single server due to a reinstall and data reload in progress. Restarting Blazegraph restored the service. Multiple restart were done over the following days.
The issue was tracked to multiple causes: a known bug in the version of Blazegraph that we use and a file descriptor leak related to Jolokia (monitoring agent).
== Timeline ==
* 2016-05-01T19:37 enabled wdqs1002, put wdqs1001 in maintenance mode for reload
* 2016-05-03T11:08 issue reported in https://phabricator.wikimedia.org/T134238 and IRC
* 2016-05-03T12:28 wdqs-updater killed as it seems to leak pipes
* 2016-05-03T13:01 restarting wdqs-updater and keeping it under close scrutiny
* 2016-05-03T17:18 restarting wdqs1002
* 2016-05-03T21:11 restarting wdqs1002
* 2016-05-04T23:26 deployed additional Icinga check increase visibility on this issue
* 2016-05-05T08:12 restarting wdqs1002
* 2016-05-05T08:57 restarting wdqs1002
* 2016-05-05T11:32 restarting wdqs1001
* 2016-05-05T21:00 deploying fix to Jolokia
* 2016-05-07T12:32 restarting wdqs1002
* 2016-05-07T20:13 restarting wdqs1001 and wdqs1002
* 2016-05-07T20:28 deploying updated Blazegraph version for WDQS to mitigate deadlock issue
== Conclusions ==
* Running on 2 servers when maintenance tasks (data reload) can take multiple days is not enough.
* We were alerted by users, our monitoring is not sufficient.
== Actionables ==
* Done: run Jolokia as a Java agent, not attaching and detaching it at each run
* Done: add response time check to WDQS
* Done: Deploy new Blazregraph version to fix BLZG-1884
* Tasks opened: Adjust balance of WDQS nodes / Deploy WDQS node on codfw
Category:Incident documentation