nyuuzyou/wikis huggingface dataset:
Incidents/2017-08-29 EventStreams

== Summary ==
The Services team needed an increase in the maximum message size that the main Kafka brokers would allow, so that they could produce large JobQueue events.
https://gerrit.wikimedia.org/r/#/c/372179/ was merged and services were restarted around 2017-08-28T19:00. This change was not applied to analytics Kafka brokers, nor was it applied to MirrorMaker producer and consumer settings. This caused the MirrorMaker processes to die when they attempted to mirror a message from a main Kafka cluster. Because no data from the main Kafka clusters could be mirrored to the analytics Kafka cluster, EventStreams, which uses topics that come from the main Kafka clusters, but consumes from the analytics Kafka cluster, stopped sending events to connected clients.
Otto was notified about no messages in EventStreams by an IRC user at on 2017-08-29, at around 14:44 EST. He looked in MirrorMaker logs, and found a message about a message exceeding max allowed size, and realized that this setting would need to be synced between all clusters (if we are mirroring messages between them) and also for clients less than 0.10 (all of our current clients).
== Timeline ==
* 2017-08-28T15:03 (gerrit time?) https://gerrit.wikimedia.org/r/#/c/372179/ is merged.
* 2017-08-28T19:24 UTC Otto begins restarting the main-eqiad Kafka cluster and evenbus producers.
* Petr confirms that JobQueue can now produce larger messages to main Kafka clusters.
* 2017-08-29T14:44 EST, IRC user notifies Otto that EventStreams is broken.
* Otto begins merging puppet patches to synchronize settings between Kafka clusters and clients:
** https://gerrit.wikimedia.org/r/#/c/374610/
** https://gerrit.wikimedia.org/r/#/c/374619/ 
* 2017-08-29T19:34 UTC, Otto applies puppet, restarts analytics Kafka cluster and all MirrorMakers
* Mirrored events begin flowing again into analytics Kafka, and to connected EventStreams clients.
== Conclusions ==
There were ZERO alerts about a drop of messages in these analytics Kafka topics, and for EventStreams clients!
== Actionables ==
* Monitor and alert on EventStream topic volume ()
Incidents/2017-09-20 Elasticsearch

== Summary ==
Yesterday, we did a cold restart of the elasticsearch / cirrus eqiad cluster. This restart did not go as planned. It did not generate any user facing impact, since we moved all the traffic to codfw before the restart. It did impact logstash (more of that in a different report).
== Timeline ==
* 13:15: search traffic moved to codfw
* 15:36: start of plugin upgrade and cluster restart on elasticsearch eqiad
* 15:46: shutdown and restart of all elasticsearch nodes, cluster is not recovering
* investigation (detailed before, no exact timeline)
* 16:34: restart of elastic1051, cluster is starting to recover, cluster is in yellow state after a few minutes (ready to serve traffic)
* 21:00 (approx): full recovery of cluster
== Conclusions ==
elastic1051 did not restart cleanly, due to failed cleanup of previous plugins. This was not investigated right away, as we thought that recovering the cluster was more urgent and that the clsuter should recover without issue even if one node is unavailable. We now know differently.
The parameters governing the recovery of the cluster are:
* recover_after_nodes: 24
* recover_after_time: 20m
* expected_nodes: 36
Our previous understanding was that the cluster would start recovering with 24 of the 36 nodes available, or start recovering anyway after 20 minutes. So we were not expecting that one node being down would prevent a recovery. Our current understanding is that elasticsearch will wait for expected_nodes to be available, and recover anyway after recover_after_time if at least recover_after_nodes are present.
== Actionables ==
* Update our documentation.
* Tune recovery settings
Incidents/2017-09-20 Logstash

== Summary ==
Logstash stopped processing logs while cirrus elasticsearch cluster (eqiad) was down for maintenance.
== Timeline ==
* Sept 20, 15:48 to 16:44: logs collected drop to almost zero (see graph)
* Sept 20, 16:42: rolling restart of the logstash collectors (this happened at roughly the same time as elasticsearch recovery, so it might have been useless)
== Conclusions ==
The only identified link between logstash and elasticsearch cluster is the logging of API features. Logs are collected by logstash and forwarded not to the logstash elasticsearch cluster, but to the cirrus elasticsearch cluster, presumably for consumption by https://en.wikipedia.org/wiki/Special:ApiFeatureUsage. While the cirrus cluster was down, we saw timeouts in the logstash logs. The hypothesis is that this blocked enough threads that the logstash ingester threadpool was saturated and basically stopped processing anything.
It is not clear that lgostash provides way to implement a circuit breaker in case elasticsearch is down. The resurrect_delay option might be useful.
== Actionables ==
* More investigation should be done
Incidents/2017-09-22 REST-API-alert-pageimage-update

== Summary ==
The pageimage property of the Barack Obama page disappeared after a quick vandalism revert. This caused it to go missing in the action API response, which in turn triggered a monitoring check on the REST API, which is using the Obama page, and expects a thumbnail (based on pageimage) to be present.
Impact: None beyond the scary alert, and temporary loss of the thumbnail for the Barack Obama article.
== Timeline ==
* 18:32:46 UTC: REST API alerts go off, indicating /api/rest_v1/page/summary/{title} (Get summary from storage) is CRITICAL: Test Get summary from storage responds with malformed body (AttributeError: NoneType object has no attribute get)
* 18:45:00 UTC Ops start investigating
* 19:10:00 UTC Daniel calls Eric (PTO) and Gabriel (who was out on lunch). Gabriel starts investigating.
* 19:15 UTC: Gabriel determined that no deploy had happened, and that the main users of the summary end point are working correctly. Stephen Niedzelinsky double checked correct operation of the page preview web feature and app. The missing properties are optional in the response schema, so all clients are expected to handle missing thumbnails.
* 19:38 UTC: Gabriel notices that "thumbnail" and "originalimage" properties are missing from the tested response, but are expected by the related spec-driven check.
* 19:57 UTC: Gabriel acks the alerts. Agreement that the issue is not critical.
* 20:12 UTC: Bernd notices that the pageimage property in the action API response for Barack Obama is empty. This is the page used for monitoring. There had been vandalism & a quick revert on that page just before alerts started.
* 20:30:26 UTC: Gabriel makes a minor edit on the Obama page, and REST API alerts recover immediately.
== Conclusions ==
* PageImage updates are not as reliable as they could be: 
* We might want to switch our monitoring to a page that is less likely to break from vandalism.
== Actionables ==
* Make pageimage updates more reliable: 
* See if we can make REST API monitoring more reliable, and make error messages more specific about which property is missing. 
* Make service-checker-swagger error messages more verbose
Incidents/2017-10-01 labvirt1015

== Summary ==
On Friday, 2017-09-29, Andrew migrated a few VMs to the recently-repaired labvirt1015 virt host. Early on Sunday labvirt1015 crashed, causing downtime for all the hosted VMs.
Some time later the problem was discovered, and the affected VMS migrated to a more trustworthy host. Affected instances were:
* puppet-phabricator
* wdqs-deploy
* tools-clushmaster-01
* search-jessie
* gerrit-test3
* phab-01
* integration-slave-jessie-1003
* integration-slave-jessie-1004
== Timeline ==
* Previously: during initial setup labvirt1015 was found to be unexpected halted several times. This issue was investigated and (we thought) fixed; this process is documented in T171473. During that process, Andrew downtimed the instance for 30 days to silence the alerts caused by both the crashes and the frequent reboots that were part of the debugging and repair process.
* 2017-09-19: Labvirt1015 has a new CPU, has been up for several days and is declared fixed.
* 2017-09-29: Labvirt1015 is still up and seems fine. As part of T176044 andrew rebuilds it with a known good kernel, and begins evacuating VMs from labvirt1016 to 1015 in order to free up 1016 for a rebuild.
* 2017-10-01 01:21:01: Labvirt1015 halts, taking all hosted VMs with it. This failure does not page or alert because the 30 days of downtime are still in progress.
* 2017-10-01 09:06: A user on IRC (paladox) reports a VM outage. The message is missed because it's in the early hours of Sunday morning for the WMCS team.
* 2017-10-01 19:00: Madhu notices that the host is down and that the downtime is affecting hosted instances.
* 2017-10-01 22:00: WMCS team converses on IRC and Whatsapp, agrees to reboot labvirt1015 and evacuate VMS.
* 2017-10-01 23:00: All VMs save one are migrated to labvirt1017 and resume normal function. The remaining VM is enormous and takes several hours to copy, but completes migration later in the evening.
== Conclusions ==
* Labvirt1015 is not actually fixed, but its failure is intermittent and sneaky and it will be hard to know when we can trust it again.
* The only real mistake here was the silencing of Icinga warnings for labvirt1015, that resulted in a considerable delay before we responded to the outage.
== Actionables ==
* Replace labvirt1015
* Include a review of current icinga acks and downtimes as part of closing any hardware-related task.
Category:Incident documentation
Incidents/2017-10-17 webm-audio-mediatype

== Summary ==
audio/webm files were incorrectly labeled as img_media_type="VIDEO" & img_major_mime="video".
After having fixed the cause (https://phabricator.wikimedia.org/T156135), there were a few old entries in the database that had to be fixed (image: 30, oldimage: 4, filearchive: 94).
Because of the small amount of rows to be updated, misjudged the impact.
== Timeline ==
 logs in to tin & connects to mysql commonswiki.
~12:58: runs UPDATE image SET
img_media_type="AUDIO", img_major_mime="audio" WHERE img_media_type="VIDEO"
AND img_major_mime="video" AND img_minor_mime="webm" AND img_metadata LIKE
'%s:9:"mime_type";s:10:"audio/webm";%';
Query OK, 30 rows affected (0.35 sec)
Rows matched: 30 Changed: 30 Warnings: 0
~13:17: runs UPDATE oldimage SET
oi_media_type="AUDIO", oi_major_mime="audio" WHERE oi_media_type="VIDEO" AND
oi_major_mime="video" AND oi_minor_mime="webm" AND oi_metadata LIKE
'%s:9:"mime_type";s:10:"audio/webm";%';
Query OK, 4 rows affected (7.45 sec)
Rows matched: 4 Changed: 4 Warnings: 0
~13:20: runs UPDATE filearchive SET
fa_media_type="AUDIO", fa_major_mime="audio" WHERE fa_media_type="VIDEO" AND
fa_major_mime="video" AND fa_minor_mime="webm" AND fa_metadata LIKE
'%s:9:"mime_type";s:10:"audio/webm";%';
Query OK, 94 rows affected (8.46 sec)
Rows matched: 94 Changed: 94 Warnings: 0
~13:25: & notice lag on s4 (https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1053&var-port=9104&from=1508243685418&to=1508248037877)
~13:25: forgot to !log what he was doing: there was confusion, and execution of another (unrelated) script was halted as a result
As a general comment, the reason why it has so much impact was because, despite the number of rows being very low, the actual number of rows read was very high (probably most or a significant part of the table, which not only was read, but also blocked, preventing other writes to go on, and blocking replication on the non-masters. There are scripts such as pt-archiver that assure writes are done by primary key on very small batches, ensuring minimal impact (e.g. for deleting >30M rows on commmonswiki https://phabricator.wikimedia.org/T177772#3670119 ).
== Conclusions ==
* Scripting should properly have been done instead of running from CLI
* DBAs should have been pinged, who are in a better position to judge impact & advise
* !log should have been used to warn what was about to be made
* Before performing database edits- backups should be performed to allow easy reversions
* Should dangerous maintenance scripts executions be limited/managed differently/documented better?
Category:Incident documentation
Incidents/2017-11-25 statsd

== Summary ==
A change in service-runner was deployed on Nov 22th which caused an ever-increase amount of metrics to be sent to statsd, eventually overwhelming the machine with UDP traffic.
== Timeline ==
* 20171121 Report GC metrics from service-runner pull request is merged, https://github.com/wikimedia/service-runner/pull/170
* 20171121T1344 cpjobqueue is deployed with GC metrics reporting enabled, http://tools.wmflabs.org/sal/log/AV_e0yGGF4fsM4DBdggW
* 20171123T0558 cxserver is deployed with GC metrics reporting enabled, http://tools.wmflabs.org/sal/log/AV_ndQBAwg13V6286Gp_
* 20171125T1117 The "carbon frontend relay drops" alarm starts going off
* 20171125T1305 Filippo starts investigating and is open
* 20171125T1340 statsd traffic from scb machines is banned to let statsd recover
* 20171125T1410 cxserver and cpjobqueue are roll-restarted to alleviate the metrics "leak"
* 20171125T1545 root cause is found and Petr rollbacks service-runner for cpjobqueue
* 20171125T1605 scb statsd traffic is unbanned
* 20171125T1609 Kartik rolls back cxserver with the previous version of service-runner
== Conclusions ==
Services sending statsd UDP traffic receive no feedback from the aggregation server, thus the client needs to do self-pacing in order to avoid overwhelming the aggregation server. Also the aggregation server had alerts for its graphite carbon traffic, i.e. after statsd aggregation has happened. Said alerts fired only when the machine was overwhelmed with network traffic and carbon traffic (over tcp) was impacted too.
== Actionables ==
* Alert on statsd udp loss (phab:T181382)
* Adaptive metrics auto-sampling in service-runner (phab:T181382)
Incidents/2017-11-30 wdqs

== Summary ==
Around 14:55 UTC wdqs1004 was caught in GC death spiral and froze. It recovered after a restart of blazegraph.
== Timeline ==
* 14:55 UTC: slowdown in updates can be observed for wdqs1004
* 15:15 UTC: icinga alert: LVS HTTP IPv4 on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds
* 15:15 UTC: icinga recovery
* 15:19 UTC: restart of blazegraph on wdqs1004
== Conclusions ==
* Looking at GC logs, I can see a peak at 17GB/s of heap allocation. This looks related to the traffic received. Much more investigation will be needed to get to the bottom of this.
* Looking at throttled requests during that period, I can see that most requests are coming from user agent "MediaWiki/1.31.0-wmf.10". This is a surprise to me.
== Actionables ==
* modify the local icinga checks to use the same check as LVS, which do a real query and not just a call to a dummy page phab:T181989
* new wdqs cluster, dedicated to synchronous and trusted traffic phab:T178492 (this is a goal of search backend for next quarter)
* investigate memory allocation on blazegraph phab:T181988
* investigate and document clients of wdqs, a tracking page has been created.
Incidents/20170617-Phabricator-spam

== Summary ==
Over the weekend of June 17th, 2017 our Phabricator instance's file upload feature was used to upload non openly licensed stuff ('pirated content') by multiple users. This is another example of people using the free rating via Wikipedia Zero to share pirated content.
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
2017-06-17
* 14:55 - Framawiki created the task: "Cleanup phabricator.wikimedia.org uploaded files, WP zero abuse" - 
* ... - volans and Reedy deleted a lot of file content and disabled approx. 110 recently created user accounts with mostly Moroccan IP addresses
* ... - Framawiki, Mainframe98, Paladox, Peachey88, Reedy, zhuyifei1999, Zppix closed dozens of tasks created by these accounts
* 19:53 - Reedy emailed the Operations mailing list (subject "Phab WP0 file upload spam")
* 21:12 - Aklapper set Phabricator's auth.require-approval bit so all newly created accounts had to be approved by an admin before they could use the service
2017-06-18
* 09:13 - 20after4 submitted D687 - OAuth: Don't authenticate accounts which are blocked in MediaWiki 
2017-06-22
* D687 deployed
2017-06-23
* volans deletes 49 more files that were missed previously. 
== Conclusions ==
What weakness did we learn about and how can we address them?
* Phabricator makes admins delete content via the command line.
* We still allowed users who were blocked onwiki to create Phabricator accounts, making blocking users harder.
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* Disallow blocked users on mediawiki to create accounts on phabricator - 
** This should also probably apply to wikitech ldap accounts, however, that may require changes to wikitech or the configuration of ldap servers - 
* Cleanup phabricator.wikimedia.org uploaded files, WP zero abuse - 
* (Temporarily) Block certain IP ranges used by mobile providers - https://gerrit.wikimedia.org/r/#/c/363001/
* Remove phabricator.wikimedia.org from WP0 zero rating - https://gerrit.wikimedia.org/r/#/c/363264/
* Revert Block certain IP ranges used by mobile providers - https://gerrit.wikimedia.org/r/#/c/363356/
== Related links ==
* T84 - Make sure Phabricator's anti-vandalism features are up to snuff
* T129845 - Commons and to a lesser extent other projects used as video hoster / file sharing site by Wikipedia Zero
* T167915 - Disable media for Morocco Wikipedia Zero
* Phabricator tools:
** Note that accessing some of these links require Phabricator administrator rights.
** Created Phabricator user accounts sorted by newest date
** Files uploaded to Phabricator by newest date (does not list deleted files)
** Log of all users' log-in/log-out activity by newest date
** Query log-in activity per user or IP range
** Phabricator Configuration: auth.require-approval
Category:Incident documentation
Incidents/20170629-ganeti

Various services were unavailable for ~40 mins
== Summary ==
A human mistake in the eqiad ganeti cluster caused widespread issues in various services. Services impacted:
 * poolcounter
 * OTRS
 * LDAP for labs/authentication for various tools (mostly performance as the failover to the other LDAP happens)
 * Grafana
 * mx1001 (minor as incoming emails are bound to be routed to mx2001)
 * etherpad (minor)
 * bromine (various static microsites)
 * kubernetes (unimportant, no services there yet)
The mistake uncovered issues in our remote access infrastructure.''
== Timeline ==
* 15:20 akosiaris in the final cleaning up steps of a migration to multi-row ganeti in eqiad manages to remove networking for 3 ganeti hardware nodes. This removes networking from all VMs on those hosts. Realizes this immediately and informs others. Tries to login to the iDRACs of above hosts in order to fix the issue manually, but all 3 host's iDRAC does not respond.
* 15:21 Puppet alerts start pouring in as one of the services on the disconnected VMs is puppetDB.
* 15:25 Realization that racktables is down, LDAP auth is down.
* 15:30 poolcounter1001 gets depooled from mediawiki config as it is own of the VMs.
* 15:33 more hosts iDRAC's are found out to not be replying to requests. It becomes clear problem is widespread, but no root cause is known.
* 15:39 ganeti1002 has been forcefully rebooted (flea power drain). iDRAC is still not responding, but host is responsive again
* 15:40 akosiaris tries to disable puppet agents in eqiad, esams using cumin to avoid some icinga spam but fails as puppetdb is unresponsive. While some alternatives are proposed, the task is not deemed worthy enough to justify more investigation/work at that point in time.
* 15:46 It becomes clear we have 50x problems with upload. It's thumbor using only one poolcounter server and that is poolcounter1001.
* 15:48 akosiaris tries to failover to the last remaining ganeti node the VMs aluminium.wikimedia.org bromine.eqiad.wmnet etherpad1001.eqiad.wmnet mx1001.wikimedia.org seaborgium.wikimedia.org. Apart from seaborgium, none managed to as they have degraded disks on said host. This has happened because ganeti1001 was rebooted a few mins before the human error and was still syncing the disks. LDAP is back up however
* 15:56 Everything is back, recoveries start pouring in
* 19:06 poolcounter1001 gets repooled
== Conclusions ==
* Humans are the weakest link
* Our remote access infrastructure has been having serious problems, to the point we could not use it when most needed and we did not know
* Thumbor only uses 1 poolcounter server
== Actionables ==
* Thumbor should use >1 poolcounter ()
* Thumbor shouldn't fail when poolcounter fails ()
* Thumbor should alert/page when thumbs aren't rendered ()
* Cumin masters: simplify usage in case of emergency 
* A single PuppetDB per-datacenter going down shouldn't impact puppet runs ()
* iDRACs unresponsive ()
* OTRS in a HA/Failover Setup ()
* Rack documentation app (racktables) in a HA/Failover Setup 
Category:Incident documentation
Incidents/20170917-MediaWiki-API

== Summary ==
The mediawiki cluster dedicated to serving the API had a sudden rise in cpu usage on Saturday September 16th; it was noticed from the restbase latency report and solved by a rolling restart of the HHVM service. The reason for the excessive CPU usage was due to multiple HHVM threads running at 100% CPU usage while not responding to clients. A quick analysis hinted this to be a deadlock between multiple threads in accessing/modify the Translation Cache.
The effect for user was a higher latencies when performing requests to the action API 
== Timeline ==
* September 15th 16:20 (approx): the load begins to rise on the API cluster 
* September 16th 07:45: The CPU usage reaches its maximum across the cluster, peaking at about 87%, more than tripled with respect of the normal operating usage of about 25%. The latency of the mediawiki API begins being almost constantly very high, according to telemetry from restbase . It is interesting to note that in all this time, no alert from pybal for depooling of API appservers happen.
* September 17th 11:59: mobrovac notices the high latencies for the 99th percentile of requests for restbase, and notifies others on IRC. 
* September 17th 12:34: _joe_ acknowledges the problem, starts investigating
* September 17th 12:40: Entry in SAL _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
* September 17th 13:00 _joe_ analyzing a backtrace notices a lot of references to HPHP::jit::enterTCImpl in the blocked threads, but decides to fight the fire at this point
* September 17th 13:10: After assessing the restarts are effective, _joe_ rolling restarts the rest of the cluster, and by this time the restbase alerts have recovered
* September 17th 13:25 After all appservers have been restarted the CPU load is back within normal parameters
== Conclusions ==
While this seems like some serious issue with HHVM, such crises can and will happen from time to time: what didn't work at all is our alerting - the occurrence during the weekend delayed our response time and exposed the inadequancy of our monitoring pretty clearly. While the restbase telemetry was a useful and pretty obvious to a human eye, it is hardly a good metric to alarm as it can vary greatly. The metrics collected about API performance are stable enough to be acted upon, but they only show part of the API requests. We do need better metrics, and to be able to act upon them.
== Actionables ==
* Measure the overall performance of the Mediawiki application at the appserver level T131894
* Create meaningful, actionable, non-flapping alerts based on backend timing information (see T123854, but the suggestions there don't really cut it for the kind of reliable alerting we want)