nyuuzyou/wikis huggingface dataset:
Incidents/20160509-CirrusSearch

== Summary ==
At 21:43 UTC on 2016-05-09 Elasticsearch started to slow down (as seen on Grafana). Investigation showed a high CPU consumption on elastic1026. Elasticsearch service was restarted and response times went back to normal. Investigation trace the cause to a large segment merge and garbage collection going crazy on elastic1026.
== Timeline ==
* 2016-05-09T21:43 increase in overall response time (95%-ile) on elasticsearch
* 2016-05-09T21:51 issue with search reported on IRC by odder
* 2016-05-09T22:14 elasticsearch service restarted on elastic1026
* 2016-05-09T22:20 response time back to normal
== Analysis ==
* more details on Phabricator
* We did not get an alert via Icinga. There is currently a check on response time, but it is done on prefix search, which now has a low volume of traffic. This shows again the fragility of Graphite checks.
* Analysis of GC timings indicates that time was spend mainly in young GC and that memory was recollected. This is usually an indication of a too high memory throughput, not of a memory leak or a too small heap.
* GC is strongly correlated with a large segment merge on elastic1026. This does not explain why it was an issue only this time.
* Graphite currentAbove() function is a good tool to identify which server is under more load.
== Actionables ==
* Task created: Enable GC logs on Elasticsearch JVM which might help in a more detailed analysis if a similar issue happens again. Note that the size of those logs might be an issue.
* Task created: Check Icinga alert on CirrusSearch response time, which might lead to either fix this alert or remove it completely
Category:Incident documentation
Incidents/20160512-LabsLdapOutage

== Summary ==
On the evening of May 12th, Andrew merged a patch refactoring LDAP config. Due to an oversight, that change resulted in puppet setting the ldap host to 'undef' on all labs systems.
Lack of an ldap host broke logins for all users apart from those with installed root keys. It also caused puppet runs to fail on hosts with self-hosted puppet.
The primary issue was resolved about 30 minutes later. Self-hosted puppet instances had to be fixed by hand, but were all fixed within an hour or two.
For the most part there were no user-facing consequences from this incident. The grid engine was briefly distressed, but it's unclear who or what that affected.
== Timeline ==
* [23:06] Andrew merges https://gerrit.wikimedia.org/r/#/c/288539/
* [23:33] Users begin reporting that labs instances are providing a password prompt on login
* [23:56] Chase calls Andrew, who appears
* [00:06] Andrew writes and merges https://gerrit.wikimedia.org/r/#/c/288555/ which resolves ldap issues on most hosts
* [00:30] Yuvi fixes some grid-engine fallout from the ldap failures
* [00:00 - 1:30] Andrew, Yuvi and Alex Monk apply various piecemeal fixes to instances with self-hosted puppet. In most cases all that's needed is adding the proper ldap host to puppet.conf and a restart of the puppetmaster. A lot of extra time is spent on deployment-puppetmaster which turns out to have an unrelated issue -- the /var/lib/git/labs/private repo has an unmerged change which prevents updating, resulting in the ldap password failing to be updated by puppet.
== Actionables ==
* The ldap and login failures did not result in any paging. That is addressed by https://gerrit.wikimedia.org/r/#/c/288603/
* We may want to improve monitoring for self-hosted puppetmasters. Specifically, when the auto-rebase tool fails, some sort of notice should be sent so that people know that the puppet state of that host is in peril.
Incidents/20160525-Toollabs Webservices

== Summary ==
An incomplete deploy of the toollabs webservice package caused restarting webservices to not register themselves with the proxy, causing an outage that lasted about 90mins.
== Timeline ==
# https://gerrit.wikimedia.org/r/#/c/290612/ gets merged 
# New toollabs-webservice package gets built and deployed on to all the bastions & webgrid nodes.
# Yuvi tests it with a few restarts, is happy, goes away
# Things happening at this point:
## webservice-runner (which is what the gridengine runs) now has a new parameter (--register-proxy) that is *required* to tell it to register with the proxy. 
## Grid has jobs running that *do not* have this parameter specified
## This is fine for right then, but as soon as they restart for any reason, they'll use the new webservice-runner script and lose their proxy registration, since the --register-proxy isn't passed
# For unknown reasons, some or all tools are restarted by webservicemonitor
# Puppet's ensure => latest is used for deploying toollabs-webservice package, and puppet is stuck on /public/dumps on tools-services-01, which runs webservicemonitor. So even when webservices get restarted by webservicemonitor, they do not get the --register-proxy flag since it's still the old version of the code on tools-service-01 but new version on the gridengine nodes.
# the qmod -rj run on all webservice nodes exaggerates this problem, since it continues to run them without the --register-proxy parameter, causing them to not register
# Yuvi updates package on tools-services-01, and deletes all webservice jobs. This causes webservicemonitor to bring them back up by launching new jobs, which do have the --register-proxy flag
== Conclusions ==
# Tools webservice infrastructure has a number of moving parts that are kinda fragile. The port assignment + proxy registration is particularly so
# Debian packages + ensure => latest in puppet do not make for a 'deployment system'
# Puppet staleness / errors should be looked at with more frequency on the tools project
== Actionables ==
# Switch to an actual deployment system for toollabs-webservice 
# Enable email nags for tools puppet failures 
# Reduce probability of unrelated NFS failures causing puppet / other issues (https://phabricator.wikimedia.org/T136222 and others)
# Better paging for tools webservices being down (Volunteer noticed: T136162)
Category:Incident documentation
Incidents/20160531-wikilabels

== Summary ==
Wikilabels wasn't been able to do some tasks in the past ten days. It was related to CORS changes that haven't been fully tested.
== Timeline ==
SAL
=== 2016-05-19 ===
* 12:51 UTC: Amir1 deployed the new version of Wikilabels into production
=== 2016-05-29 ===
* 14:52 UTC: An issue in github created that it's not possible to request a workset.
* 15:32 UTC: halfak creates the task
* 15:34 UTC: halfak responds in the github 
* 16:01 UTC: First fixes goes to production.
* 17:36 UTC: Final fixes goes to prodcution
== Conclusions ==
* For people who were using the Wikilabels js from meta (and not using the loader from the server). It wasn't possible to do any write actions. I didn't have access to copy-paste the new wikilabels to Aaron's subpage thus I skipped this step. Won't happen again.
* Authentication wasn't possible at all. In other words, requests to https://labels-staging.wmflabs.org/auth/ wasn't being accepted due to lack of cookies. It got solved in 08aadf590d79ba1d2b9455d24ddfe64355a1ae0b
* Assigning tasks wasn't possible. It got solved in 
== Actionables ==
A broad actionable item is to test every possible aspect of patches before going to prod when a big change is coming.
More special actionables:
* Fix access issue with halfak's subpage. (phab:T137626)
* Write and deploy more CI tests, specially for oauth. A test unit for service is much more desirable.
Category:Incident documentation (phab:T137625)
Incidents/20160601-CentralNotice

== Summary ==
TL;DR:
* Mukunda failed to follow up with Bartosz after the deploy (or verifying the fix himself)
** This would have caught the lack of submodule update mistake.
* The patch in question was a patch to wmf.3 that was not applied to master before the wmf.4 branch cut (hence some of the confusion).
Longer:
There was an issue in CentralNotice () that was unexpectedly also in wmf.4 (as the patch that was made to wmf.3 wasn't done in master before the wmf.4 branch cut). Bartosz asked Mukunda to deploy the fix for wmf.4 at 21:52. Mukunda deployed the revert (or so he thought: he failed to do a `git submodule update`) after a ping from Bartosz at 22:30 UTC (after pining Bartosz that he was deploying at 22:17). At this point Mukunda believed the fix was deployed and since he notified Matma that he was doing the deploy took it to mean everything was done. Mukunda moves on to preparing for the Phabricator upgrade (coming at 00:00).
A little over an hour later (23:21) Bartosz pings Mukunda to see if it was deployed as the symptoms were still present. Without response, forty minutes later Bartosz complains again in -operations and get's others' attention. At that point Adam Wight said "MatmaRex: We're paying attention and trying to set up a lightning deploy right now."
== Timeline ==
(UTC)
: 21:52 - Bartosz asks Mukunda to deploy the fix for wmf.4
: 22:17 - Mukunda pings Bartosz that he's deploying
: 22:30 - Deployment completed (which did not include the fix, failed to `git submodule update`)
: 22:30+ - Mukunda moves on to preparing for the Phabricator upgrade
: 23:21 - Bartosz pings Mukunda saying it was not fixed
: 00:08 - Bartosz pings again
: 00:10 - Adam W confirms he/Fundraising is also aware and working on fixing it
: 00:26 - Adam deploys (for real this time) the fix
== Conclusions ==
* All deployers, including Mukunda, should explicitly verify that every deployed patch was A) actually deployed and B) had the intended effect.
== Actionables ==
* All deployers, including Mukunda, should explicitly verify that every deployed patch was A) actually deployed and B) had the intended effect.
* Mukunda will look into moving the Phabricator upgrade window to not conflict as much with the evening SWAT window
* "Spike: Plan reforms of the CentralNotice deployment branch" - 
Category:Incident documentation
Incidents/20160606-otrsmail

== Summary ==
A broken clamav update prevented mail delivery on ticket.wikimedia.org (OTRS) for about 11 hours. The new clamav package deprecated a config option, but if that option is still present (which was the case in our puppetised config), clamd and freshclam refuse to start. No mail was lost, the queue was processed once the option was removed.
== Timeline ==
* 08:39: Moritz deploys the new clamav version as part of the jessie 8.5 point release
* 18:55: The error is reported by an OTRS admin in wikimedia-tech
* 19:14: Alex M pokes ops, Rob and Brandon start investigating
* 19:44: Brandon deploys a hotfix (and one verified via puppet)
== Conclusions ==
* Monitoring didn't spot the error, the "OTRS Icinga" check didn't flag an error and we're missing Icinga checks for ClamAV and FreshClam
* The problematic behaviour wasn't noticed before deploying the update, all further ClamAV updates need more scrutiny (ClamAV is handled differently in Debian compared to other packages: Due to sometimes nontransparent security changes clamav is always updated to the latest version instead of applying isolated changes. Also, virus pattern updates often need newer scan engine features)
== Actionables ==
* Icinca checks for ClamAV and FreshClam, double check OTRS Icinga plugin behaviour if ClamAV fails ()
* Debian update to handle the presence of AllowSupplementaryGroups gracefully (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826607)
Incidents/20160608-SCB

== Summary ==
After a planned, but uncoordinated Zookeeper restart whose purpose was to increase the maximum number of allowed connections, the Zookeeper client in the ChangeProp service reconnected at an excessively high rate. ChangeProp also processed Kafka events in the backlog at an elevated rate, which led to a user-facing outage affecting services sharing the same hardware.
After checking the logs and discussing with several ops in charge of traffic, the user facing issues observed on the logs were minimal due to: a) the cache layer serving most of the queries to non-logged in users (~90% of the total traffic) b) Only certain, and not all, logged users' actions being potentially affected: visual editor editions, graphoid users, etc. c) mobile users either using or failing back to the MW API.
== Timeline ==
12:59 - https://gerrit.wikimedia.org/r/293298 gets merged and all nodes in the Zookeeper cluster are restarted by Puppet.
13:09 - Icinga alert regarding scb services, affecting citoid, the mobile content service, graphoid, mathoid & cxserver.
13:18 - Powercycling scb1001
13:24 - Incinga recovery on scb services
13:26 - Powercycling scb1002
13:29 - Change-prop stopped on scb1001
13:30 - Change-prop stopped on scb1002
5xx response rates from RESTBase during the outage
== Conclusions ==
* The node kafka driver is not very robust. While this was known, we should accelerate the search for alternatives, possibly in the form of a decent binding to librdkafka.
* We need to be more thorough about testing failure handling code. This outage revealed several issues in how the node kafka driver handles connection errors, which would have likely been discoverable in a test environment. Tools like ChaosMonkey can help by injecting failures in test environments & production.
* Isolation between services on shared service clusters is not sufficient. A mis-behaving service can cause an outage for other services on the same hardware by consuming excessive RAM or CPU.
== Actionables ==
* Kafka-node: Rate-limit Zookeeper reconnect attempts in the current kafka client (Status: Done)
* Zookeeper conf: limit the number of client connections to 1024 (Status: Done)
* Change Prop: limit message concurrency to 30 down from 100 (Status: Done)
* Change Prop: start only 8 workers per node (Status: Done)
* Kafka-node: Avoid opening new Kafka connections on each Zookeeper connection retry (Status: Done)
* Puppet: Avoid uncoordinated Zookeeper restarts (Status: Done; manual restarts for now)
* Improve isolation in scb cluster. Use cgroups / containers?
* Replace the node-kafka client with a binding to librdkafka (Status: Todo)
* Create general guidelines & processes to ensure thorough fault testing of services, both pre-deployment & later in production. (Status: Todo)
Incidents/20160610-ORES

This is a template for an Incident Report. Replace notes with your own description.
== Summary ==
ORES was down for an unknown amount of hours today due to a broken configuration file (99-redis.yaml). 
== Timeline ==
* ??? -- https://github.com/wikimedia/operations-puppet/commit/78119152c47b7873fdd7bd0c38a356b5bff27226 was merged to the production branch of wikimedia puppet
 at least 6 hours passes 
* 2016-06-10 @ 1930 UTC -- 503 errors and timeouts were noted
* 2016-06-10 @ 2030 UTC -- 99-redis.yaml files are deleted and the workers are restarted. Service is restored. 
== Conclusions ==
https://github.com/wikimedia/operations-puppet/commit/78119152c47b7873fdd7bd0c38a356b5bff27226 should not have been merged. We need a better testing process around puppet merges to make sure that they don't take down the service. Unlike a deploy, there's to a clear event at which puppet is run. 
Also, this downtime did not cause a paging event. 
== Actionables ==
* Investigate why we were not paged when the downtime started Phab:T137592
Category:Incident documentation
Incidents/20160620-ores

== Summary ==
ores.wikimedia.org was down today for about twenty minutes because of deploying a commit that changed reading config directory without proper order.
== Timeline ==
SAL log 
* 10:58 Amir1: deploying bdc1e2b in ores nodes
* 11:04 deployment finished and ores went down
** puppet agent ran and services got restarted (uwsgi-ores, celery-ores-worker). Didn't solve the problem
** Checking logs showed the problem persists due to bad config reading
* 11:27 Amir1: rollbacking ae71d842dfc0958e06922062dd09d49243332a6a
** ORES went live again
* 12:13 Amir1: deploying bdc1e2bd only to ores on scb2001 (codfw) 
** Did not work as expected. (No down time because it only affected that node in codfw).
* 13:04 Amir1: deploying 8e65182 to scb2001
* We fixed it in 
** Worked perfectly fine
* 13:06 Amir1: deploying 8e65182 to all ores nodes
== Conclusions ==
A very shallow reasoning would be the issue of reading config directories which got changed a lot and now it's in a rather stable situation but that's dangerous. What we really need is a safe method to deploy ores which we did the second time today. The only thing is documenting them 
== Actionables ==
* Document safe steps to deploy ores in prod ()
Category:Incident documentation
Incidents/20160628-wdqs

Switching WDQS to Scap3 deployment caused some issue and a downtime of the service.
== Summary ==
Tuesday June 28 around 19:30 UTC we migrated WDQS deployment to Scap3. Scap3 deployment replaces the application directory by a symlink, leading to files outside of scap in that directory to disappear. This lead to WDQS failing for about 20 minutes between 19:53 UTC and 20:11 UTC.
== Timeline ==
* 19:53: WDQS deployed and restarted by Scap3, service failing
* 19:58: issue tracked to missing symlink to wikidata.jnl (blazegraph data file)
* 19:59: running puppet to restore the missing symlink -> did not work (failed dependency between due to application folder now being managed by Scap3)
* 20:08: stopped puppet while restoring service manually
* 20:11: Service restored by manually restoring symlink
== Notes ==
* Migrating WDQS to scap looked like a simple operation. Plenty of services have already done this migration without issue.
* puppet agent --noop is a great tool especially in non standard situation (Gehel did not use it, but should have).
* Restoring service did take longer than expected. Gehel was not familiar enough with the details of how WDQS work and took some time double checking it before making changes.
== Conclusions ==
* Scap3 supports canaries, they should *always* be used.
== Actionables ==
* Use canaries for WDQS deployment https://gerrit.wikimedia.org/r/#/c/296459/.
* Modify puppet now that WDQS application directory is managed by Scap3 https://gerrit.wikimedia.org/r/#/c/296446/.
* WDQS scap deployment should recreate symlink https://gerrit.wikimedia.org/r/#/c/296445/.
* Migration to Scap3 documentation should emphasize the use of canaries more. Documentation already exists, but we probably need more emphasis, or a checklist for migrated services See new Scap3 Migration Guide.
* Cleanup to puppet code to not duplicate what is managed by scap3, or refactor it to move all mutable files out of the application directory. 
Category:Incident documentation
Incidents/20160713-ContentTranslation

== Summary ==
ContentTranslation extension used up all available SQL connections causing issues for extensions using the shared database (at least Flow, Echo). This was caused up by thousands of hits in short time to the translation draft saving API by on user. What caused those requests is still unknown.
== Timeline ==
See also .
* 2016-07-12 [15:48:53] (UTC) icinga-wm> PROBLEM - MariaDB Slave SQL: x1 on db1031 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
* [15:50] jynus notices x1 master is down
* [15:52] jynus identified ContentTranslation\Translation::update / cx_translations as the cause
* [15:59] https://phabricator.wikimedia.org/T140123 is created by jynus
* [16:05] Language team developers were pinged in IRC
* [16:10] Nikerabbit notices the ping and starts checking what is going on
* [16:12] Patch to disable CX is deployed
* [16:19] Nikerabbit checked the code and believes it can only be caused by high number of API hits (no loops in the relevant code etc.)
* [16:28] Nikerabbit is able to confirm this is true via https://graphite.wikimedia.org/render/?width=586&height=308&target=MediaWiki.api.cxsave.executeTiming.count static copy. Based on other data we assume these are all coming from one or at most few users.
* [16:30-16:35] Brainstorming for solutions: blocking the user, poolcounter, ping limiter
* [16:40] Nikerabbit thinks ping limiter is the best option and starts working on the patches, ostriches indicates he can help with review and deploy
* [16:54] Nikerabbit has finished the patches: https://gerrit.wikimedia.org/r/#/q/topic:cxsave,n,z
* [16:55-17:45] Patches are being reviewed and deployed. Tests for CX extension patch takes 10+ minutes and they need to run multiple times.
* [17:48-18:08] Patch to turn CX back on is created and deployed
* [18:09-18:13] Initially CX does not work. Special pages only display user toolbar, no JavaScript errors. After few minutes it starts to work, indicating it was caused by the 5 min caching.
== Conclusions ==
* Surprised there is no general request limiting automatically applied, but code changes were required.
* Language team has no monitoring in place that would have detected this issue.
== Actionables ==
* The ping limiter which was added can stay in place. The values should be verified. T140615
* Audit the front-end code for possible causes of these requests. T140622
* Try to contact the user for more information and advice T140619
Category:Incident documentation
Incidents/20160713-MediaWiki

== Summary ==
On any request that used the PoolCounter feature (search, rendering of pages on view), users experienced a 503 due to a HHVM fatal error.
This was caused the because the duplicate PoolCounterClient.php entry point referenced in the MediaWiki configuration was removed in favor of the properly named PoolCounter.php one and extension.json. Reedy had prepared a patch to do the switch, but it hadn't been deployed yet, and the wmf.10 train went ahead. Furthermore, the entry point was using PHP's "include" and not "require", meaning that PHP would not fatal if the file was missing, so we didn't notice it immediately. It would only fatal when something actually invoked PoolCounter, leading to intermittent user facing errors, depending on which articles they were viewing.
After checking fatal.log and seeing the error, Legoktm fixed up Reedy's patch and deployed it.
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
* 17:48 First report by Steinsplitter in #wikimedia-operations about getting 503s
* 17:50 Confirmed by others in channel on the <https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions> page
* 17:53 legoktm looks at fatal.log on fluorine and sees errors like:
*:2016-07-13 17:53:08 [V4aABApAMEoAAFRU2eIAAABO] mw1239 mediawikiwiki 1.28.0-wmf.10 fatal ERROR: [5c5c2c0c] PHP Fatal Error: Class undefined: PoolCounter_Client
* 17:56 legoktm fixes, rebases and merges https://gerrit.wikimedia.org/r/298096 ("PoolCounterClient.php -> extension.json")
* 17:57 legoktm syncs to mw1017 and tests using X-Wikimedia-Debug
* 17:58 legoktm deploys to all servers
* 18:00 confirmed fixed by #wikimedia-operations channel members
== Conclusions ==
What weakness did we learn about and how can we address them?
This should have been noticed much sooner. We should have noticed in beta that the file was missing, and then again when wmf.10 was first deployed
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* Set up PoolCounter on beta cluster ()
* Audit configuration to make sure all extensions are being loaded using "require" and not "include"
* The proper canary deploy stuff that would notice fatals earlier...
* …
Category:Incident documentation