nyuuzyou/wikis huggingface dataset:
Incidents/20161204-labservices1001

== Summary ==
At about 02:20 on 2016-12-04, labservices1001 become unresponsive and unreachable. It was rebooted and returned to normal service about 20 minutes later.
Any new VMs created during the outage window (including those used for CI) would have failed to register DNS records -- as it happened, no VMs appear to have been created during the window, so there were few user-facing consequences from this outage. Labservices1001 is also a DNS server, but most DNS traffic fell to the secondary server, labservices1002, which successfully handled it.
The cause of the system crash is yet undetermined. Additionally, there were several monitoring and paging issues revealed by this incident that require investigation.
== Timeline ==
* 02:17 Labservices1001 locks up
* 02:19 Page: 'toolshecker service itself needs to return OK on checker.tools.wmflabs.org os CRITICAL: HTTP CRITICAL: HTTP1.1 502 Bad Gateway'
* 02:21 Given that the toolschecker web service is itself down, a series of other related pages are sent over the next few minutes.
* 02:21 Some CI operations begin to fail due to DNS resolution issues:
 02:21:30 git.exc.GitCommandError: 'git remote prune --dry-run origin' returned with exit code 128
 02:21:30 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/core/': Could not resolve host: gerrit.wikimedia.org'
* 02:22 Andrew and Alex Monk confer on IRC but both have limited access for troubleshooting (Andrew didn't have his key and Alex didn't have the necessary privileges).
* 02:25 Filippo arrives on IRC (with full access) and begins investigation.
* 02:35 toolschecker starts sending recovery alerts. Presumably it has figured out to switch over to the secondary DNS server, but labservices1001 (aka labs-ns0) is still down.
* 02:40 Filippo reboots labservices1001 from the management console.
* 02:43 Labservices1001 is back up, all services returned to normal.
== Conclusions ==
The actual failure of labservices1001 is unexplained, and may remain so if it does not recur. For the most part, the redundancy in Labs DNS served us well. There are nonetheless a few pressing concerns:
* Why did the labservices1001 failure not page?
* Why was toolschecker so slow to cope with the loss of labs-ns0?
* Why did CI tests not fail over to labs-ns1 gracefully?
== Actionables ==
* Explain the cause of the hardware failure 
* Fix paging for designate services and DNS on labservices1001 
* Investigate DNS failover for toolschecker 
* Add monitoring for a full Labs instance lifecycle 
* Make sure CI boxes know about both DNS servers 
Category:Incident documentation
Incidents/20161221-ToollabsKubernetesAPI

A non-user affecting Kubernetes API downtime lasting about 1hour
== Summary ==
Kubernetes was down a for a about 1 hour due to a faulty puppet patch, a deployment script not doing much error handling and human error
== Timeline ==
* Dec 21 2016 ~12:10 UTC Alex merges the first (kube-apiserver) of the 3 patches https://gerrit.wikimedia.org/r/326441, https://gerrit.wikimedia.org/r/326430 and https://gerrit.wikimedia.org/r/326429' on tools-puppetmaster
* Dec 21 2016 ~12:15 UTC Alex merges the kube-scheduler one (https://gerrit.wikimedia.org/r/326429) after the kube-apiserver one is succesfull
* 8 Dec 21 2016 ~12:16 UTC Alex notes the kube-scheduler not starting with an error of--leader-elect=true not being a valid parameter. Realizes something has gone wrong version wise and finds out /usr/local/bin/kube-scheduler is now a symlink to /usr/bin/kube-schedular. That's a mistake in the patch, should have been vice-versa. /usr/bin/kube-scheduler is a left over on tools-k8s-master-01 from a previous unrelated deployment method.
* Dec 21 2016 12:20 UTC Alex runs /usr/local/bin/deploy-master since that is the deployment method that was used for that binary in the first place in order to restore the correct version. The script has a bug and truncates /usr/local/bin/kube-apiserver, but that goes unnoticed for a while. 
* Dec 21 2016 12:21 UTC Alex, decides to rollback the kube-apiserver puppet patch as well. Turns out that was a mistake, puppet restarts the service and the empty binary now fails to start (obviously). Now both services are down.
* Dec 21 2016 12:25 UTC Alerting in #wikimedia-labs, alex continues investigation
* Dec 21 2016 12:26 UTC Yuvi offers help
* Dec 21 2016 12:30 UTC After some discussion on IRC, Yuvi kicks off a new kubernetes build to get a working binary
* Dec 21 2016 ~12:35 UTC Alex realizes that puppet clientbucket can save some work and get kube-scheduler restored. kube-apiserver still missing though
* Dec 21 2016 ~12:45 UTC Alex, since everything API is down anyway, tries the 3rd patch just to see if it works (it did!), while waiting for the build to finish. 
* Dec 21 2016 13:03 UTC Yuvi is done with the build partially. The build had failed due to some disk space issue but what at least generated the kube-apiserver, kube-scheduler and kube-controller-manager binaries, but not kubectl.
* Dec 21 2016 ~13:05 UTC deploy-master is ran by Yuvi, deploys the above binaries and truncates kubectl
* Dec 21 2016 13:09 UTC Alex sees everything running now on tools-puppetmaster
* Dec 21 2016 13:10 UTC Yuvi notes kubectl not returning anything. Some confusion ensues.
* Dec 21 2016 13:16 UTC The truncated binary is noticed and a copy is brought in from a worker.
* Dec 21 2016 13:21 UTC Everything is up and running
== Conclusions ==
* Half-baked deployment script are land mines waiting to be stepped on. That specific one is going away
* More communication before deployment of puppet changes can be useful
== Actionables ==
* Have labs use the debian packages for kubernetes
Category:Incident documentation
Incidents/20161227-ores

== Summary ==
ORES wasn't able to score a growing proportion of edits in Wikidata for several weeks.
== Timeline ==
* The quantity change API in Wikibase got deployed in mid-November (probably on November 18). (phab:T133042). Pywikibase didn't catch up and failed on items that have statement without boundaries. It wasn't much but started to grow.
* The failure rate started to grow.
* December 27th, 00:16 UTC Quantity changes broke ORES gets reported.
* 00:53 The fix in ores-experiment.wmflabs.org is pushed and confirmed to fix this issue.
* 05:00 The fix in beta cluster is pushed and confirmed to fix it.
* <SAL> [2016-12-27T05:06:06Z] <Amir1> starting deploy of ores:228b9b4 in canary nodes (T154168)
* <SAL> [2016-12-27T05:14:37Z] <Amir1> starting deploy of ores:228b9b4 in all nodes (T154168)
* <SAL> [2016-12-27T05:25:27Z] <Amir1> finished deploy of ores:228b9b4 in all nodes (T154168)
== Conclusions ==
Unexpected breaking changes can happen all the time. We need to have better monitoring of failure ratio.
 
== Actionables ==
* Clean up failure ratio monitoring and set up an alarm when it goes more than a certain threshold ()
Category:Incident documentation
Incidents/2017-01-04 MonologSpi

== Summary ==
DB shard column added to MediaWiki monolog SPI entries in logging.php with invalid processor callback return value.
== Timeline ==
* 22:27 (CR) BryanDavis: [C: 1] Include DB shard as a logstash column [mediawiki-config] - https://gerrit.wikimedia.org/r/328618 (owner: Aaron Schulz)
* 23:09 (CR) Aaron Schulz: [C: 2] Include DB shard as a logstash column [mediawiki-config] - https://gerrit.wikimedia.org/r/328618 (owner: Aaron Schulz)
* 23:11 aaron@tin: Synchronized wmf-config/logging.php: Include DB shard as a logstash column (duration: 00m 41s)
** This is the root of the problem: the change was deployed 2 minutes after merge without either A) testing in Beta Cluster nor B) testing on mwdebug*, either of which would have (and did in the case of Beta Cluster) caught the issue.
* The "currently experiencing technical difficulties" notice appears for essentially all wiki page access; a local fix was never committed to the existing patch set (possibly due to push failure) and the bug was not noticed in review
** Error: Catchable fatal error: Argument 1 passed to __invoke() must be an instance of array, string given in /srv/mediawiki/wmf-config/logging.php on line 89
** Error spike as seen from Kibana
* 23:14 Aaron notices the error browsing VP/T after sync, others from IRC and SMS pages
* 23:14 Roan starts reverting, locally on deployment host only, and notices some scap errors (which turned out to be irrelevant)
* 23:15 catrope@tin Synchronized wmf-config/logging.php: revert (duration: 00m 41s); outage state is over
** Stashbot did not log this to the Server admin log wiki page
* 23:22 Aaron notices his local mediawiki-config repo is corrupt and unusable for git commands, starts local re-clone.
* 23:49 (CR) Aaron Schulz: [C: 2] "Cleaning branch" [mediawiki-config] - https://gerrit.wikimedia.org/r/330606 (owner: Aaron Schulz).
* 00:00 !log aaron@tin Synchronized wmf-config/logging.php: No-op sync of 7e103f21a3555fc0b8f7fdea4fd8df4cb7cb939e (duration: 00m 42s); deployment host and gerrit repos are back in sync again
== Conclusions ==
* Root cause: A configuration change was not tested on mwdebug* or beta prior to it going out, which would have trivially caught this failure
* Automatically catch in the future: The scap canary check does not currently catch large HHVM fatal spikes due to those being unreliable/noisy. The scap canary check does check exceptions, however.
* Noisy scap import errors can be confusing, especially during something urgent.
** Addressed in 
== Actionables ==
* Configuration changes should be tested on mwdebug* as well as Beta Cluster (when possible) prior to syncing to production--this did not happen
* Account for fatals in scap canary check? This probably will need some kind of filtering to avoid common noise cases. - 
* Make sure SAL logging works during outages? Both Roan's revert of this change, as well as Tyler's revert of the group1 train about 3 hours prior to this incident were not logged there.
Incidents/2017-01-06 Cache-upload

From 16:08 to 18:11 UTC on Friday, 06 January 2017 several requests for images
and other multimedia files resulted in user-facing 503 error messages. The root
cause of this outage was an administrative command being executed on multiple
cache servers in a data center disabled earlier during the morning due to network
issues (codfw). Because of an operator mistake, the command was executed on all
machines at the same time rather than in a rolling fashion, which triggered a
bug in the caching proxies (varnish) on the majority of our CDN nodes,
including those in other data centers. When a caching proxy process crashes it
gets restarted, and its cache contents emptied. The origin servers responsible
for serving multimedia content thus became overloaded and started misbehaving.
In particular, all swift frontends showed high load, as well as the backends
responsible for accounts: ms-be1002, ms-be1005, and ms-be1022.
The issue was solved by directing requests for image thumbnails to the
secondary data center, thus relieving pressure from the overloaded origin
servers in the primary DC.
== Timeline ==
All times UTC.
* 16:05 Administrative command executed
* 16:08 Outage begins
* 16:12 First page received
* 16:17 bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation
* 17:12 root cause identified (caching proxies crashes)
* 18:08 switch thumbs to codfw
* 18:12 all requests back to normal
== Conclusions ==
Depooling and repooling cache nodes has important consequences. It should be
done in sequential order even on depooled data centers. Varnish reloads should
not be considered 100%-safe operations, which is something we
already noticed in the past.
A dashboard plotting the number of objects currently in cache, as well as
alerting on crashing varnish processes, would have helped speeding up
root-cause identification.
Pressure was relieved from swift in eqiad by routing requests for thumbnails to codfw. A knob allowing to choose which percentage of traffic should go to which data centre could be an interesting idea for the future.
On the swift side, the 3 backend servers responsible for accounts have been identified as one the bottlenecks. The swift replication factor for accounts should be increased to allow for more fan-out. Furthermore, we have noticed swift proxy had their CPU saturated during the time of the incident. This is likely due to swift frontends in eqiad having Hyper-threading disabled, it should be enabled across the swift cluster.
== Actionables ==
* Further investigate varnishd crashes 
* Plot number of cached objects on per-server / per-cluster grafana dashboard 
* Increase swift replication factor for accounts 
* Possibly alert on varnishd crashes (TBD)
Category:Incident documentation
Incidents/2017-01-11 multiversion

== Summary ==
The multiversion code is poorly understood by many deployers. The code is complex and the entry points are a mess. An ongoing effort has been underway to address this. On Jan 11th, a fairly involved refactor landed and caused a brief outage, despite testing in beta, on mwdebug*, and the canary checks.
== Timeline ==
* 18:28: Gerrit # was merged
* tested on beta, mwdebug, etc
* 18:56 demon@tin: Synchronized multiversion/MWMultiVersion.php: Attempt #2 for Multiversion cleanup (duration: 00m 41s)
* 19:27 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/FlaggedRevs: Stupid errors (duration: 00m 46s)
** Not technically related, but weird autoloader bugs became more apparent (seen also in TMH) in testing this, so we backported a fix here
* 19:34 demon@tin: Synchronized multiversion: MWVersion fallbacks & such (duration: 00m 56s)
* outage immediately reported, began rollback
** PHP fatal error: Call to undefined method stdClass::get()
* 19:36 demon@tin: Synchronized multiversion: rollback (duration: 00m 56s)
== Conclusions ==
The canary checks for MediaWiki remain insufficient to catch production errors prior to code rolling out live. mwdebug* is nice for testing specific config changes, but does not get "real" traffic so it's hard to test things extensively. The multiversion code is incredibly fragile--but we knew this. This refactor is complicated, should be broken down even further (than it already is)...small changes are best with this endeavor.
== Actionables ==
* Include fatal log rate check in scap canary test - 
* All entry points (including cli) should be subject to canary checks - 
* T152005 did not cause/exacerbate the outage, but was noticed at the time, priority raised
Category:Incident documentation
Incidents/2017-01-18 Labs

== Summary ==
~270 user home directories across labs instances that didn't have NFS mounted (non-tools/maps instances) got erased as an unintended side effect of the planned NFS migration() for moving rest of labs projects on NFS to the secondary NFS cluster. This happened due to merging an erroneous puppet patch that made /home a symlink to the mount path across all instances, instead of just on nodes where /home was on NFS. We were able to restore most of the data using puppet filebucket restore mechanisms, but a few home directories, and some executable file permissions have been permanently lost.
== Timeline ==
* [17:09] Madhu starts labs (misc share) NFS migration to the new secondary NFS cluster
* [17:13] Madhu disables puppet across Labs instances with NFS mounted
* [17:58] Madhu merges patch (https://gerrit.wikimedia.org/r/#/c/332735/) that sets up symlinks for /home and /data/project to mounts from labstore-secondary
* [17:59] Tests on a few NFS mounted nodes and then starts rolling it out
* [18:30] Roll outs and puppet runs complete - spot checks nodes - realizes that some instances where /home isn't mounted from NFS have broken symlinks
* [18:40] Abogott and mutante report puppet failures across tools and labs because /home is a broken symlink
* [18:46] Madhu rolls out fix https://gerrit.wikimedia.org/r/#/c/332803/, but puppet has already run in a lot of nodes and wiped out all the home directories
* [19:00] Chase and Yuvi work on writing a script that uses puppet filebucket to restore the missing /home directories - after a lot of tweaking and setting up user permissions, seems to work fine. Madhu rolls out script across broken nodes and most directories are recovered.
== Conclusions ==
* List of affected nodes in this process -
* List of nodes where /home is missing - 
* Most of the data was recovered, but we lost executable permissions set on files, and there has been one report of partially missing data.
* Having nfsclient.pp applied to every single labs instance, and then optionally mounting/unmounting shares based on config is scary - it leads to cases like this where instances that don't have NFS and shouldn't be affected or watched out for do end up being affected
== Actionables ==
* Do not include nfsclient.pp globally across all instances (https://gerrit.wikimedia.org/r/#/c/333227/)
Category:Incident documentation
Incidents/2017-01-24 WikibaseClient-InterwikiSorting

== Summary ==
Due to an automatic switch in WikibaseClient adding configuration for the InterwikiSorting extension caused Wikibase to start using it in production.
This wouldn't have been an issue, except WikibaseClient only checks for one setting but uses 4.
One of these settings no longer exists in the extension and was thus not included in the config change.
Fatals on all wikis using WikibaseClient for about 3 minutes where the parser cache was missed and getParserOutput was thus called.
Although mwdebug1002 was checked, the issue was not spotted, probably due to the issue only arising on getParserOutput calls, cached output would not expose the error.
== Timeline ==
* 12:12 Copy InterwikiSorting settings from wmgWikibaseClientSettings merged by jenkins
* 12:13 addshore@mwdebug1002 scap pull
* 12:14 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155995 gerrit:333882Copy InterwikiSorting settings from wmgWikibaseClientSettings noop (duration: 00m 39s)
* 12:17 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert last (duration: 00m 39s)
* 12:19 Reported as https://phabricator.wikimedia.org/T156123
* 12:26 Revert "Copy InterwikiSorting settings from wmgWikibaseClientSettings" added to gerrit
== Actionables ==
* Do not use wgInterwikiSortingAlwaysSort in WikibaseClient () 
* Check all config options used...
Incidents/2017-01-26 API Slowdown

:Very WIP- still under heavy research
On the 2017-01-26, from 17:51 to 18:15 (all times UTC) there was a slowdown/increase in 500 responses on Wikimedia wikis' Mediawiki Action API. While there was scheduled maintenance at the time, no user impact should have been seen, the underlying cause is still being researched.
== Summary ==
* A core router started rebooting/behaving strangely since 12 January 
* DB and other services impact was mitigated by moving essential services away from the affected rack (e.g. s1 master)
* Maintenance started on router at 17:51- mediawiki should have just depooled affected services (dbs), and continue unaffected, as usual, but it didn't work/didn't work as expected
* API latency/thoughput impact can be seen at:
** https://grafana.wikimedia.org/dashboard/db/api-summary?from=1485431717947&to=1485475144648
** https://grafana.wikimedia.org/dashboard/db/navigation-timing?var-metric=saveTiming&from=1485431717947&to=1485475144648
** https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?from=1485431717947&to=1485475144648
* Depooling affected API servers resolved the issue
== Timeline ==
* 17:46 paravoid: stopping pybal on lvs1001/lvs1002/lvs1003
* 17:51 paravoid: replacing asw-c2-eqiad
* 17:57 elukey: boostrapping aqs1007-a cassandra instance
* 18:14 paravoid: rebooting newly provisioned asw-c2-eqiad to enable mixed mode
* 18:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055, 56, 57, 59 (duration: 00m 54s)
* 18:32 paravoid: starting pybal on lvs1001/lvs1002/lvs1003
== Conclusions ==
More research is needed to understand why the issue happened and how mediawiki model works, and if it has a bug for this particular scenario.
== Actionables ==
* 
Category:Incident documentation
Incidents/2017-02-22 www-portals

== Summary ==
At about 17:00 UTC Feb. 22 the www.wikipedia.org page was severely broken for about an hour. 
The text on the page was invisible. This bug was caused by a JavaScript file being improperly cached and returning a 404. 
== Timeline ==
* A bug was filed at around 17:09 UTC Feb.22 noting that the text on www.wikipedia.org is invisible. 
* We were made aware of this bug at about 17:40 UTC 
* at 18:15 UTC an attempt was made to rollback to the previous deploy. The deploy was visible on mwdebug1002 without error, but the error persisted in production. 
* at 18:20 UTC we purged the URL of the specific JavaScript file, fixing the issue. 
== Conclusions ==
* The wikipedia.org portal depends on a specific order of syncing followed by purging urls, which is fragile and needs some rethinking. 
* Errors in JavaScript should not make the page unusable. 
== Actionables ==
* Adding an entire list of asset URLs to purge ()
* Preventing JavaScript from hiding page content indefinitetly ()
* Use query params for cache-busting ()
Category:Incident documentation
Incidents/2017-03-06 maps

== Summary ==
Map of Tasmania is covered in water at zoom level >= 10.
== Timeline ==
On Sunday March 5 Tasmania started to be rendered as mostly water .
The task was triage as UBN! on Monday Mar 6.
The problem was a combination of problematic data in OSM upstream (The whole Tasmania was marked as natural=bay) and the fact that our styling identifies water in a different way than most OSM styles. A detailed postmortem is available on .
Issue was resolved around 10pm Mar 6.
== Conclusions ==
Investigation was longer and more difficult that it should be due to missing knowledge and missing investigation tools.
== Actionables ==
* A detailed post mortem is available on , actionables are still being created as subtasks.
Category:Incident documentation