nyuuzyou/wikis huggingface dataset:
Incidents/20160721-LocalRenameUserJob

== Summary ==
MediaWiki 1.28.0-wmf.11 included changes to the jobs for CentralAuth's global renaming process which left local accounts of renamed user unattached on some wikis while the Wikimedia production cluster was running mixed versions of MediaWiki. At 2016-07-20T20:06Z the group1 wikis were updated to 1.28.0-wmf.11. Group1 includes meta which is the wiki where stewards and global renamers approve rename requests. The 1.28.0-wmf.11 version of this process starts by detaching the local wiki accounts for the user being renamed as part of the mitigation for . The accounts are reattached at the end of each associated LocalRenameUserJob. Between 2016-07-20T20:06Z and 2016-07-21T19:09Z (when all wikis changed to 1.28.0-wmf.11), the local jobs run on group2 wikis (functionally the Wikipedia family) did not include the new code needed to reattach the local accounts to the renamed central account.
== Timeline ==
* 2016-07-15T07:49Z merged to master
* 2016-07-15T19:21Z bd808 asks about consequences of new job introduced in related patch and mixed cluster state. This turns out to be ok, but the changes to the existing job in were not discussed.
* 2016-07-20T20:06Z group1 to 1.28.0-wmf.11
* 2016-07-20T20:06Z Global renames start leaving group2 wikis unattached after rename. 
* 2016-07-21T17:47Z opened by Steinsplitter to examine and correct six renamed users with unattached accounts.
* 2016-07-21T19:09Z group2 to 1.28.0-wmf.11
* 2016-07-21T19:09Z Global renames stop leaving unattached local users.
* 2016-07-21T19:23Z legoktm pings bd808 on irc to look at .
* 2016-07-21T19:35Z bd808 determines likely root cause of mismatched MW versions and starts trying to find all affected users in logs.
* 2016-07-21T19:44Z Priority of raised to UBN! by MarcoAurelio due to potentially severe end user impact.
* 2016-07-21T20:27Z bd808 finds 54 renames that may have been affected.
* 2016-07-21T23:19Z bd808 runs custom maintenance script to reattach any unattached local accounts for the 54 renames. The script finds and fixes 18 accounts.
* 2016-07-21T23:23Z bd808 lowers priority of to High.
* 2016-07-22T20:53Z bd808 follows up by grepping all renames done in July from logs on fluorine and running them through the cleanup script. Of the 748 accounts checked, 3 more were found with unattached local accounts. Only one of the three was due to the same root cause and happened during the previously identified time period.
* 2016-07-22T22:34Z bd808 writes up incident and closes as resolved!
== Conclusions ==
* Letting changes to existing jobs ride the train is potentially dangerous if the job runs on the local wiki rather than the submitting wiki. More care should be taken when writing and merging such changes to ensure that they are rolled out to all wikis in a very short time window.
== Actionables ==
* Do we still track "scap traps" somewhere?
Category:Incident documentation
Incidents/20160807-CI

== Summary ==
CI had a roughly 4 hours outage which unfortunately was due to a known issue where Nodepool tries to create too many files (and thus exhausting inodes) on the Jenkins master.
== Timeline ==
* 11:01 < Amir1> Zuul seems to be extremely slow: https://integration.wikimedia.org/zuul/
* 11:03 < paladox> Hi nodepool seems to be down in zuul 
* 11:08 <+ Reedy> Aug 07 11:07:54 labnodepool1001 nodepoold[16727]: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403)
* 11:15 <+ Reedy> I'm not restarting nodepool on a whim
* 11:16 <+ Reedy> I'll text hashar
* 11:35 - Reedy texted Antoine
* 11:39 <+ hashar> Reedy: paladox around :)
* Diagnosis/attempts to fix by deleting unused nodepool instances that were stuck
* 12:30 - Antoine started deleting files in /var/lib/jenkins/config-history/config
** ssh gallium find /var/lib/jenkins/config-history/config/nodes \ -path '*_deleted_*' -delete
* 12:41 <+ hashar> Reedy: paladox ci back
== Conclusions ==
* We need to cleanup unused config files on a schedule
== Actionables ==
* Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - ()
Category:Incident documentation
Incidents/20160810-MediaWiki

TL;DR VipsScaler extension converted to extension registration, which meant scaling configuration was merged, not overridden by WMF config. This meant Vips was being used to scale jpg etc, when it should have just been doing large tiff and png.
Similar to Incident documentation/20160809-MediaWiki.
== Timeline ==
* Conversion of VipsScaler extension to extension registration was merged a few days before the .14 branch cut
* 19:30Z/19:34Z .14 was rolled out to majority of wikis (including Commons, where fault were clearly noticeable)
* Commons noticed/reported issue at 20:32Z
* Various IRC reports of thumbnailing issues in the few hours that followed and phab:T142638 was filed 21:09Z
* Some initial confusion as to where the problem lay, due to imagescaler changes/reinstalls the same day
* Comments on IRC suggested it started "a few hours ago" pointing at .14 changeover
* All wikis rolled back to .13 at 23:17Z, purging some thumbnails proved issue was resolved.
* Image scalers ruled out at this point
* Group discussion and looking at changelog identified a few Media related changes to core. Nothing seem directly relevant, but change of VipsScaler to extension registration seemed likely candidate
* eval.php on .14 wiki showed $wgVipsOptions had incorrect WMF config, as it had been merged with the the default config in the extension. This meant Vips was incorrectly being used to create thumbnails for jpeg files
* Gilles pointed out it was stupid for the extension to have a seemingly broken default config. So was all removed in https://gerrit.wikimedia.org/r/#/c/304137/ cherry picked and deployed, easiest possible fix
* Thumbnails tested on .14 after cherry picking the fix, most wikis had .14 reinstated at 23:46Z (reverting the earlier revert), and fix was confirmed on commons on various files
* Aaron tweaked and ran a cleanup script to re-thumbnail images affected during the time window. Which can't work for images that have been manually purged during this timeframe, which we can't easily account for. Users will have to manually purge when they encounter these images.
== Casualties ==
* Evening SWAT didn't happen.
* From 19:30Z to 23:17Z jpeg images were wrongly thumbnailed using Vips resulting in horizontal grey lines on them.
== Conclusions ==
* Default VipsScaler configuration is broken, resulting in grey lines across jpeg. As the fix, because they were wrong, the default configuration was removed from the extension in https://gerrit.wikimedia.org/r/#/c/304140/ and cherry picked to .14
* Method of unconditionally overwriting extension config, rather than merging config arrays needed for extension registration. Filed as phab:T142663
* Minor improvements needed to purgeChangedFiles.php maintenance script to purge old versions too, done in https://gerrit.wikimedia.org/r/#/c/304137/
* Method of identifying other "changed" thumbnails (not just uploads or new versions) to be purged in case of thumbnailing errors is needed phab:T142638#2542591. No tracking apparently done, at least in MediaWiki. Can this information be obtained from swift?
Category:Incident documentation
Incidents/20160818-Elasticsearch

== Summary ==
During planned network maintenance, elasticsearch search cluster in eqiad lost connectivity between nodes, resulting in 17 nodes parting the cluster, and elasticsearch stopping to respond to requests for around 10 minutes (see graph).
== Timeline ==
* 12:09UTC: icinga alert on elasticsearch state
* 12:21UTC: all primary shards allocated, secondary shards recovering (see graph)
* 12:40UTC: all traffic directed to codfw search cluster
* 17:18UTC: less than 300 secondary shard not allocated
* 17:18UTC: traffic switched back to eqiad cluster (except more-like, which are by design sent to codfw)
* 19:02UTC: all secondary shards allocated (see graph)
== Conclusions ==
* In its current configuration, elasticsearch is too sensitive to loss of networking. This is the second time we have a similar issue. Previous analysis by discovery team was in reluctant to change default parameters in elasticsearch fault discovery, which runs the risk of not detecting faults early enough. This analysis needs to be reviewed.
* As long as we don't make elasticsearch more robust to loss of connectivity, we should preemptively switch traffic away from a datacenter during maintenance.
* Switching traffic took too much time. Part of it is Guillaume not being ready for this procedure, part is the procedure being too complex (a change in wmf-config).
* Elasticsearch icinga alerts are too noisy. Checks on global cluster state are done for each node instead of doing it once for the cluster.
== Actionables ==
* Increase timeout and retry in elasticsearch fault detection configuration ()
* Switching search traffic between datacenters should be faster ()
* Improve elasticsearch alerting ()
Category:Incident documentation
Incidents/20160823-ToolsProxy

== Summary ==
Tool Labs proxy (and thus all webservice accessibility from the internet) were down for approximately 2 minutes, between UTC 0546 and 0548, Tue Aug 23. 
== Timeline ==
* 0546: Yuvi gets a page for PAWS being down
* 0546: Yuvi investigates, notices tools is down too
* 0547: SSHs into tools-proxy-01, looks at error.log. Notice lots of 768 worker_connections are not enough errors
* 0548: Restarts nginx, fixing the issue for now.
== Conclusions ==
* Our current worker_connections limit is too low.
* There was no widespread paging for this. PAWS alert is set to alert only Yuvi, and also caught this only incidentally.
* We had a higher worker_connections limit, but that was killed in favor of the default number in https://gerrit.wikimedia.org/r/#/c/297829/. 
* There's no quick way to failover tools-proxy, making intense debugging a priority over failover & calmly investigating.
== Actionables ==
* Increase the worker_connections limit, tune nginx properly 
* Setup paging with a super simple webservice, to replace the killed tools home page check 
* Make a script that facilitates failover of tools / nova proxy 
</onlyinclude>
Incidents/20160829-ToolLabs-Outage

== Summary ==
On labstore1001, a snapshot device of the scratch device, made for testing - was mounted accidentally on to the directory that was the acting as the scratch NFS share. This caused NFS server to back off and stop serving traffic, and all the file handles held by NFS were rendered stale. It affected all of tools - puppet runs failed across hosts, the proxy didn't serve traffic, and instances with NFS mounted couldn't be ssh-ed into. It was fixed by stopping NFS server, fixing the mounts, and starting it again. The outage lasted ~30 minutes.
== Timeline ==
* 22:49 - 23:04 - Puppet runs on all tools hosts fail, tool labs home page unreachable
* 23:05 - Chase and Yuvi look, Chase attempts to restart nginx because tools-proxy is serving 499s everywhere, and Yuvi bumps up worker connections and restarts again - which doesn't help.
* 23.15 - bd808, Chase and Yuvi spot check a bunch of nodes by ssh-ing into them, find that they cannot ssh into nodes with nfs, and can ssh into nodes without - Chase suspects the issue lies with storage and not networking.
* 23.21 - Chase pokes Madhuvishy to see if she was doing anything with nfs - she says she isn't.
* 23.28 - Chase notices super high load on spot checked exec nodes and tools bastion, and outbound for NFS on labstore1001 was doing nothing, so restarts nfs-kernel-server on labstore1001 and 1003(this was not necessary, he did it for fun anyway).
* 23.29 - Puppet runs succeed, things start to recover
* 23.33 - Madhuvishy realizes she had actually messed up the scratch nfs share on labstore1001, by mounting a snapshot device created for testing on the directory that serves scratch(/srv/scratch). She reports it, and Chase figures out that when the shared device was mounted over, nfs backed out and stopped serving traffic, after nfs server restart that was fixed but all the old file handles to scratch had been rendered stale. It isn't possible to unmount the incorrectly mounted backup device since it's being used, so he stops nfs server, switches the mount over, also fixes a binding mount at /exp/scratch. and start the server again. 
* 23.34 - Things are fixed at this point, Madhuvishy checks all the tools hosts and makes sure there aren't any stale scratch mounts. 
== Conclusions ==
* It is still not clear why the Proxy served 499 - the proxies don't have NFS enabled. Could it be because they were trying to hit the enough hung tools hosts for data and failing - if so why not 5xxs and why 499?
* Labstore1001 is a huge single point of failure for all of tools. We couldn't failover to another box and debug this one without breaking everything, nor could we be testing in some place that isn't part of the live system, and such mistakes wouldn't cause all of tools to fail. (This is being worked on)
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* Madhuvishy wears kid gloves on labstore1001 for eternity
* The labstore overhaul - HA setup cannot happen fast enough ()
Incidents/20160831-ulsfo

== Summary ==
Due to the concurrent failure of 3 different network links across two different carriers, our connectivity between our San Francisco ulsfo data center and the rest of our network was briefly severed for a few times, resulting in downtime for the US west coast and parts of Asia until traffic was rerouted.
== Timeline ==
* 05:38 - Link between ulsfo and codfw went down and started flapping
* 08:17 - Links eqord and ulsfo, and between eqord and codfw went down for planned maintenance. Due to 3 links being down or flapping, connectivity between ulsfo and the rest of our network became intermittent. Traffic to public services from US west coast states and parts of asia was be impacted.
* 08:24 - Traffic was rerouted from ulsfo, removing impact from users
* 08:45 - A matching maintenance announcement for the ulsfo-codfw link was discovered to be sent to the wrong address
* 13:36 - Several hours after the completion of both maintenance windows and the links were confirmed stable, traffic was routed back to ulsfo
== Conclusions ==
All 3 links, which were from two different vendors, had overlapping maintenance windows. The maintenance announcement from one of the two vendors was sent to the wrong e-mail address and therefore not processed by Clinic Duty and put in our vendor maintenance calendar. 
== Actionables ==
* Request vendor to correct e-mail contact for future maintenance announcements
* Establish a more rigid process for evaluating maintenance announcements with particular attention for potential overlap for redundant links
Incidents/20160918-Wikitech2fa

== Timeline ==
* 2016-09-17 While resetting Greg's 2FA on wikitech, reedy notices that the oathauth_users is due for a schema update. reedy also files a security issue in phab:T145915. He runs OATHAuthHooks::schemaUpdateOldUsers() via eval.php and then drops columns secret_reset, scratch_tokens_reset and is_validated from the wikitech database.
* 2016-09-20 19:30 valhallasw reports his 2fa credentials have been disabled -- phab:T145973
* 2016-09-21 00:00 Reedy diagnoses the problem. Andrew recovers a backup of the labswiki database from earlier on the 16th. Reedy restored the old oathauth_user table into a temporary table, and then selected records present in the older table but missing in the current version were copied across.
** Candidates identified via SELECT id,user_name from oathauth_users_restore INNER JOIN user ON id = user_id where id NOT IN (select id from oathauth_users)
== Casualties ==
* Any new (or users that had disabled and re-enabled) 2FA users since was merged; the is_validated column was not populated (taking the default of 0), meaning they were eligible to delete in OATHAuthHooks::schemaUpdateOldUsers()
== Conclusions ==
* Migrations, once started, should be seen through, across the cluster. If the DB columns had been dropped (as it would have been done if run via update.php). It is unknown if wikitech's oathauth_users table on previously migrated. If the columns had been running OATHAuthHooks::schemaUpdateOldUsers() would've been a no-op, and no data loss would've occurred. However, running it so far after the code going live... Would just generally cause the same problem. It seems the columns were already dropped on centralauth.oathauth_users.
Category:Incident documentation
Incidents/20160924-ORES

== Summary ==
ORES review tool (= ORES extension) couldn't score edits made in 14 hours between rolling out of wmf.20 and the fast fix made in 2016-09-23
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
* (2016-09-22) SAL: 20:00 thcipriani: rolling out wmf.20 to all wikis
* (2016-09-23) 9:44 The phab task is created
* 9:45 The gerrit patch is made to fix it in master
* 9:47 The patch is merged.
* 9:48 The backport to wmf.20 is made.
* 9:51 The backport is merged
* SAL: 09:58 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s)
* SAL: 10:00 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki
* SAL: 10:05 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki'
== Conclusions ==
* There should be an alarm to scream when jobs such as ORESFetchScoreJob is not triggered for more than an hour.
* The lapse was easy to notice, ORES extension should have extensive CI tests.
== Actionables ==
* Extensive CI tests for ORES extension ()
* High failure rate of account creation should trigger an alarm / page people ()
Category:Incident documentation
Incidents/20160925-ores

== Summary ==
At September 25th, ORES service had higher ~14%) timeout ratio for six hours. Because it ran out space due to too verbose logging.
== Timeline ==
* Sept 25 10:34:40 UTC 2016: icinga test on ORES failed due to timeout.
* 14:13 UTC: phab:T146581 is created.
* 16:03 The fix deployed in labs.
* 16:26 The fix deployed in prod.
== Conclusions ==
We should have better monitoring disk space and be careful on verbosity of production services logs
== Actionables ==
* Less verbose ORES 
* Grafana monitor on disk space in ORES 
Category:Incident documentation
Incidents/20161021-Maps

== Summary ==
Between 18:50 UTC and 19:20 UTC, October 21st, maps.wikimedia.org stopped rendering tiles due to Cassandra backend being unavailable.
== Timeline ==
* 18:50 UTC: cassandra wrongly reinitialized on maps2004.codfw.wmnet, deleting all cassandra data on maps2004. Kartotherian starts failing with org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE.
* 19:20 UTC: traffic redirected to maps eqiad cluster, user traffic is served again without error
* 19:40 UTC: full deployment of new traffic configuration
* 21:13 UTC: permissions are reset on maps/cassandra codfw cluster, kartotherian starts working again on the codfw clsuter
== Conclusions ==
* The main trigger for this is human error.
* maps/cassandra has a replication factor of 1 on the "system_auth" keyspace. This means that loosing one node potentially breaks authentication.
== Actionables ==
* increase replication factor on system_auth keyspace 
Category:Incident documentation
Incidents/20161127-API

== Summary ==
[DRAFT]
On 2016-11-27 at around 2 UTC HHVM on the API cluster suffered from excessive memory consumption, eventually leading to an API outage. One leading factor to the outage was an expensive template being asked repeatedly as part of batch re-renderings. This is likely a reoccurrence of a similar event on 2016-10-17 (at https://phabricator.wikimedia.org/T148652) with similar underlying reasons. The investigation was carried out in https://phabricator.wikimedia.org/T151702.
== Timeline ==
* ~2.00 memory starts increasing
* 2.27 First page for api.svc.eqiad.wmnet dispatched, Filippo starts investigating
* ~2.40 First round of rolling restarts to help lessen the issue.
* ~3.30 5xx are down, after three/four peaks, investigation continues. It is noted that there are many failing requests from euwiki
* 5.00 symptoms are back, ori deploys a bandaid to mw1290 to ban (500) requests for euwiki coming from parsoid
* ~5.30 5xx are back
* 5.50 bandaid is confirmed working
* 6.20 bandaid is deployed to APIs
== Conclusions ==
Effective protection against excessive resource usage is still an open issue, in particular for the API cluster. Further, there's no isolation per-usage and that leads to bigger blast radiuses than needed.
== Actionables ==
Category:Incident documentation
Incidents/20161202-20161201-PageImages

== Summary ==
A change went live on the train that was supposed to update the page images API to provide an additional api parameter that would allow you to select a page image depending on license type. The current behaviour was that the page images API would return images that were free. The new parameter would allow API callers to also explicitly include images that were non-free. Due to an issue in implementation, the new change reset all known free page images and as a result for most API requests the page images API would return no image. This caused images that previously were appearing in many of Wikimedias projects including our Wikipedia apps, the Wikimedia portal and mobile web search results to disappear for about 3 hours.
== Timeline ==
* @Deskana commented that images had disappeared from the portal. Asked @ebernhardson and @MaxSem to drop everything and fix it.
* @MaxSem tracked down the offending change
* @ebernhardson pinged reading web team (Baha and Jon R) to make them aware of his proposed fix
* @jdlrobson, @ebernhardson and @MaxSem chatted and agreed patch to enable `any` image as default was best short term solution and patch was merged
* @ebernhardson SWATed the fix
* Discovery team enabled a script to re-populate all images on the script. It is estimated that it will take 3 weeks to run. In the meantime API subscribers will be returned any image by default in all Wikimedia projects (which may include non-free images). If they request free images they will only get free images, but there will be many missing images until the maintenance script has completed. Requests for an image with license type `any` may also return an inferior result compared to what images are available.
== Conclusions ==
* Had this change been enabled explicitly by a config changes it would have minimised the time frame from this change going live and being detected and minimised disruption. We should consider flagging big changes like this in future.
* Changes to the PageImages extension should probably be run via the discovery team in future and have been subjected to more thorough review.
* These kinds of big changes should have been thoroughly tested on beta cluster before being enabled in production servers.
== Actionables ==
* Update documentation of extension to make developers wary of this problem
* Improve maintenance script so that recovery from such problems can be quicker.
</onlyinclude>
Category:Incident documentation