nyuuzyou/wikis huggingface dataset:
Incidents/20150820-PageCategorization (Privacy)

== Summary ==
Between Aug 18th and Aug 20th, a patch was deployed to some wikis (group0 and group1 deployment groups) on the WMF cluster that caused MediaWiki to report IP addresses of editors who cause a category to be added to a page under certain conditions. When the category was added as part of a parser function triggered from a subsequent call to the purge API, the recent changes log reported the IP address from which the purge originated, even when called by a logged in editor. The proximity in the recent changes log of these changes to other edits could have allowed an observer to associate the IP address and username of an editor. The publication of this combination of data is considered a violation of our privacy policy.
Once the issue was confirmed and problem patch identified, the Security team immediately began the process of reverting the feature, and deleting all recent changes entries created by the patch to prevent leaking this private data further.
During the time that the patch was deployed, the API call that could have triggered this leak of private information was called 340 times (326 times on commons.wikimedia.org, 10 times on pl.wiktionary.org, and 4 times on en.wiktionary.org) by 69 different IP addresses.
== Timeline ==
* 2015-08-18 18:30 UTC: 1.26wmf19 (which contained Gerrit ) was deployed to group0 wikis (testwiki, test2wiki, mediawikiwiki, zerowiki, testwikidatawiki)
* 2015-08-19 18:06 UTC: 1.26wmf19 deployed to group1 wikis (non-Wikipedia wikis)
* 2015-08-19 21:49 UTC: User Peter Bowman reports 
* Approximately 2015-08-20 18:00 UTC: Release Engineering deploys revert (Gerrit )
* 2015-08-20 20:25 UTC: Script to delete recentchanges rows finishes
== Conclusions ==
The patch that caused the issue was created, reviewed, and merged by experienced staff and volunteer developers, yet each of these developers failed to identify the potential security impact of the design. This indicates that the security ramifications of the functions involved are likely not structured or documented in a way that makes the issues clear, and this particular code pattern has not been highlighted as dangerous in documentation or training. Additionally, there are currently no automated or manual testing procedures specifically to identify privacy issues in new version of MediaWiki.
== Actionables ==
In the near term, we will add notice about User::newFromId(0) to secure code training (), and investigate updating the function definition or documentation to make the security impact more clear to developers.
Longer term, we will investigate using static analysis tools to automatically flag this code pattern (depends on , and look at developing security training specifically for QA testers, which would include looking for privacy issues.
Category:Incident documentation
Incidents/20150830-labs-nfs

== Summary ==
CPU soft lockup on labstore1002 caused load to skyrocket and NFS mounts to fail. A reboot had two shelves not being assembed and sdaw device not readable, resulting in missing PVs (causing the LVs to be inactive). Another powercycle (via shutdown -h now and then a mgmt server up) did not help with the arrays, but the sdaw errors disappeared. A manual reassambly of the two shelves worked, and service was restored, for a total of 4h of downtime.
== Timeline ==
(All times in UTC +2, 30 Aug 2015)
* 11:30: Shinken alerts about toollabs being down. A quick IRC check shows multichill reporting the same.
* 11:33: Yuvi ssh's to labstore1002, confirms high load. Sees 'NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/5:5:17071]'. <link-to-logs>. 
* 11:40: Yuvi starts paging people.
* 11:50: godog responds and starts investigating
* 11:53: Yuvi saves kernel logs, reboots labstore1002.
* 12:04: labstore1002 comes back up, but with errors about activating all lvm volumes. 
* 12:06: start-nfs fails, lvs reports device sdaw is unreadable.
* 12:26: After digging around a bit, godog attempts to activate tools volume. Fails citing missing PVs. godog continues investigation.
* 12:28: Yuvi verifies that backups to labstore2001 were working properly, and that latest backup was about 6h old
* 12:50: Valhallasw sets up error page on tools.wmflabs.org to announce NFS failure
* 13:17: Assuming it's the RAID controller being wonky again, labstore1002 is 'hard' rebooted, from mgmt. No more sdaw errors, but PVs are still missing - two arrays aren't assembled, can't mount LVs.
* 13:56: Yuvi pages Chirs, to attempt to get to the DC and do an inspection of the controller / maybe switchover to labstore1001.
* 14:04: Chris responds. Also tells us the last time multiple reboots fixed the controller, so Yuvi proposes to try that.
* 14:16: godog attempts to assemble the two missing arrays manually, which will restore service. 
* 14:32: godog assembles the two missing arrays manually
* 14:48: manuall assembly seems to have worked - LVs are activateable now
* 14:51: After verifying that the LVs are mountable, godog runs start-nfs. Mounts still not back in labs instances, however.
* 15:02: Yuvi realizes we need to run the 'sync-exports' script as well
* 15:14: sync-exports by itself doesn't seem to have worked, needed a restart of NFS as well
* 15:15: NFS is back, and instances start recovering!
== Action Items ==
# Switch NFS Server to labstore1001 
# Fix / Test / Replace labstore1002 
# Better documenation for our NFS setup 
# Fold sync-exports into nfs-exports daemon 
# Verify and add more checks for Labs NFS paging
Incidents/20150901-Elasticsearch

== Summary ==
Elasticsearch service (on elastic*.eqiad.wmnet nodes) backing the search functionality went red for few minutes. We didn't lose any real data and we failed to service some searches during 10 minutes.
== Timeline ==
* 05:28: dcausse pauses write before applying the firewall rules to master (elastic1001)
* 05:32: chasemp applies the rules
* 05:32: master is starting to lose track of its nodes
* 05:33: cluster is red
* 05:33: chasemp revert the rules
* 05:34: cluster is starting to recover
* 05:39: cluster is back to yellow
* 05:48: there's a 10 min spike of "Pool errors", dcausse and chasemp test some queries on enwiki and they all worked
* 07:58: cluster is back to green
* 08:00: dcausse unfreeze the indices
== Conclusions ==
* https://phabricator.wikimedia.org/T104962#1594537
== Actionables ==
* https://phabricator.wikimedia.org/T104962#1594537
Category:Incident documentation
Incidents/20150908-ores

Between 1350 UTC and 1550 UTC, ORES was down during an upgrade. 
== Summary ==
During an upgrade, version issues cause the ORES service to go down for approximately two hours. The problem was the result of the wrong version of ORES being pushed to the production servers. This was due to an outdated 'master' branch being pushed to the 'deploy' branch before the 'fab deploy_web' and 'fab deploy_celery' scripts were called. This resulted in the service being unable to even start. 
Another issue worth note is the extreme amount of time spent waiting for pip to update packages (that did not need updating in the first place). While revscoring and ores specify a range of versions requirements that was satisfied by packages that were already installed, pip insisted on installing new versions repeatedly. 
== Timeline ==
* 2015-09-08 @ 13:50 UTC -- 'fab deploy_worker' command is run
** ... moments later ores.wmflabs.org stopped responding
* 2015-09-08 @ 15:50 UTC -- ORES is able to serve requests again
== Conclusions ==
* pip is great for development, but it's a problem in production. We should never be surprised with a 5 minute compile time when we're deploying code
* It's critical to make sure that the right version is on the 'deploy' branch before starting the deployment.
== Actionables ==
* phab:T111806 -- ORES load balancer doesn't rebalance on 500 reponses
* phab:T111826 -- [Spike] Under what circumstances does pip want to re-compile scipy/numpy?
* phab:T111828 -- ORES/Wikilabels deploy checklist
Category:Incident documentation
Incidents/20150914-extension1

== Summary ==
On 2015-09-14 from about 09:47 UTC until 09:59 UTC, the x1 master was saturated with slow queries caused by a missing index. This caused all Flow boards to be inaccessible: trying to view a Flow board displayed a database error message.
== Timeline ==
* 09:43 Jaime runs the last ALTER statements for a complex schema change for Flow
* 09:47 Matthias notices that every Flow page view dies with (Cannot access the database: Too many connections (10.64.16.18))
* 09:51 Jaime figures out this is because all queries are now unindexed: the ALTERs dropped the indexes that were being used
* 09:56 Roan retrieves the old index definitions from git history
* 09:59 Jaime recreates the old indexes; Flow comes back up immediately
== Conclusions ==
The commit message of the change contained detailed instructions about how to carry out the migration. It instructed that patch3.sql (setting the field to NOT NULL and dropping the old indexes) be run before turning off $wgFlowMigrateReferenceWiki. However, this was incorrect: as long as $wgFlowMigrateReferenceWiki was turned on, the old indexes would be used, so the correct order was to create the new indexes, turn off the setting, then remove the old indexes. This mistake was missed in code review and missed again by the Flow devs who were present when the schema change was executed. The other statements in phase3.sql (setting the fields to NOT NULL, and DROP INDEX statements for temporary indexes that were added later) were appropriate to run at that stage, further hiding the incorrectly placed DROP INDEX statements.
== Actionables ==
* Do we have (or should we develop) something that tells us which indexes are used and which aren't? That would have allowed us to verify each index was unused before dropping it.
*: Create a debug mode for MW that blows in your face when an unindexed query is run and enable it by default in Vagrant and betalabs? maxsem (talk) 20:06, 18 September 2015 (UTC)
* ...?
* Better monitoring for connections reaching max_connections and query performance: 
Category:Incident documentation
Incidents/20150930-enwiki-readonly

== Summary ==
Yesterday (Sept 29th) Chad H was poking at a fairly old bug that asks for cleanup of superfluous rows in user_properties, specifically the subtask of removing old skin entries that just take up space and end up falling back to Vector (our default) anyway.
After completing the work for all small and medium wikis (per the dblists), Chad began running some sample queries against enwiki to see how the cleanup would scale. Obviously, it's a much larger table. Chad then attempted to delete 16 rows from the table--what appeared to be low hanging fruit to test the queries on a large table--but it took about 2m9s to execute on the master.
The query plan for this delete did not involve the PK, so it ended up selecting about 17m rows to find which ones to delete. I immediately kept an eye on the replag, which did climb to about 3 minutes depending on the slave. Replag began to recede as the query finished but it did happen long enough for icinga to start noticing and enwiki to (very briefly) go into read-only mode.
As a result: Chad stopped all further work on his cleanup and made sure people knew it was him that caused the issues with S1. 
Around this point is also when people reported that the small and medium wikis lost their monobook skin preferences. The cause looks to be a bug in the cleanupSkinPrefs code.
After discussion between Chad, Jaime, and Greg it was decided to not restore from the DB. It was deemed to be more disruptive than leaving things as they were at that point. The state of those db rows had changed since the incident, some portion of users had already changed some or all of their personal settings back). 
: Extensive discussion in the following 2 weeks: phab:T114208
: Users who want their skin preference set across all wikis are asked to look at phab:T119206
== Conclusions ==
* Cleanup of databases is not very useful and mostly disruptive. At a minimum, it should only be conducted under the supervision of the DBA.
* Cleanup/migration processes need a clear rollback plan in case they go wrong. Doing so would've prevented this.
* Batch scripts should always always use PKs. (see phab:T17441, others)
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* - Remove cleanupSkinPreferences. It is a dangerous script with no benefit. - Gerrit 242374
* - Add changeSkinPref maitenance script to change a user's skin preference across all wikis. - Gerrit 254407
* - Restore individual user's previous skin preferences, as requested (until December 21st, 2015). - phab:T119206
Category:Incident documentation
Incidents/20151005-MediaWiki

== Summary ==
Typo in MediaWiki config change took all of the wikis down for around a minute.
== Timeline ==
Times are approximate and based on Alex's IRC logs.
* 23:37:35 Alex merges , to change mw:Extension:WikidataPageBanner config to include all namespaces on ruwikivoyage
** The change is not ideal, but looks harmless enough and passes automated checks (e.g. syntax)...
* 23:37:52 sync-file starts
* 23:38:09 sync-file finishes
* 23:38:34 First report of fatals in #wikimedia-operations from MaxSem, users begin to notice in other channels
* 23:38:57 Alex resets on tin, starts sync-file again
* 23:39:14 sync-file finishes
== Conclusions ==
* There are exactly two 'r's in array.
* Adding an extra r (arrray()) will form a valid PHP function name so passes a PHP syntax check.
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* mediawiki-config needs better test coverage - 
Category:Incident documentation
Incidents/20151014-MediaWiki

== Summary ==
A MediaWiki config change that should have been a no-op was synced and caused pages to appear with no content (and get cached) as well as Special:Random throwing exceptions.
== Timeline ==
* 17:56: Chad asks Legoktm if he as any objections to merging gerrit:232966 (written back in August). Legoktm quickly looks over it and says no objections
* 17:59: !log demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
* 18:01: First report in #wikimedia-operations that something is wrong. "A database query error has occurred. This may indicate a bug in the software." and "I can't see any page"
* 18:03: Legoktm checks fatalmonitor and sees a bunch of 303 Compilation failed: two named subpatterns have the same name at offset 263 in /srv/mediawiki/php-1.27.0-wmf.2/includes/MagicWord.php on line 960
* 18:03: Luke081515 reports the problem in a Phabricator task (phab:T115505)
* 18:04-05: Confusion over whether twentyafterfour had started the train deploy (he hadn't)
* 18:06: Legoktm reverts config change in gerrit, while Chad deploys it: !log demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
* 18:07: enwiki s1 slave lag triggers alert, paging ops (unrelated)
* 18:07: People are still reporting blank pages, purging them fixes it
* ... Discussion about what pages are cached, people trying to find a test case
* 18:11: Andre spots that https://en.wikipedia.org/wiki/GNU_General_Public_License is broken, Legoktm realizes that we need parser cache purges
* 18:14: Ori proposes using the RejectParserCacheValue hook to empty the cache and purge varnish
* 18:17: Legoktm determines that the outage was caused by double loading of the Disambiguator extension
* 19:11: Ori deploys hook to purge blank pages
== Conclusions ==
* It's probably safer to do these mass extension changes one at a time
* We should leave some room before train deploys so we can pin-point what deploy caused issues?
* MagicWord.php should have thrown exceptions instead of silently failing.
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
* MagicWord.php should check return value of preg_* calls instead of assuming they succeed ()
Category:Incident documentation
Incidents/20151015-EventLogging

== Summary ==
On Wednesday, 14 October, 2015 at 06:48 UTC, the Event Logging mysql consumer stopped inserting events. We would not have noticed except other alarms about raw vs. valid count led us to look at the dashboard, where we saw the eventlogging.overall.insertAttempted.rate dropped to zero.
== Investigation and Recovery ==
Andrew found the mysql consumer died with this stack trace: https://gist.github.com/ottomata/39aa0f4b3dda1a60ed49. The mysql consumer was restarted at 15:03 UTC and started reading from kafka, recovering. However, it had a lot of back-pressure built up and used up a lot of resources while it was trying to catch up. It went back to normal around 20:20 UTC and has been normal since then. For about one hour after the mysql consumer restart, the difference between raw and valid events spiked up to over 4800, and then went back to normal. I'm not sure why that was affected.
== Data Loss Report ==
I looked into three schemas that are usually pretty consistently getting events every few seconds: Edit, NavigationTiming, and ServerSideAccountCreation. All three of these schemas look like they're missing data, but not completely. There's a much lower than normal amount of data from 06:00 UTC to about 17:00 UTC. This means that when the consumer started back up it tried to recover but could not. We will try to save the data from Kafka and backfill manually. For future reference, this is how we grabbed the events that we needed to backfill:
ionice kafkacat -o 48850000 -c 1150000 -t eventlogging-valid-mixed -b kafka1012:9092 > recover.el.log
# NOTE: This, very weirdly, only got about 2 hours of data, even though the offsets used separately should span from 05:50 to 22:00. Still looking into it.
== Code Changes ==
The root cause is a timeout that killed the mysql consumer. That should be looked into more, but in the meantime we can make the mysql consumer take things slower when it tries to catch up. We think it tried to consume everything and ran out of memory, dying several times as it was catching up. We're tracking that and the fix here: https://phabricator.wikimedia.org/T115667
== Actionables ==
* Look for holes in a few different schemas in the database and update this report.
* Consume kafka topic to a file and backfill manually (we ended up just using the all-events backup files).
* Is there a way to make pykafka's timeout more robust? Or to automatically recover from this type of problem?
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20151023-graphite-grafana

== Summary ==
Graphite was unable to serve data properly due to large queries in dashboards
== Timeline ==
* 16:16 - 502 bad gateway from graphite
* 16:18 - investigation begins
* 16:31 - recovery, root cause still unknown, large query suspected
* 16:47 - stop graphite-index cronjob, suspected as a factor and later excluded
* 17:57 - the offending queries are found and the related grafana dashboard deleted
* 18:54 - offending client banned from apache on graphite1001
== Conclusions ==
Graphite doesn't include query cancellation or timeout capabilities for local queries it seems, so queries involving a lot of time series can occupy all uwsgi workers, resulting in "bad gateway" from apache.
In addition, grafana clients don't seem to reload dashboards when the dashboard definition is itself updated. This results in clients keep requesting the same (problematic, in this case) dashboard and thus needing a bad server-side.
== Actionables ==
* limit the impact of heavy/large graphite queries ()
Category:Incident documentation
Incidents/20151024-LabsNFS-Lag

== Summary ==
NFS lagged for all labs instances when I (milimetric (talk)) executed a
tail -n +2
on a huge 35GB file in /data/project/milimetric/. This used up all the bandwidth.
== Timeline ==
* 2015-10-24 00:52:44UTC Started the operation
* 2015-10-24 01:12:42UTC Coren alerted me on IRC
* 2015-10-24 01:33:16UTC I killed the operation
== Conclusions ==
* I could have asked someone from ops to do the file operation locally on labstore1002
* I could have rate limited with pv -L
* I probably should have just fixed the source file on stat1002 (where it came from originally) and re-copied it to labs
Category:Incident documentation
Incidents/20151026-MediaWiki

== Summary ==
During the SF morning SWAT deploy I sent out patches that moved the Search eventlogging schema from CirrusSearch repository to the WikimediaEvents repository. Upon deployment ResourceLoader started emitting an error about duplicate module registration. This occured because the patch for the CirrusSearch repository that removed the schema should have been deployed before the change that adds it to the WikimediaEvents repository. This was not noticed immediately because they are not included in `fatalmonitor` from fluorine which was open in my shell to monitor deployment issues. After a couple minutes this was noticed in https://logstash.wikimedia.org/#/dashboard/elasticsearch/fatalmonitor and the revert process started.
```
MWException from line 331 of /srv/mediawiki/php-1.27.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader duplicate registration error. Another module has already been registered as schema.Search
```
== Timeline ==
* 15:13 ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s)
* 15:13 varnish starts reporting errors to logstash
* 15:14 Reports in #wikimedia-operations of the site going down
* 15:18 Revert the three patches to WikimediaEvents in gerrit
* 15:21 ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s)
* 15:22 All error graphs return to normal
== Conclusions ==
The fatalmonitor on fluorine is the easiest monitor to keep on screen while working from a laptop with minimal screen space, but does not contain all information about fatals on the site. Perhaps this could integrate 5xx reporting from graphite or logstash. Rolling back changes in gerrit takes much too long when there are multiple patch sets and time is of the essence, this should be done on tin directly and fixed up in the deployment branches after the production issue has been fixed. Finally, ResourceLoader should not fatal the site due to configuration issues. The problem should be logged and the site should continue on serving requests the best it can.
== Actionables ==
* Include information about 5xx rate in fatalmonitor ()
* ResourceLoader should not fatal the site due to configuration issues ()
Category:Incident documentation