nyuuzyou/wikis huggingface dataset:
Incidents/20131030-Wikidata-ZeroRatedMobileAccess

== Background ==
ZeroRatedMobileAccess has always depended on MobileFrontend and used it liberally, including calls to its classes. However, it was done in hooks called by MF so Zero simply stopped working in absence of MF. This, however, changed in where Zero started using a ResourceLoader module from MF.
== What happened ==
At 23:02pm UTC, after deploying Zero extension updates, fatal monitor was flooded with:
 Fatal error: Class 'MFResourceLoaderModule' not found in
 /usr/local/apache/common-local/php-1.23wmf1/includes/resourceloader/ResourceLoader.php on line 408
The issue was tracked down to Wikidata having MobileFrontend disabled, while ZeroRatedMobileAccess was enabled. It didn't impact page views directly, however all load.php calls that requested the startup module caused fatals because it attempted to instantiate MFResourceLoader class and couldn't find it. As a consequence, people might have seen pages without styles or scripts.
A number of people (MaxSem, Reedy, Roan, and Greg, and possibly others) gave great assistance to track down the issue and rapidly disable the ZeroRatedMobileAccess extension in Wikidata. Furthermore, mobile configuration will add an additional guard against calling ZeroRatedMobileAccess.php unless it's explicitly within the context of MF.
Thank you to everyone!!!
== Timeline ==
All times in UTC
* 22:48 Zero 1.22wmf22 deployed, no errors
* 23:02 Zero 1.23wmf1 deployed, first errors appear - initially unnoticed
* 23:08 A small MobileFrontend change deployed
* 23:09 Errors noticed, initially linked with MobileFrontend push
* 23:17 Max reverts his MobileFrontend changes, errors don't go away
* 23:22 Problem narrowed down
* 23:27 Fix deployed
== Recommendations ==
* Allow a bit more time between deployments and observe fatalmonitor before and after
* Ensure Zero extension checks if Mobile extension is loaded before enabling itself if it relies on MFResourceLoader.
== Follow-up ==
* Generically, deployment calendar enforcement appears to be stricter.
* Guard rail code was put in place.
Category:Incident documentation
Incidents/20131104-Parsoid

== Summary ==
The Parsoid cluster corrupted the charset in wikitext converted from HTML for about 55 minutes after a broken Parsoid deployment.
== What happened ==
After several delays due to npmjs.org server issues and broken file permissions on tin, we deployed Parsoid version d7b556f25353 at 19:45 UTC. This version included a host of changes including the new busboy form data parser library. We switched to this library as the bodyParser library we used so far is deprecated in the connect framework and suffers from a potential DOS vector with temporary file creation on file upload requests.
At 20:04 UTC, there was a first IRC reports about VisualEditor edit corruption on the French Wikipedia. At 20:14, we became aware of this. The issue looked a lot like charset corruption. Due to some issues with git-deploy revert this took a while to revert. At 20:40 UTC, Parsoid was reverted to the pre-deploy version and the edit corruption stopped. At 20:47, both the code and the dependencies were fully reverted and Parsoid was back up.
The issue was caused by the busboy library defaulting to ISO-8859-1 encoding for incoming POST data rather than UTF8, which corrupted incoming HTML to Wikitext serialization requests from VisualEditor. This affected the full posted HTML if it included non-ASCII chars, so corruption was not limited to edited parts of the page.
== Timeline ==
All times in UTC
* 19:45 Parsoid d7b556f25353 deployed, no issues in logs
* 20:04 First error reports in visualeditor channel
* 20:14 Parsoid developers informed
* 20:30 First revert attempts fail
* 20:40 Parsoid cluster down after successful config repository revert
* 20:47 Parsoid cluster back up with old version after the code is reverted too
== Recommendations ==
* create HTTP-based Parsoid API tests in addition to parser tests and round-trip tests so that the full stack is tested. Tracked in bug 56590. 
* Manually test VisualEditor on non-English wikis before and after deployment. Added to the deployment instructions. 
* allow Parsoid deployers to quickly stop Parsoid machines to avoid further corruption in case of emergencies.
* consider sanity checks on wikitext diff vs. HTML diff size in VE to flag corruptions at any point along the way including the VisualEditor PHP extension
* maybe only list non-active commits in the git deploy revert dialog and default to the previous deploy to reduce the potential for mistakes
Category:Incident documentation
== See also ==
* Corresponding bug: https://bugzilla.wikimedia.org/56583
Incidents/20131205-Swift

== Summary ==
rsyslog was automatically upgraded via puppet ("ensure => latest") which triggered a known bug in Swift (media storage) causing it to hang thus cascading down to the imagescalers.
== What happened ==
written by Faidon
At about 18:55 UTC, the imagescaler (rendering) cluster started being unresponsive, ultimately throwing a LVS rendering CRITICAL alert. The error was cascading from Swift, on both pmtpa & eqiad and eventually a CRITICAL alert for ms-fe LVS as well was also issued.
Service recovered with no manual action(?) at ~19:40 UTC but still at an increased system load. I fixed this by restarting Swift front/back daemons at about 01:00 UTC.
The root cause was a combination of:
# Swift having a bug where it gets into a busy loop when losing the syslog socket: the sendto() operation to the syslog socket produces a ENOTCONN, it raises an exception, tries to log it, gets an ENOTCONN and so on and so on (strace shows this nicely, I saw it from the backlog, when Ori pasted it)
# Ubuntu releasing a new rsyslog package to precise today.
# puppet having the rsyslog package as ensure => latest (puppet, base module) and upgrading to the new version automatically (puppet.log, dpkg.log).
puppet ran across all boxes in the 30' window; the rsyslog package was upgraded, the package's postinst restarted the service, Swift lost the socket, busy looped, got overloaded and caused all kinds of errorenous responses, including cascading into the imagescaler cluster.
Iv'e fixed the evil (3) behavior with https://gerrit.wikimedia.org/r/99576. I'll have a look at (1) and see if it's fixed in newer upstream.
== Lessons learned ==
* don't automatically upgrade rsyslog at least until the bug in Swift is fixed.
== Actions ==
* Upstream: Swift daemons die when syslog stops running LP:1094230
** Abandoned (because of inactivity) change to fix: https://review.openstack.org/#/c/24871
* - Figure out something since the upstream issue probably won't be resolved:
** ALT1: use udp for syslog messages from swift?
** ALT2: upstart hook to restart swift when syslog is restarted?
** DONE: swift machines moved to trusty in which doesn't seem to be affected
* - ping swift people (Faidon) - (unnecessary)
* - syslog is not autoupgraded, so that shouldn't happen again
Category:Incident documentation
Incidents/20140121-BitsApplicationServers

== Summary ==
A change to the default JavaScript and CSS payload of pages across Wikimedia projects caused the Bits application servers to overload, resulting in some users being served pages with no CSS or JavaScript for a period lasting about eighteen minutes.
== Timeline ==
* 11:24 - Change deployed to 1.23wmf11
* 11:25 - Change deployed to 1.23wmf10
* 11:28 - mw1149, mw1150, mw1151 and mw1152 begin flapping, with icinga-wm reporting alternating socket timeouts / recovery. Users on IRC were now reporting that some pages were loading without CSS or JavaScript. Ganglia shows a spike in busy Apache workers.
* 11:32 - Load peaks; hosts begin to recover.
* 11:35 - Smaller second peak
* 11:46 - All hosts recovered.
== Actionables ==
Already on RT according to mark:
* Ensure changes to default JS / CSS payload are deployed in a staggered fashion.
* Add redundancy to Bits application cluster or dissolve into broader app server pool.
Category:Incident documentation
Incidents/20140211-Parsoid

== Summary ==
Verbose logging combined with broken log rotation led to disks on about 3/4 of the Parsoid nodes filling up, which caused the Parsoid daemons to stop accepting requests. This led to some user-visible errors for VisualEditor users in a 17-minute window. An estimate is that less than 10% of VE page loads / saves were affected during this period.
== Timeline ==
All times UTC on Tuesday the 11th (03:00 UTC = 7pm PST on Monday evening):
03:02 First disk space alerts for wtp* [1]
03:06 First connection refused alert
03:10 <springle> those look real. root full on wtp1008
03:12 Most wtp* servers now refusing connections [2]
03:12 Sean removes log file on wtp1008 and restarts service
03:14 parsoid.svc.eqiad.wmnet LVS check goes CRITICAL, sends pages but not to me
03:16 wtp1008 comes back up
03:17 Roan gets on IRC, having been dragged out of a conversation by Erik
03:20 wtp1021 and the LVS check magically come back up (??)
03:20 Roan saves a copy of wtp1005's log for analysis; later discovered it
was the wrong file
03:23 Roan starts a rolling restart of the Parsoid cluster using the
command documented on wikitech
03:26 The LVS check goes CRITICAL again; wtp10{01,02,04,10} go down
03:28 Roan uses the old init script to restart Parsoid instead
03:29 Entire Parsoid cluster comes back up
== Conclusions ==
* Log rotation in puppet was not properly tested, and did not run often enough to prevent failures
* Current Parsoid logging via stdout/stderr redirection can block. Work on async logging is ongoing, but was not ready before this outage.
* Disk space monitoring on Parsoid boxes should trigger much earlier
* Need to better check the logging volume in the Parsoid tests (recursion bug in error logging code produced megabytes of log data per error)
* salt restarts were using old init script instead of upstart, see bug
== Actionables ==
* - Fix log rotation, run it hourly instead of daily
* - Remove old init scripts and update documentation on the log file path
* - Lower the warning threshold on parsoid node disk space to provide time to react
** RT 6851
* - Finish migration to async logging backend in Parsoid so that a full disk does not affect the service availability
* - Check the logging volume in Parsoid unit tests, less critical once logging is async
Category:Incident documentation
Incidents/20140318-EventLogging

== Summary ==
Due to repeated database connection failures an EventLogging writer automatically shut down on 2014-03-18. Although Icinga alerted about the issue, it was only fixed when people noticed reports showing no up-to-date data. 
== Timeline ==
From Ori's email to the Analytics list
At about 2014-03-18 00:04 UTC, db1047 stopped accepting incoming
connections. At some point during the subsequent hour, MariaDB had either
crashed or been manually restarted. Sean noticed that the database was
choking on some queries from the researchers and notified the wmfresearch
list.
During the time that the database server was out or rejecting connection,
the EventLogging writer that writes to db1047 was repeatedly failing to
connect to it:
:sqlalchemy.exc.OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on 'db1047.eqiad.wmnet' (111)")
The Upstart job for EventLogging is configured to re-spawn the writer, up
to a certain threshold of failures. Because the writer repeatedly failed to
connect, it hit the threshold, and was not re-spawned.
This triggered an Icinga alert:
:[00:04:24] <icinga-wm> PROBLEM - Check status of defined EventLogging jobs on vanadium is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/mysql-db1047
This alert was not responded to. Ori finally got pinged by Tillman, who
noticed the blog visitor stats report was blank, and by Gilles, who noticed
image loading performance data was missing.
== Conclusions ==
* Mixing analytics slaves with other work is fragile, as the root cause for the db1047 downtime were research queriesLook at timestamp 08:54:10f in the Ops channel log.
* There is no clarity who is to respond to EventLogging alerts by when.
== Actionables ==
* - Clarify who is to respond to which EventLogging alerts by when.
** Analytics now owns EL, and can respond to some alerts, but not all. Discussion between Ops and Analytics is still ongoing.
* - - Move EventLogging database to m2.
* - Figure out a way to allow joining EventLogging data against enwiki, as this seems to be critical for researchers.
:Replication back to db1047 is included in the required events to move EventLogging database to m2.
== References ==
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/20140328-DB-Queries

== Summary ==
We have more than one db user because it lets us treat wmf jobs and tasks specially. We should avoid breaking that.
== Timeline ==
The symptoms we saw were jobs that normally run as the wikiadmin DB user, were running as wikiuser on the dbs. This started around 6 p.m. UTC on Thursday March 27 but was not noticed right away. Sean saw it and told me about it when I came on line Friday morning my time. We spent some time looking at repos and git logs before realizing that if it was scap related we might be looking at a file that had changed up to a week ago.
Our maintenance scripts and our xml dumps run as the wikiadmin user; this is a user that gets treated specially. One of the forms of special treatment is that jobs run as this user do not get shot after 5 minutes. This meant that once we tracked down the issue we had to rerun a number of jobs (I had a number of broken dumps for instance).
== Conclusions ==
The cause was a change made to PrivateSettings.php which, in keeping with the release notes for mw 1.23, removed the remaining references to AdminSettings.php (though thank goodness not the file itself).
PrivateSettings.php isn't in a repo so we wouldn't be notified about the change or be able even to be sure what changed. Additionally, the change didn't take effect until the next scap a week later. Boom!
== Actionables ==
* - PrivateSettings.php should be in a repo so we can be sure what's changed.
** 
* - Db user and password settings should go into PrivateSettings (and not be removed from AdminSettings until anyone relying on that file has converted their jobs).
** 
* - Changes made should go out immediately as they do for all configuration files.
* - Better coordination
== Warnings ==
Warnings:
This time all we had to do was waste some hours and rerun some jobs. In the future the grants for our db users are likely to be different, which could mean outright breakage of all rather than just long-running tasks. WikiDb updates go via wikiadmin I believe, as well as the job queue.
Category:Incident documentation
Incidents/20140403-Deploy

== Summary ==
=== Timeline ===
Thursday:
* Broke Thursday night during the nightly l10nupdate.
* debugging etc with Andrew and Chad
* Chad reverted all phase0 wikis but testwiki to wmf20
* Niklas woke up and fixed testwiki (see his section below)
Friday:
* Bryan and Roan have long discussion in -operations trying to figure out what exactly made it break
* Greg suggested to Bryan to put us back to all phase0 wikis to wmf21, he did
* Friday night it broke again with the nightly l10nupdate
* Andrew re-ran l10n by hand, didn't fix it
* Andrew put us back again to all phase0 wikis but testwiki to wmf20
=== Roan ===
(responding to Chad/Andrew's late night attempt at determining cause)
The actual issue last time was with ExtensionMessages-wmfNN.php not being regenerated at the right time so the new JSON files weren't included. I believe this can be fixed by running scap a second time. Someone who isn't about to go to sleep (a European perhaps) should try that and see if it fixes testwiki.
I should have flagged this after fixing it last week, but I totally forgot, because it was late and Ori had to babysit the last step while I drove home.
=== Niklas ===
As far as I understand this had nothing to do with LU. As Roan already mentioned, we had this issue last time and it was caused by ExtensionMessages-X.php being out of date. Before last branch point core i18n files were converted to json, and they were added to $wgMessagesDirs. That and many more entries were missing from ExtensionMessages-1.23wmf21.
This seems to be confirmed by the fact that running "mw-update-l10n", "sync-l10nupdate" and "sync-l10nupdate-1 1.23wmf21" fixed test.wikipedia.org.
I can share the blame with Roan of not following up that issue. Now that conversions are over, core is unlikely to break again, but new extensions might be broken due to same cause?
While debugging and browsing through the docs, I found many places where the docs were outdated 1, 2, 3. I also cancelled "sync-l10nupdate-1 1.23wmf21" after ~30 minutes as it gave no progress output, so please do full scap before moving more wikis to 1.23wmf21.
=== Bryan ===
The creation of ExtensionMessages-X.php for a new branch has been broken on and off for quite some time 0. I thought I had finally squashed this when I added the `--force` to the l10n build following the bootstrap phase 1, but I wouldn't be surprised at all if the php -> json changes found another way to break generating the file. It would be really nice to fix all of this so that you don't have to have an l10n cache in order to generate the file that tells you what to add to the 10n cache 2.
== Actions ==
* - Fix ExtensionMessages-X.php generation
* - Review Niklas' doc fixes
* - revive and come to conclusion on testwiki served from mw1017 issue
** 
Category:Incident documentation
Incidents/20140509-EventLogging

== Summary ==
From 5/09 to 6/06 we had issues regarding throughput. Vanadium was not able to validate as many events as they were received and thus, events were being dropped. The issue was affecting all events. Users that are extrapolating absolute counts from data in EL will have likely seen small drops of data in the same interval. The interval in which we dropped data is marked in red the image below. Root case was CPU maxed out in Vanadium.
== Timeline ==
Publishing of events to EventLogging was increased to about twice the usual volume of data. The MediaViewer team had just recently instrumented their code 
and they were logging at too high of rate. 
EventLogging lives in one machine (vanadium) where events are published in raw json (blue line below) and they are validated againsta json schema before hitting the database (green line below).
We found out that our validation throughput was capped to 250 events per second in vanadium due to heavy usage of CPU resources by the validation code. Thus, we were dropping events that 
we could not validate. The time in which this was happening is marked in red on the graph below.
 
There should always be a small disparity in the raw versus validated events, as some events sent are not valid. Now, the difference present in the graph below among both streams
is much too high. 
== Conclusions ==
We learned that we were CPU-bound when it came to event validation. We looked for optimizations we could do in the code and we
found we were validating event schemas too frequently.
We did a fix on that regard here: https://gerrit.wikimedia.org/r/#/c/137810/
It is important to know that the drop of data will be seen in our users dashboards. Specially for metrics that calculate absolute counts from a given 
sampling rate. The drop of data will affect all events equally.
== Actionables ==
* - We have set up alarms on Event Logging regarding throughput so analytics team is notified if events per second go beyond a certain threshold.
* - We have set a policy of notifying our users when these type of events happen.
* - Repopulate data?
** Analytics talked to the affected teams and looks like there is no need to repopulate the data lost.
Category:Incident documentation Category:EventLogging/Incident documentation
Incidents/20140526-m1

== Summary ==
The kernel OOM killer sniped the mysqld process on the m1-master, db1001. This would have had the immediate, though brief, effect of taking down:
* bacula
* blog
* bugzilla
* civicrm
* etherpad
* puppet
* racktables
* rt
== Timeline ==
At 2014-05-26 08:30:25 the kernel OOM killer sniped the mysqld process on the m1-master, db1001. The DB restarted and InnoDB crash recovery completed, leaving only a warning due to that service still using a MyISAM table for fulltext search which failed startup checks and required manual repair. Hashar and _joe_ resolved that.
Why did mysqld run out of memory? Ganglia shows something unusual happened on db1001 around that time of 0830.
Oddly, db1001 syslog reported that the actual process triggering the OOM was diamond, the graphite collector. Is it possible to prove that diamond was just in the wrong place at the wrong time, or that it possibly played a more critical role?
Tendril logged a spike in mysqld connections, aborted connections, bytes recv/sent, and queries on db1001 at the same time. It tentatively assigns blame to "bloguser" based on user and query sampling, however the relevant reporting script is new and untested and the sample rate is set too low to prove the culprit without doubt.
== Conclusions ==
An RDBMS was hit by a spike in traffic and overwhelmed.
== Actionables ==
In the short term:
* MAX_USER_CONNECTIONS is 512 for "bloguser" on db1001.
* The affected MyISAM tables have been switched to ARIA.
In the long term:
* Phabricator will use proper InnoDB tables, plus Elasticsearch.
* Tendril query sampling will steadily reach more hosts.
Category:Incident documentation
Incidents/20140529-appservers

== Summary ==
At 20:25 UTC, change ("Remove wikimedia-task-appserver from app servers") was merged. There were two things wrong with it:
# The package was configured to delete the mwdeploy and apache users upon removal. The apache user was not deleted because it was logged in, but the mwdeploy user was. The mwdeploy account was declared in Puppet, but there was a gap between the removal of the package and the next Puppet run during which the account would not be present.
# The package included the symlinks /etc/apache2/wmf and /usr/local/apache/common, which were not Puppetized. These symlinks were unlinked when the package was removed.
Apache was configured to load configuration files from /etc/apache2/wmf, and these include the files that declare the DocumentRoot and Directory directives for our sites. As a result, users were served with 404s. At 20:40 Faidon re-installed wikimedia-task-appserver on all Apaches. Since 404s are cached in Varnish, it took another five minutes for the rate of 4xx responses to return to normal (20:45).
== Timeline ==
* 20:25 - merged
* 20:26 - 4xx rate begins to rise
* 20:40 - wikimedia-task-appserver reinstalled
* 20:43 - 4xx rate starts to drop
* 20:46 - 4xx rate normal
== Actionables ==
* - wikimedia-task-appserver is no more, and site is operational.
* - The postrm script of packages should be inspected prior to their removal from nodes that power critical services.
Category:Incident documentation