nyuuzyou/wikis huggingface dataset:
Incidents/20120120-Varnish

==Problem==
Some US users of WP site experienced slow page rendering time for about 10 - 15 minutes (from 23:18 UTC to 23.33 UTC) on 20th Jan, 2012, because the Ashburn (EQIAD) bits.wikimedia.org servers were overloaded. Most however had incorrectly formatted wiki pages due to missing javascript and css.
==Background==
One of the Operations Engineers was investigating an earlier report that the bits servers at ESAM were experiencing occasional network saturation.
It turned out that even though there were two bounded network cards, outbound traffic was sent through only one of them due to all the traffic having the same next-hop (the gateway). The fix was to change the kernel's xmit_hash_policy to hash the address destination using layer2+3 in all of our bonding algorithms instead layer2 (default). Correctly balancing the traffic across both links effectively doubled the network throughput of those servers. More detail on xmit_hash_policy can be found on the kernel.org bonding page.
==Root Cause==
After putting in the fix into Puppet and pushing out the interface bonding interface, we had to a "ifdown/ifup" on all the bonded interfaces. When the loss of connectivity happened, the varnish processes on both of these servers spiralled upwards out of control. (Note: Varnish has this behavior of spawning threads when it experiences network package lost.)
==Fix==
We redirected the bits.wikimedia.org traffic from our Ashburn (EQIAD) site to our Tampa (SDTPA) site and that resolved the issue instantly. At the same time, we restarted the affected EQIAD Varnish instances. (We switched back to EQIAD shortly after that.)
==Recommendation==
Operations will look into a way to implement an automatic failover solution (from EQIAD to SDTPA and vice-versa). Ops is also investigating limits within varnish to prevent this type of query overload from bringing down the servers (similar to the use of Apache max clients).
Category:Incident documentation
Incidents/20120206-Squid

== Outage Summary ==
* Duration: about 25 minutes
* Effect: Thumbnails failed to load giving a squid error. Some wiki pages on all projects and languages gave a database error for both logged in and logged out users. Roughly 17% of all page views resulted in 500 errors for about 5 minutes, ~8% for about 10 minutes, and a small percentage for the other 10 minutes.
* Cause: Human error (restarted too many squids too fast)
* Resolution: Manually bringing each failed squid back into service. Waiting for disk and checks to complete.
==Detail==
During the swift deploy I pushed out a squid config for upload.wikimedia.org that contained errors. When squid reads a config that contains errors, it exits. The deploy process only HUPs squid (never restarts it) so after reverting the change, affected squids failed to reload. Hoping to kick them back into service, I restarted them. Unfortunately, I restarted both text and upload squids instead of just upload squids. After restarting, many of them went into a forced cache consistency check. During this check, squid passes all requests through to the back end instead of returning cached results. The apache cluster was able to handle the flood of traffic, but it overwhelmed db40, the parser cache. The parser cache is backed first by memcached then db40, so frequently accessed pages (such as those that nagios and watchmouse check) remained available, since their parser-cache data is in memcache. This split (where some pages have their parser data in memcache and the rest in the db) is why only a portion of page loads failed. Ganglia graphs from emery show the total number of 5xx responses which, when compared to the total number of page views, gives you the percentage failed page views. Once the upload squids fully restarted ms5 also became overloaded and so loading and creating thumbnails slowed down to the point where some thumbnails failed to load. This caused thumbnails to be completely unavailable for a short time at the beginning of the outage and intermittently slow or unavailable for the remaining 20 minutes while ms5 caught up with demand.
==For next time==
* Always deploy squid config changes to a specific host and test there before deploying to the rest of the cluster. Our deploy tools make this easy, and 'curl' is your friend:
** ./deploy sq86
*** # only deploy to sq86
** curl -o /tmp/foo -vvv -H "Host: upload.wikimedia.org" http://sq86.wikimedia.org:3128//wikipedia/commons/thumb/a/a2/Little_kitten_.jpg/62px-Little_kitten_.jpg 
*** # request specifically from sq38's backend squid process
* Avoid restarting squids at all costs. It takes a long time (sometimes 20 minutes) if they need to verify their cache.
* Our squids are not in sequential groups, it just looks that way sometimes. Always use a specific list of squids, not a range.
Category:Incident documentation
Incidents/20120430-MWCoreVersions

== Summary ==
We experienced some problems with failed images, scripts, and other static resources (anything served from bits.wikimedia.org). This problem started 2012-04-30 23:34 UTC (4:34pm PDT), and was cleared up 2012-05-01 00:26 UTC (5:26pm PDT).
 
== Details ==
This started as a problem isolated to test2.wikipedia.org, as part of the 1.20wmf2 deployment. It turns out that, in the process of deploying MediaWiki to those hosts, we filled up the main partition on some of them (notably mw60). This manifested itself as some URLs (notably this one) returning a persistent 404 error, even though the file appeared to be there. Appending "?foobar" to the end of a broken URL caused the resource to load. We determined that the problem was that, for this URL, it was trying to load from mw60, which didn't have the file. Since we've enabled a feature in varnish to persistently request from a single back end, the 404 was persistent.
In an attempt to fix the problem, we deleted the "php-1.19" directory from all hosts, which (we thought) was no longer needed, since 1.20wmf1 is used. This freed up a substantial amount of space. However, php was symlinked to php-1.19, and many scripts were expecting to find the images there, resulting in 503 errors for all sites (both 1.20wmf1 and 1.20wmf2). We deleted the broken php->php-1.19 symlink, symlinked php to php-1.20wmf1, and attempted to push the link out. Unfortunately, our sync scripts dereferenced the link, and started pushing out the whole directory. Switching tactics, we manually used dsh to delete the php directory and replaced it with the proper symlink. After this, the problem was resolved.
== Things to work on ==
* Need to use tools for checking availability of resources on all sites in the event of 404s. Had we done this, we would have caught the mw60 issue sooner than we did.
* Need to make php symlink fixup part of the deployment script process, and hopefully deprecate usage of these. See bug 36363
* Need to fix up partitioning on our Apaches, so that we aren't constantly hitting artificial resource constraints.
Category:Incident documentation
Incidents/20120607-LastModifiedExtension

== Outage Summary ==
* Duration: from about 0235 UTC to 0355 UTC; approximately 80 minutes
* Impact: API service was flapping throughout that period
* Cause: LastModified extension was causing every page view to make a POST request to http://en.wikipedia.org/w/api.php.
* Resolution: Manually disabled it.
==Detail==
Here is the update from Tim Starling on this site incident.
A number of people complained on IRC that the API was mostly down, with 503 errors especially coming from cp1004 and cp1005.
On cp1005, the CPU for the backend squid was maxed out at 100% of one core. Cachemgr for cp1005:3128 reported that there were 1000 connections open to the non-API backend, almost all of which were for the URL "http://en.wikipedia.org/w/api.php". Presumably cp1004:3128 was also overloaded and the frontend squids were falling over to cp1005.
I navigated to Wikipedia with Firebug enabled and discovered that every page view was leading to an attempted POST request to http://en.wikipedia.org/w/api.php. Now armed with the post parameters, and with jeremyb's help on IRC, I was able to identify LastModified as the cause. I disabled it, and the CPU usage of the backend squid on cp1005 dropped to 40%.
If cp1004 and cp1005 hadn't implicitly limited the request rate, then it's very likely that the main Apache cluster would have been overloaded and the whole site would have gone down.
Analysis was made somewhat more difficult by the fact that ganglia was not working. On the two gmond aggregators for the eqiad text cluster, cp1001 and cp1002, gmond was using 100% CPU and tens of gigabytes of RAM. No data was visible for the last 3 days. I haven't identified the root cause yet. Restarting gmond didn't fix it.
-- Tim Starling
Category:Incident documentation
Incidents/20120620-srv268Load

== Site Incident Summary ==
* Duration: from about 20:20 UTC to 21:00 UTC; approximately 40 minutes
* Impact: Intermittent performance problem
* Cause: It is still unknown at this point why srv268 had a load spike. The team is still investigating it.
* Resolution: Manually rebooted that apache/memcached servers.
==Detail==
Here is the update from Faidon on this site incident.
At 20:21 UTC we started getting alerts on IRC (but not pages, see below)
with several application servers getting Apache HTTP timeout and
subsequently, LVS IPv4 CRITICAL alerts. The LVS alerts were for
foundation-lb.{pmtpa,esams}, wikiversity-lb.{pmtpa,esams} and
appservers.svc.pmtpa.wmnet. We got no reports for other service IPs,
even though we got reports of slowness on IRC & enwiki village pump for
them as well. We didn't get alerts for IPv6 LBs or SSL either.
Leslie, Peter and myself were online and began investigating and Asher
logged in halfway through and joined us as well.
It took us quite a while (more than half an hour) of going through
dozens of alerts, but at some point we noticed that one of the
application servers, srv268, was getting increased load for unknown
reasons. Other appservers were having normal load but were extremely
slow at serving responses.
As Asher explained, srv268 is part of memcached slot 0, and that's where
the key "enwiki:lag_times:db38" maps to (with db38 being the enwiki
master). That key was requested on every enwiki article among other
things. Since srv268 was slow but not completely down and our custom
memcache client doesn't handle timeouts well, the Apache children were
delaying the requests while waiting for a reply.
We both power reset srv268 and replaced it in the memcache pool with
another server and things quickly returned to normal.
It is still unknown at this point why srv268 had a load spike in the
first place. Besides that, and according to Asher, there are two bugs at
hand:
 1) memcache client not handling timeouts: it's in the process of
 being replaced, with the MW code already there. RT #3155
 2) MediaWiki requesting the lag for the db master. There's an old bug
 in MW for this one.
- Faidon
Category:Incident documentation
Incidents/20120806-Fibercut

== Outage Summary ==
Wikimedia sites experienced an outage on 6th August 2012 that started at about 6:15am PDT (13:15 UTC). Except for the mobile site, the sites were brought back up by 7:18am PDT (14:18 UTC). Mobile site services resumed at about 8:35 am PDT (15:35 UTC). The team worked around the outage by rerouting traffic to Tampa, bypassing the Ashburn site and failing over services to Tampa data center.
* Duration: From about 13:15 UTC to 14:18 UTC; approximately 63 minutes
* Impact: Wikimedia sites were down throughout that period. The mobile site was not up till 15:35 UTC)
* Cause: Fiber cut - resulting in network connectivity loss
* Resolution: Fail-over network traffic and services from Ashburn to Tampa data center
==Detail==
:At about 6:15am PDT, we were alerted to a site issue and our team found severed network connectivity between our two data centers. Upon checking with our network provider in Tampa, they informed us that a third party crew working in the Tampa area inflicted a fiber cut and caused the outage.
:The data centers — one in Ashburn, Virginia and the other in Tampa, Florida — are connected by two separate fiber links (for redundancy). While Ashburn serves most of the traffic, it needs to talk to our Tampa data center for backend services (e.g. database).
:To provide network redundancy, we engaged our Tampa provider to supply us two independent and segregated DWDM systems to deliver the Wikimedia services. Each of these DWDM systems is routed over diverse fibers using the dual entrances into the carrier's Tampa POP, making the design capable of delivering the two diversely routed 10G waves to Wikimedia as long as the metro segment of wave #1 and long haul segment of wave #1 are on the same route. 
:Our provider, after performing initial troubleshooting of the Wikimedia waves, revealed the location of the unexpected alarms. It occurred in a section of a folded fiber segment through which both of the unprotected Wikimedia services traversed.  The fiber cable damage occurred within this folded fiber segment causing the loss of service. The post outage investigation also showed that the metro access segment of wave #1 was incorrectly routed on the same side of the long haul segment of wave #2. That fiber cut and the incorrect routing resulted in lost of the two network connections between our data centers.
:We have since asked the provider to audit our waves to ensure such single point of failure is no longer in their system. We are also in the midst of replicating and migrating the rest of our backend services to Eqiad, creating full service redundancy in the two data-centers. The plan is to complete that work in Q2.
Category:Incident documentation
Incidents/20130315-AFTv5

==Outage Summary==
Wikimedia sites experienced an outage on 15th March 2013 from about 16:14 to about 16.23 PDT (23:14 to 23:23 UTC). 
* Duration: From about 23:14 UTC to 23:23 UTC; approximately 9 minutes
* Impact: Wikimedia sites were not available for edits; Most users were able to continue use the sites for 'read'.
* Cause: Database contention issues caused by the new version of AFTv5 that was deployed
* Resolution: Reverted the deployed extension
==Detail==
(per Asher's email)
:We experienced an site outage this afternoon across all projects. Service was degraded for non edge cached requests between approximately 4:13 - 4:23pm PDT.
:Initial root cause appears to be database contention issues caused by the new version of AFTv5 that was deployed today. Once identifying that as the likely issue, I asked Ryan to disable the extension. The cluster returned to normal as soon as that deploy went out. 
:I actually thought that todays AFTv5 deploy was backed out, so this was a bit of a surprise. The new version of AFTv5 uses its very own database shard that currently does nothing else. The entire dataset for enwiki is only around 350Mb and 580k rows. Yet most apache children were tied up waiting on queries to the slave in that shard (db1030).
 
:I saved the output from "show full processlist" at fenari:/home/asher/db/aftv5-outage-db1030-proclist. It's 22Mb. Kind of amazing! I checked explains for a few of the queries and those that I picked weren't covered by any index (i.e. searching for feedback by ip address on aft_user_text which is unindexed) and all included an ORDER BY + LIMIT. Not bad if that's covered by an index, not good if the query can't use one.
== Response from Matthias ==
copy/paste from his email
The culprit was query hooking into "my contributions", one I had completely overlooked. If you were to attempt to construct a query avoiding all possible indexed columns, this would have to be the leading example.
Category:Incident documentation
Incidents/20130418-Jenkins

from Antoine's email to engineering@
Jenkins has been down for a couple hours on Wednesday April 18th from
9pm to 11pm.
== Summary ==
The root cause is that I forgot to deploy a configuration change for the
Zuul daemon (the software that takes patches from Gerrit and trigger
jobs in Jenkins).
The consequence is that on each job, git had to copy the repository
being tested from a disk to another one instead of using hardlinks.
That caused jobs to take a huge time and basically killed Jenkins.
== Chain of events ==
The chain of events (all times are GMT) is roughly:
Friday 12th April:
* Operations team kindly inserted an SSD device in the continuous integration server. The device has been mounted on /srv/ssd/
* At 3pm I have configured Jenkins to use the new workspace.
* At 4pm I prepare Zuul related changes to later migrate its git repositories to the SSD.
Monday 15th April:
* The CPU usage has been fine most of the weekend.
* 9:20am Restarted Jenkins due to an unrelated bug.
* 10pm Load raise a bit but nothing to worrying. I am barely monitoring the activity while doing conf calls.
Tuesday 16th April:
* 4pm load start raising.
* 10pm Jenkins is stalled with a huge queue
* around midnight an attempt is made to tweak the Jenkins jobs to use the replicated Gerrit repositories.
Wednesday 17th April:
* 6:45am the tweak is reverted. The Jenkins jobs must use the Zuul repositories which contains the patchset merged on the tip of the branch. The symptoms were that all jobs were not able to fetch the refspec they needed.
* During the morning, a few more jobs need to be updated. I investigate the git slowness and eventually move to something else.
* 8pm Jenkins job queue is full. I track down the actual root cause:
: jobs are cloning from the slow disks to the ssd. Being different devices, git has to fully copy the repository instead of using hardlinks.
* 8pm18 I shutdown Zuul to stop triggering new jobs and cancel all pending Jenkins jobs.
* 8pm30 I migrate the Zuul repositories to be on the same disks the Jenkins jobs are. That would let git clone creates hardlink and dramatically speed up the cloning processes.
* The changes I have prepared on Friday 12th get merged and applied on the server. Thus making Zuul use the new configuration.
* 9pm16 Git repositories have been migrated.
* 9pm20 Jenkins is starting.
* 10pm Jenkins is back up.
* Started migrating all the jobs to use the new git path
* 11pm Jobs migrated. Zuul restarted.
== Lesson learned ==
* I should not have switched Jenkins workspaces on Friday. That should have been done on Monday together with the Zuul changes.
* git clone can be scary.
* hardlinks do not work from a device to another one.
* iotop is a must
* Jenkins need to start up faster (there is a patch upstream) [fixed May 1st, 2013 bugzilla:47120)
* Testing in labs does not catch everything.
Category:Incident documentation
Incidents/20130508-memcached

== Problem ==
copy/pasted from Mark Bergsma's email
We had an outage today, due to a SFP+ module for a single memcached server (mc1009) going bad in a switch, causing the switch to partially reboot, and causing all application servers to get blocked due to a single down memcached. As usual this caused all kinds of cascading failures on other clusters such as Squid/Varnish. When not overloaded, these clusters would only serve cached pages at that point.
At 13:30 UTC, the EEPROM of the SFP+ module for mc1009 on asw2-a5-eqiad became unreadable over the i2c bus, and apparently this caused the switch to reboot its entire ethernet chassis subsystem. This gave a lot of noise, with the entire rack going offline, and lots of monitoring and log messages. Fortunately, the other servers came back up after a minute or two. I noticed the switch problems after a few minutes, but it took me a while to check and wade through everything and determine that in the end only a single memcached server was still missing. At 13:58 I removed mc1009 from the MediaWiki memcached pool, and two minutes later, all services were back up.
Obviously having all Memcached servers still on a single switch in a single rack is very bad, and I've just restarted our old plans to move half of them (as well as half of Varnish) to another rack in row C.
Also very bad is that the failure of a single memcached server can still knock us over completely. Can we not improve on this, with the new PECL client? Or what's the status of our plans with twemproxy/nutcracker?
== Solution ==
copy/pasted from Asher Feldman's email
As of last night, we've been routing all memcached requests in eqiad via twemproxy. A short while ago, I rebooted mc1009 without any other preparation. The main impact was just log noise (libmemcached can be misleading when logging "SERVER HAS FAILED AND IS DISABLED UNTIL TIMED") but as far as I've been able to tell, there was no impact to site
availability or general performance. Graphite showed an increase in external store selects as I'd expect but php request latency (at the average, 50th, and 90th percentiles) was unimpacted, as were memcached get times. Using enwiki as a logged in user seemed completely normal, and I found no user reports of issues during that time frame either. Huge difference vs. the mc1009 failure on May 6 that started this thread! It also helps shave around 12ms of setup time from every request.
The twemproxy deployment is still provisional and needs additional work (mostly around logging and stats collection) but I'm pleased so far and am going to proceed.
=== Startup Performance ===
90th percentile Setup.php-memcached time went from 14.8ms to 4ms as a result of the libmemcached upgrade alone, and to 0.9ms with twemproxy.
Category:Incident documentation
Incidents/20130607-CI

== Summary ==
Continuous Integration services fell over.
Two associated bug report for this incident: bugzilla:49294 and bugzilla:49330
== What Happened ==
# Niklas Laxström reported that CI was stuck. Via email to Engineering on 09:03 (UTC) and via Bugzilla at 08:41 (UTC)
# Beta was running out of space - Antoine fixed this so the Jenkins job queue was able to catch back up
# Gerrit replication was failing to antimony, causing the replication jobs to get re-queued and clog Gerrit. This blocked stream-events which caused Zuul to not see anything as happening
== Lessons Learned ==
* For #1, better disk space monitoring for Beta would help keep this from happening again.
* For #2, we need more monitoring for Gerrit, Jenkins and Zuul to make sure the whole pipeline is working as expected. Antoine and Chad have already discussed taking care of this together first thing next week when he's back from vacation. Also we should make replication a little less greedy on retrying when it fails...this has caused cascading problems before.
== Action Items ==
Explicit next steps to prevent this from happening again as much as possible, with Bugzilla bugs or RT tickets linked for every step.
* Chad and Antoine will work on better monitoring of CI infrastructure (see above).
Category:Incident documentation
Incidents/20130617-AccountCreation

== Summary ==
Due to code in VisualEditor's A/B testing sorting on new accounts, newly created user accounts were not having their email and password set. Thus, many thousands of new accounts were only partially created.
Reported as bugzilla:49727.
== What Happened ==
Basically, this commit caused the whole thing. This one fixed it.
== Lessons Learned ==
Our logging works, but we don't monitor those logs and report spikes/errors.
== Action Items ==
* Errors and exceptions are currently broadcast to fluorine and vanadium via UDP. Ori has code that parses the stream and generates the Ganglia graphs, but it isn't hooked up to Icinga or any other form of monitoring. We need someone from Ops to pair with Ori to set this up. This is bugzilla:49757. 
Category:Incident documentation
Incidents/20130725-Wikivoyage

== Summary ==
Wikis on 1.22wmf11 that use Wikibase (only testwikis and Wikivoyage at the time) went down for 40 minutes with a fatal error on every apache hit.
== Timeline ==
* ...
== Conclusions ==
== Actionables ==
== See also ==
* bug 52038
* ../20130628-Site/
Category:Incident documentation