nyuuzyou/wikis huggingface dataset:
Incidents/2017-03-08 Labstore

== Summary ==
During a planned cleanup of Tools NFS log/err/out files over 100M, an unexpected/unplanned truncate of a large number of files across tools that were over 100M occurred. No tools were reported to be down because of this, and almost all unintended erased files are being restored from the backups in codfw.
== Timeline ==
* [19:02] Madhu makes list of files intended to be truncated in the cleanup and posts it on 
* [19:50] She runs `cat gt100M | truncate -s 0` on the list of files on labstore1005
* [19:54] Looks at tools usage post truncate and realizes something went horribly wrong
* [20:00] She tells Chase, and they realize that the list of files was all the files > 100M, and didn't contain the final filtered list posted on task. 
* [20:04] Madhu posts the list of affected files https://phabricator.wikimedia.org/P5030 and makes lists of files intended to restore, and not being restored(log/err/out files that should have been truncated in the first place)
* [20:20] The last backup in codfw is from ~23 hours before the data loss in eqiad, and Madhu starts an rsync of all the files that should be restored (https://phabricator.wikimedia.org/P5030#26707)
* The rsync completed 1 day later and the error log (with list of files that couldn't be restored) is here - https://phabricator.wikimedia.org/P5030#26817
== Conclusions ==
* This was mostly human error, caused by not super carefully verifying that the list of files being truncated was correct. Thankfully since it was files only over 100M, no running tools/services seemed to be affected, and most of the data lost seemed to historical/large files generated from dumps stored in tools/videos etc. Except for the files that were generated in the 23 hours after the backup snapshot was created at 2017-03-07 20:00:03, everything is recoverable.
* This is also a process error, in that we still do this file cleanup in order to do space conservation manually, and if a well tested automated process to handle this existed, this could have been entirely avoided. This work is already in progress, and we hope to move to that and will stop doing this manually. 
== Actionables ==
* Complete work on https://gerrit.wikimedia.org/r/#/c/326153/ - which well let us truncate logs periodically in an automated way, which will prevent this type of incident in future
* Update mailing lists after the rsync completes on status and files that were lost in the 23 hour period between the backup and the truncate happening.
Category:Incident documentation
Incidents/2017-04-26 ORES

== Summary ==
Today, an ORES deployment resulted in a big pile of timeout errors in CODFW, but not in EQIAD. It looks like uwsgi was running old code while celery was running new code. No service restarts rectified the situation. So we switched traffic to point to EQIAD instead. The problem was then resolved. 
== Timeline ==
;2034 UTC
: ORES deployment is completed
;2041 UTC
: Icinga warns of an outage
;2048 UTC
: Halfak confirms that a bunch of requests are timing out CODFW but EQIAD is doing OK.
;2102 UTC
: Phab:T163944 is created to track the issue.
;2114 UTC
: A service restart is issued via scap and directly (by mutante) via systemctl "100.0% (6/6) success ratio (>= 100.0% threshold) for command: 'systemctl restart uwsgi-ores"
;2135 UTC
: mutante posts a patchset to re-route traffic to eqiad. (https://gerrit.wikimedia.org/r/#/c/350487/)
;2143 UTC
: the patch is merged and puppet is run
;2145 UTC
: everything is OK again
== Conclusions ==
No idea what could have caused this. Filed a task to investigate. T163950 -- Investigate failed deploy to CODFW
;Update @ 2017-05-02
: The problem was caused by scb2005 and scb2006. We had no idea they existed. They weren't in our scap config so new code wasn't getting deployed to them. The problem was solved once we added them to the scap config and deployed to them. 
== Actionables ==
* T163950 -- Investigate failed deploy to CODFW
Category:Incident documentation
Incidents/2017-04-27 archiva

== Summary ==
The archiva.wikimedia.org suffered an intermittent unavailability due it's public IP being stolen by another host between 20:26 and 21:56.
== Timeline ==
* 20:26 The partially decommissioned host gadolinium boot up and used it's old configuration that had the same public IP used by archiva.wikimedia.org (208.80.154.73), making the IP flapping between it's legit owner meitnerium and gadolinium, causing the intermittent unavailability of the archiva.wikimedia.org service.
* 21:48 Riccardo starts investigating.
* 21:56 Riccardo shut down the old decommissioned host gadolinium to stop the outage.
* 22:02 Rob opened a task for the full decommissioning of gadolinium: T164036
== Conclusions ==
* Despite the large number of alarms on IRC due to the flapping nature of the event, no one was around to have a look at it for a long time. On the other hand, none of the alarms that went off triggered a page to the operations team.
* The decommissioned host gadolinium boot up for reasons not yet clear.
* The decommissioned host gadolinium was not fully decommissioned in the past, so when booting up, it still had it's full configuration, that in this case was having the same public IP already announced by another host.
== Actionables ==
* Investigate to know why gadolinium boot up in the first place.
** It was booted up to troubleshoot T158131. It seems whenever this host was returned to spares, it didn't have its switch port disabled, or disks wiped.. Since it was not disabled, when the system booted up, it had full access to the network. The Server_Lifecycle steps are clear on the checklist being used for all reclaim and decommissions, so this shouldn't happen in the future, this host was likely offline and sitting from prior to the checklist implementation.
* Fully decommission gadolinium: T164036
* Improve our alarming to ensure a shorter response time, potentially adding alarms via email and/or improving the ones that page.
* Improve our procedures around decommissioned hosts to ensure that:
** they are fully decommissioned as soon as possible
** they are isolated network wise, in particular if they need to stay racked with their data untouched for backup/recovery reasons
** they are clearly marked as decommissioned in racktables: T164042
Category:Incident documentation
Incidents/2017-04-27 ocg

== Summary ==
During eqiad row D switch maintenance OCG (the service that renders PDFs) suffered an outage due to all of its service hosts being unreachable. During the time of outage users were not able to create PDFs from pages and collections.
== Timeline ==
* 16:55 pybal alert for ocg.svc.eqiad.wmnet being down comes in and investigation begins
* 17:08 _joe_ stops pybal on lvs1006 to stop announcing via BGP
* 17:24 XioNoX restores switch port configuration for LVSes in row D
* 17:27 pybal recovers, ocg.svc.eqiad.wmnet reachable again
* 17:30 _joe_ starts pybal on lvs1006 again
== Conclusions ==
OCG was down for ~30min due to a combination of factors, namely row diversity isn't fully implemented with two service hosts being in the same rack and row that was undergoing maintenance . The remaining OCG host is in row C but under maintenance at the time of the incident .
== Actionables ==
* Complete the OCG deprecation
Incidents/2017-04-27 redis-ipaddress

== Summary ==
A Puppet change was deployed that renamed the temporary fact ipaddress_primary to ipaddress, overriding the stock fact, as part of the work on . The ipaddress_primary fact was already being used in the Redis manifest (and only there), which was missed during the preparation for the merge of that change. This, in combination with the structure and code of the Redis manifests, resulted into the Redis servers failing to respond and cascading into a user-visible login/session outage.
== Timeline ==
* 12:25 Faidon merges Change 8bf0cbda12e910f921367b055d86773c2a415c1a. puppet is deliberately left to gradually roll the change over its regular cycle of 30 minutes.
* --:-- Puppet starts to runs via crontab on mc2* hosts where it was removing the ferm rule to allow the connection on the specified port, causing the connections to them to fail.
* 12:34 Puppet run via crontab on rdb2001, that is one of the master hosts in codfw, active datacenter at this time. The puppet run didn't fail, but failed to retrieve the value of ipaddress_primary, that will later make the rdb slaves fail puppet runs and make them alert on IRC.
* 12:38 Catalog fetch failed alert for rdb1008. Riccardo beings investigating. Faidon has a look as well but misjudges the issue as a low priority one since there was no user-visible impact and catalog failures are usually non-impacting.
* 12:43 Catalog fetch failed alerts for rdb1002, rdb2002.
* 12:51 Catalog fetch failed alert for rdb1006.
* 12:57 MediaWiki exceptions and fatals per minute alert triggers.
* 12:57 Giuseppe finds the root cause of the puppet failure and merges a change.
* 12:58 First user report of users unable to login.
* 13:00 MediaWiki exceptions and fatals per minute alert recovers.
== Conclusions ==
* The severity of the issue was misjudged, especially since there was no page at the time.
* The Redis manifests were perhaps a little too brittle, causing a widespread outage due to what it really was a fact not resolving.
* The call to a fact that doesn't exists do not trigger an error as it should.
* The ferm rule that open the port for redis depends indirectly on a call to a fact
* The hosts that caused the user-facing outage di not alert in any way, only the rdb* hosts started alerting for puppet failures on the slaves.
== Actionables ==
* Make sure to validate every call to facts, considering to add an helper function to ensure this behaviour.
Category:Incident documentation
Incidents/2017-05-07 labs-dns

== Summary ==
Around 05:00UTC on Sunday 2017-05-07 new Labs instance creation started to fail. Andrew noticed the issue later that day. Most of the various parts of designate and powerdns were functioning properly but the AXFR sync between the two was failing, so new records were failing to be registered in the public DNS servers.
Andrew switched the active designate server from labservices1002 to labservices1001, at which point normal service was restored (at around 17:30 UTC). The ultimate cause of the issue remains obscure, although labservices1002 seems generally troubled.
== Timeline ==
* 05:03 icinga alerts because the nova-fullstack process is no longer running. This happens after 7 creation attempts had failed (one every 5 minutes), so most likely the original issue appeared about 30 minutes earlier.
* 16:00 while testing an unrelated fix Andrew notes that new instances are failing to get DNS connections and begins investigation
* 17:00 It's clear that most Designate services are working properly. Instance creation is detected, the new records are created properly and registered in the database. Pdns is not noticing the new records, and the syslog on labservices1001 and 1002 is full of timeout messages rather than proper syncs with designate-mdns:
May 7 06:44:02 labservices1001 pdns[47578]: Received serial number updates for 0 zones, had 161 timeouts
there are also many deadlock warnings in the designate-central log. These are not unheard of during normal operation but are much more frequent than usual:
designate-central.log.1:2017-05-07 15:36:15.730 21604 WARNING designate.central.service [req-5064db6f-5238-491c-a5fa-447464baa1aa - - - - -] Deadlock detected. Retrying...
Puppet runs are also extremely slow on labservices1002, over 5 minutes per run. This is reminiscent of https://phabricator.wikimedia.org/T159835 which was fixed with a reboot.
* 17:20 Andrew gives up on understanding what's happening an begins the failover process to labservices1001 as documented in https://wikitech.wikimedia.org/wiki/Labs_troubleshooting#Fail-over_2
* 17:40 Failover is completed and normal service restored. No DNS interruptions were detected for running services.
== Conclusions ==
This outage was long but probably didn't affect any actual users (other than Andrew.) The outage would have been detected much earlier if the fullstack test paged on failure.
The failover process from labservices1002 to 1001 went very smoothly with little or no additional service interruption.
We still don't know what's going on with labservices1002 -- investigation continues as part of https://phabricator.wikimedia.org/T164675.
== Actionables ==
Category:Incident documentation
* Understand and fix whatever is wrong with labservices1002
* Add paging to the fullstack alert. A single failure is probably not worth paging but if we get 6 in a row and the test switches off I'd like to know about it.
Incidents/2017-06-01 Maps

== Summary ==
Migration of maps to upload caches resulted in query parameters being stripped, breaking some subservices ().
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
== Conclusions ==
* There is never enough monitoring.
* An important change was considered ops-only, was never communicated to its product team, neither beforehand for coordination nor afterwards for a quick check that everything still works.
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
<onlyinclude>
* Add geoshapes and geolines to monitoring spec ()
* Services need external monitoring ()
* Kartographer should handle external data errors gracefully ()
Category:Incident documentation
Incidents/2017-06-07 scrape

== Summary ==
A high request rate from a single bot (same UA/IP) of some specific queries on the API caused high concurrency for such queries on the database for enwiki. As a consequence, those requests were served very slowly. The slowness then propagated to the non-API mediawiki servers due to lagging on the API databases, and those caused an increased load on both the API and standard appserver cluster, see:
https://grafana.wikimedia.org/dashboard/db/prometheus-apache-hhvm-dc-stats?panelId=17&fullscreen&orgId=1&from=1496806612437&to=1496818165547
== Timeline ==
* 04:44 A first, smaller spike of 5xx happens on text caches
* 05:08 Icinga sends the first alarm to IRC < icinga-wm> PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
* 05:28 _joe_ notices the flapping alerts and starts investigating
* 05:29 time correlation with a l10-update is suggested by mutante, but is determined to be a red herring
* 05:31 Investigation on the varnish layer starts. _joe_ notices there are no icinga alarms for mailbox lag and that most 5xx are evenly distributed on all caches
* 05:39 Varnish is ruled out as source of the issue 
* 05:43 Investigation on logstash and oxygen is inconclusive. 
* 05:53 The problem seems to happen for APIs only, mostly concentrated on enwiki. HHVM and apache logs at a first glance do not indicate any issue at the appserver level
* 05:56 A sudden shower of ProxyFetch failures from pybal is noticed for all appserver pools (not just API):
< _joe_> Jun 7 05:55:45 lvs1003 pybal[6265]: [api-https_443 ProxyFetch] WARN: mw1204.eqiad.wmnet (enabled/up/pooled): Fetch failed, 5.891 s
since pybal is fetching Special:BlankPage on both clusters, a problem with s1 is suggested. marostegui is investigating
* 08:02 _joe_ finds evidence of slowness in the API apache logs, in short bursts of a few seconds around the time of the pybal alerts
* 08:01 All service send out recoveries, and the issue doesn't seem to be repeating on the load balancers
* 08:20 After more digging in the logs, _joe_ notices a bot, coming from a single ip at Michigan State University with UA wikiScrape/0.0.0 is doing tons of requests in short bursts to the API for enwiki.
* 08:25 ema confirms that UA/ip combination made huge bursts of requests to the API between 04:44 and 05:56 resulting in 503s. Peak requests were at 1200 req/s
* 09:40 jynus narrows the issue down to the fact that ApiQueryContributions::execute, Revision::getParentLengths, normally taking 1 second to execute (already slow), at the time was taking >10 seconds to execute on S1 because of the extra load
== Conclusions ==
It seems clear that this partial outage was caused by db resources starvation due to a heavy hitter on the APIs. The only way to solve this would be to have a good rate-limiting solution on the API.
== Actionables ==
* Ban the UA/IP in case of need - prepare a patch 
* Implement better rate-limiting for the API
* Investigate and implement varnish-level rate-limiting https://phabricator.wikimedia.org/T163233
Incidents/2017-06-13 ORES

== Summary ==
ORES had an intermittent outage from 1600 - 1940 UTC on June 13th. The issue was traced to scb1001.eqiad.wmnet. 
== Timeline ==
See https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&from=1497366649350&to=1497383640251&panelId=2&fullscreen
* 1600 UTC: Errors rise for ORES (not noticed. no icinga pings) 
* 1700 UTC: Deployment for begins
* 1715 UTC: During canary check, error rate is noted and is created with "Unbreak now"
* 1740 UTC: Problem is independent of deploy. The decision is made to continue with deploy. 
* 1816 UTC: Ops is pulled in (mutante responds). Rollback of deploy is considered but rejected.
* 1828 UTC: Problem is narrowed down to scb1001 specifically. Logs show no errors despite intermittent 500s
* 1923 UTC: Mutante notes that pdf rendering is taking a lot of CPU and kills it
* 1940 UTC: Recovery confirmed.
== Conclusions ==
* icinga didn't tell us about the issue
* for some reason, the error wasn't being written to app.log
* it looks like there was some conflict with resource usage WRT pdf rendering
* memory was very tight on SCB for the duration of the outage: 
== Actionables ==
* -- "Extend icinga check to catch 500 errors like those of the 20170613 incident"
* -- "Limit resources used by ORES", move ORES to dedicated hardware. See .
* Limit resources used by the pdfrender service: 
Category:Incident documentation
Incidents/2017-06-16 PHAB

== Summary ==
Phabricator was down for approximately 8 minutes due to an unintentional/badly executed upgrade.
== Timeline ==
* At approximately 13:18 UTC, I (User:20after4) attempted to fix a minor bug in a phabricator extension (https://phabricator.wikimedia.org/T168058) by deploying https://phabricator.wikimedia.org/rPHEX721eca9b64cc38b495a255bad46cbe11f3d5c390.
* Instead of cherry-picking the dependent change, the code was mistakenly fast-forwarded to a version which depended on new database migrations.
* 13:19 UTC - icinga alerts and several people notice that phabricator is displaying a setup error page instead of the normal application.
* 13:20 UTC - I !log in #wikimedia-operations to let people know that I'm working on it.
* Because phabricator makes it easier to roll forward than to roll back, and my experience that migrations are usually fast, I ran the storage upgrade script to apply the migrations. ''
* 13:22 UTC - I acknowledged the icinga alert to prevent further paging.
* The script (phabricator/resources/sql/autopatches/20170528.maniphestdupes.php) was moderately large and slow compared to most phabricator migration scripts so this took several minutes to complete ''
* 13:25 UTC - Service restored. 
 
== Conclusions ==
* Almost every update to Phabricator requires database migrations to be applied. These are usually quite fast but not always.
== Actionables ==
* Don't attempt code changes on friday, even trivial ones.
* Always assume that code will have unexpected dependencies.
* All upgrades, even minor ones should be during the scheduled window.
* jynus requested that I coordinate with him to schedule the database slave to be offline when running migrations. 
** I will file a task to remind about this (https://phabricator.wikimedia.org/T187143)
Incidents/2017-07-11 EventLogging

== Summary ==
Within the previous month, we started importing EventBus generated events into the MySQL EventLogging Analytics database(s) for https://phabricator.wikimedia.org/T150369. These events have a different schema format and schema repository than the original EventLogging Analytics events.
EventBus style events are versioned just like the EventLogging Analytics ones, but thus far, we had not bumped any version numbers for backwards compatible schema changes. However, now that MySQL is being used for these events, and MySQL tables are not automatically evolved, we must bump EventBus style schemas from now on. This was not done for a change that added a rev_content_changed field (https://gerrit.wikimedia.org/r/#/c/362321/). When events with this new field started flowing in from EventBus, the eventlogging process that was attempting to insert these events died. It was immediately restarted by upstart, but it flapped every 5 minutes, every time a batch of mediawiki_page_create events were attempted to be inserted.
This caused minimal data loss from other unrelated schemas. These events were then backfilled from secondary log files into MySQL. Along the way, Otto forgot that we were now filtering out bot events, and inserted over 2 million extra events into the MySQL databases for the days of 2017-07-10 - 2017-07-12.
== Timeline ==
===Mon July 10 11:31:10 UTC 2017 ===
https://gerrit.wikimedia.org/r/#/c/362322/ is merged. This is slowly deployed to wikis as part of the regular deploy train.
===Mon Apr 6 18:32 UTC 2015===
Roan files https://phabricator.wikimedia.org/T170486. Otto investigates, realizes what happening. He then alters the existing eventbus event tables in MySQL to add the missing rev_content_changed field to allow insertions to happen. He also merges a puppet change (https://gerrit.wikimedia.org/r/#/c/364881/) to separate the processes responsible for inserting EventLogging Analytics events, and events from EventBus to prevent something like this from happening again.
== Conclusions ==
EventBus event schemas from here on out need to have version numbers bumped when we modify them.
== Actionables ==
* Monitor if eventlogging processes flap
Category:Incident documentation
Category:EventLogging/Incident documentation
Incidents/2017-07-20 novaadmin LDAP password

== Summary ==
While on a hangout and trying to reset Coren's LDAP password, Bryan instead changed the password of the uid=novaadmin,ou=people,dc=wikimedia,dc=org account. This broke things like Wikitech, OpenStack Keystone, and Striker that use the novaadmin account to do privileged things in LDAP. The bad password stayed in place long enough that when Bryan tried to change it back LDAP had locked the account and would not let it be changed by the user directly. Andrew was able to use the uid=admin account and credentials to change uid=novadmin's password back to the expected value.
== Timeline ==
* 2017-07-20T01:16:53Z ldappasswd -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -s "uid=marc,ou=people,dc=wikimedia,dc=org"
** "Hmm... that's not right, it didn't prompt me for a new password"
** ldappasswd -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -S "uid=marc,ou=people,dc=wikimedia,dc=org"
** "Huh. It says that auth failed when I try to change your password Coren."
** ldapsearch -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W uid=marc
** "Now I can't even do a search? I must have the wrong password for novadamin."
* 2017-07-20T01:13 < bd808>	keystone may be down all together
* 2017-07-20T01:14 < bd808>	https://tools.wmflabs.org/openstack-browser/ is not working too
* 2017-07-20T01:15 < bd808>	[REACTED]. I think I broke novaadmin's LDAP password
* Bryan tries and fails to set the password back after realizing it is set to Coren's dn due to the in initial typo of -s instead of -S.
* 2017-07-20T01:28:11Z Andrew resets password for uid=novaadmin back to expected value.
== Conclusions ==
* ldappasswd has horrible CLI arguments.
* We should have better tools for regular LDAP maintenance operations. There has been discussion of building a web portal for managing LDAP accounts too.
== Actionables ==
* …
Category:Incident documentation