nyuuzyou/wikis huggingface dataset:
Incidents/2018-10-15 eqsin-power

== Summary ==
The eqsin PoP went offline for approximately 2h10, between approximately 14:15 and 16:25 UTC on 2018-10-14. Of those, 11 minutes were service impacting for the PoP's service region (APAC) for the majority of users, with the long tail of non-conforming resolvers being unable to reach our sites for longer periods of time.
Root cause was a combination of scheduled power maintenance by the facility and lapses in our processes.
== Timeline ==
* Sep 07 08:17 Facility notifies us of a "SERVICE IMPACTING Scheduled Customer Outage Level" with power feed A window.
* Sep 07 08:20 Facility notifies us of a "SERVICE IMPACTING Scheduled Customer Outage Level" with power feed B window.
* Sep 10 10:06 Facility notifies us of power feed B window being rescheduled.
* Oct 12 13:01 Facility sends a "maintenance listed will commence in one hour" notification for power feed A.
* Oct 12 14:16 Power feed A is lost in eqsin, and is noticed by monitoring.
* Oct 12 14:26 Site gets depooled and task is filed.
* Oct 13 00:00 Maintenance window is over, facility sends a "completed" notice.
* Oct 13 03:07 Power feed A is (seemingly) back up, so the site is repooled. Unknown at the time was that multiple alerts were still outstanding at that point, and power had not been recovered in the top-half of rack 0603, which included all networking equipment.
* Oct 14 13:00 Facility sends a "maintenance listed will commence in one hour" notification for power feed B.
* Oct 14 14:15 Power feed B is lost in eqsin, and with it the PoP goes dark. Users in APAC are unable to access our sites. Monitoring notices and pages, multiple respondents react.
* Oct 14 14:16 Site is depooled.
* Oct 14 14:26 DNS TTLs expire for the majority of recursors, site back up for most of APAC users via ulsfo.
* Oct 14 14:38 Facility's service deks is being contacted to report a double power failure.
* Oct 14 16:02 Service desk responds with a work order; will "be contacted by a technician in 30 to 45 minutes".
* Oct 14 16:25 Site is back up, facility technician informs us that they reset a feed A in-rack breaker that was tripped.
* Oct 14 19:24 Power feed B is restored.
* Oct 15 00:00 Maintenance window is over, facility sends a "completed" notice.
* Oct 15 08:48 Site is repooled.
== Conclusions ==
* Manual triaging of maintenance alerts is error-prone.
* Repool should always be preceded by a monitoring health-check where all alerts have been cleared or can be reasonably explained.
* eqsin does not have router or rack redundancy, so a power feed + PDU or PSU failure can take the site down.
* eqsin's PDUs are not monitored by neither the facility nor us.
== Actionables ==
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
Incidents/2018-10-25 train-1.33.0-wmf.1

== Summary ==
* Six blockers closed, one opened on Friday after conclusion of the train.
* Otherwise it was a mostly uneventful train.
== Timeline ==
* Group1 was delayed until Thursday and Group2 finally happened late Thursday evening.
* See 1.33.0-wmf.1 deployment blockers 
== Conclusions ==
* The blockers were perhaps more worrisome than destructive, no outages occurred.
== Actionables ==
None
Incidents/2018-11-06 maps

==Summary==
Tilerator failed on maps100[1-3]. Tilerator is a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage. This happened on the 6th November 2018. Icinga first reported this failure around 00:18 UTC.
==Timeline==
This is a step by step outline of what happened to cause the incident and how it was remedied.
00:18 UTC: Icinga reported failure of Tilerator ports :
PROBLEM - tilerator on maps1003 is CRITICAL: connect to address 10.64.32.117 and port 6534: Connection refused
1:19 AM PROBLEM - tilerator on maps1002 is CRITICAL: connect to address 10.64.16.42 and port 6534: Connection refused
1:19 AM PROBLEM - tilerator on maps1001 is CRITICAL: connect to address 10.64.0.79 and port 6534: Connection refused
07:25 UTC: Tilerator Service was restarted on maps100[1-3]
07:26 UTC: Tilerator Service came back up.
==Conclusions==
* There's need for a non paging alert whenever problem arises and persists.
* Similar problems have occurred around Tilerator which was caused by lock contentions - https://phabricator.wikimedia.org/T204047
==Links to relevant documentation==
Maps Runbook: Maps/RunBook
==Actionables==
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
*Create Runbook for Maps - Maps/RunBook
Category:Maps outages, 2018
Incidents/2018-11-07 TwoColConflict

== Summary ==
Hook handler code in the TwoColConflict extension resulted in users who had the TwoColConflict Beta feature enabled facing core's edit conflict resolution interface when trying to preview or diff any edit.
== Timeline ==
* 2018-11-02 14:52 (MEZ): gerrit:466937 containing the bug was merged.
* 2018-11-06 21:40 (UTC): The wmf/1.33.0-wmf.3 branch containing the bug went live on group0.
* 2018-11-07 16:59 (MEZ): A QWERTY team member started running into the issue. The issue was identified and fixed, and the backport gerrit:472175 scheduled for the next days SWAT.
* 2018-11-07 20:12 (UTC): The unpatched wmf/1.33.0-wmf.3 branch went live on group1.
* Reports started coming in (e.g. phab:T209012).
* 2018-11-08 11:49 (MEZ): The extension got disabled via gerrit:472409.
* 2018-11-08 13:29 (MEZ): The fix got backported.
* The extension got re-enabled step by step.
== Conclusions ==
The team expected the MediaWiki core's tests to report a failure as fundamental as not being able to preview/diff an edit. This did not happened, because:
# The relevant code in core is not covered by PHPUnit tests (see https://doc.wikimedia.org/cover/mediawiki-core/includes/EditPage.php.html).
# A dedicated browser test for the preview/diff functionality does not exist.
# The extensions CI was not set up to run core's browser tests.
# Even if, the browser tests must then run with the Beta feature enabled.
== Links to relevant documentation ==
* https://docs.google.com/document/d/1fEHquQx9CmGiWNCbok3OJKuP9sd41GMzm5iLSItVtYQ
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
* Check if there really are browser tests in core covering the relevant feature set, e.g. "the user wants to create a new talk page, but preview it first, uninterrupted".
** Result: Yes, 7 test cases exist in cores tests/selenium/specs/page.js.
* Run the core browser tests on a local dev machine and make sure at least one fails with the bug in place.
** Result: Yes, 1 test case fails, the "should be undoable" one. Dedicated test cases for preview/diff do not exist, though.
* Add a dedicated preview test case to core that fails with the bug in place.
** phab:T210758 (gerrit:476585)
* Change the TwoColConflict CI configuration to run core's browser tests as part of every merge process, with the Beta feature enabled.
** phab:T212214
Category:Incident documentation
Incidents/2018-11-29 ores

== Summary ==
ores.wikimedia.org was sending 500 for all score requests for 3 hours starting from 6AM UTC. It was due to config changes that was done as part of upgrading celery version of ores from three to two causing it to change its task serializer.
== Timeline ==
* November 28th 12:04 UTC: the problematic puppet change got merged
* November 29th, 6:25 UTC: Logrotate restarted uwsgi services of ORES causing it to pick up the new config and start sending 500s
* 9:51 UTC: The revert was created and deployed
== Conclusions ==
* Puppet should bind ores services to ores configs so it picks up the changes right away.
*Logrotate should restart services in a better time. Not really doable
* Contact number of WMDE staff should be avalible to SREs.
== Links to relevant documentation ==
* ORES/Deployment
* celery4 setting change
* Phabricator: Change default serializer of celery from pickle to json
== Actionables ==
* ORES services should bind to ores config files
* Logrotate should restart services when more people are around
* Contact number of some WMDE staff should be avalible to SRE/RelEng
Incidents/20180111-LegacyEncoding/minutes

== Post Mortem Meeting for train blocker - 20180111-LegacyEncoding ==
Date: February 16, 2018
Attendees: Mukunda, Greg, JR, Daniel, Zeljko, Victoria, Subbu
For reference:
Incident Report-https://wikitech.wikimedia.org/wiki/Incident_documentation/20180111-LegacyEncoding
=== Topics: ===
==== Brief Summary of Problem(s) ====
Changes to the revision class in which rolled out as part of 1.31.0-wmf.16 caused a serious issue on wikis with legacy encoding. The problem was caught on group1 so the problem was effectively was limited to Swedish Wiktionary.
Each revision has a flag indicating if it's using legacy encoding
How problem(s) was/were discovered
Editors on sv.wiktionary.org reported that all non-ascii characters were managed on edit <https://phabricator.wikimedia.org/T184749>
==== How problem(s) were introduced: ====
Refactoring of the revision class missed an edge case.
==== Learnings: ====
Unit tests and documentation are almost universally a good thing. 
The tests themselves would not have uncovered this without a configuration change in the environment.
Ideally application would not touch global variables.
Currently you need to manually change configurations in-between test runs, due to the large number of combinations, it becomes difficult to cover. 
In order to help address testing a large number of combinations, you could use https://en.wikipedia.org/wiki/All-pairs_testing
Last defense successfully caught issue, first three layers didn't 
Unit test should not depend on local config. General comment is that our test should be better defined/scoped (unti test vs system test, etc...) 
Parsing team's visual diff (maybe should call it pixel diffs now since visual diff is taken by VE team for a feature) testing
* That testing setup is based on https://github.com/wikimedia/integration-visualdiff which combines it with the testreduce bulk testing setup https://github.com/wikimedia/mediawiki-services-parsoid-testreduce
* The two instances where they are used are 
* Comparing code changes to core (ex: replace Tidy with RemexHtml) -- https://www.mediawiki.org/wiki/Parsing/Visual_Diff_Testing
* Comparing output rendering of Parsoid to PHP parser -- https://www.mediawiki.org/wiki/Parsoid/Visual_Diffs_Testing 
* Core idea: generate rendering of a wiki page via codepath 1 and another via codepath 2 and compare the 2 images. If you combine it with a bulk testing tool (can be testreduce or something else), you can run this on 10s of thousands of pages. For refactoring efforts, the diffs should be zero (or if not zero, easily explainable).
==== Action Items: ====
JR: find out if this an existing task/plan to get rid of globals.()
Create test that tests non-ascii content.
Zeljko: run browser tests for core against beta.()
JR: Follow up with Cindy about large scale refactoring plans currently being laid out.()
Incidents/20180222-wdqs

== Summary ==
Around 2018-02-22 21:00 UTC, wdqs1004 loss network access. A reboot fixed the issue, but it crashed again shortly after. The server has been depooled pending further investigations.
2 short spikes in HTTP 5xx rate can be seen on grafana, but the failover seemed to have work well and we had minimal user facing impact.
== Timeline ==
Taken from , times in UTC
* 21:07 < icinga-wm> PROBLEM - Host wdqs1004 is DOWN: PING CRITICAL - Packet loss = 100%
* 21:08 < gehel> I'm on the console on wdqs1004, it looks reasonnably well except that I cant reach network
* 21:11 < gehel> !log powercycling wdqs1004 (complete loss of network)
* 21:13 < icinga-wm> RECOVERY - Check size of conntrack table on wdqs1004 is OK: OK: nf_conntrack is 0 % full
* 21:13 < icinga-wm> RECOVERY - Host wdqs1004 is UP: PING WARNING - Packet loss = 64%, RTA = 0.22 ms
* 21:14 < icinga-wm> RECOVERY - WDQS HTTP Port on wdqs1004 is OK: HTTP OK: HTTP/1.1 200 OK - 435 bytes in 0.031 second response time
* 21:23 < XioNoX> mutante: any idea what caused that wdqs1004 issue?
* 21:34 < icinga-wm> PROBLEM - configured eth on wdqs1004 is CRITICAL: Return code of 255 is out of bounds
* 21:34 < icinga-wm> PROBLEM - WDQS HTTP Port on wdqs1004 is CRITICAL: Return code of 255 is out of bounds
* 21:34 < mutante> XioNoX: there are java errors but all that stuff seems like red herrings and normal before the incident as well.. all i can really see is it ... stopped working
* 21:36 < XioNoX> Network interface Carrier transitions: 649, there is definitively something wrong with that host
* 21:36 < icinga-wm> PROBLEM - Blazegraph process on wdqs1004 is CRITICAL: Return code of 255 is out of bounds
* 21:36 < icinga-wm> PROBLEM - puppet last run on wdqs1004 is CRITICAL: Return code of 255 is out of bounds
* 21:37 <+logmsgbot> !log dzahn@puppetmaster1001 conftool action : set/pooled=no; selector: name=wdqs1004.eqiad.wmnet
* 21:37 < icinga-wm> RECOVERY - Check whether ferm is active by checking the default input chain on wdqs1004 is OK: OK ferm input default policy is set
* 21:37 < icinga-wm> RECOVERY - DPKG on wdqs1004 is OK: All packages OK
* 21:41 < icinga-wm> PROBLEM - SSH on wdqs1004 is CRITICAL: connect to address 10.64.0.17 and port 22: Connection refused
== Conclusions ==
This looks like a hardware issue, we will investigate and follow up.
== Actionables ==
* Investigation is tracked on phab:T188045.
* Note that the LDF service had to be re-routed to another node manually. This is a known issue, tracked as phab:T161240.
Incidents/20180312-Cache-text

This is a draft, edit heavily please.
==Summary==
From 05:08 to 08:32 UTC on Monday, 12 March 2018 the number of user-facing 503 error messages increased due to piling backend connections in several CDN nodes (cp3011,cp3033 and cp3042) on esams datacenter. The root cause of this incident is unknown, but is likely related to known scalability issues in the varnish file storage backend. The issue has been successfully mitigated by restarting the varnish backend instance on the mentioned nodes.
==Timeline==
All times are UTC.
* 05:08 Incident begins. Related graphs: Varnish HTTP errors on esams, varnish backend connections (cp3031, cp3033, cp3042)
* 08:06 Restart cp3042 varnish backend instance.
* 08:32 Restart cp3031 and cp3033 varnish backend instances.
* 08:32 Incident mitigated.
==Conclusions==
TBW
==Actionables==
*Continue ongoing related investigations in phab:T181315
*Set up paging alerting on backend connections piling up (TBD)
*Move backend restarts from weekly to bi-weekly (done in gerrit:419090)
*Long term: Move to ATS as caching solution for cache backends (phab:T96853)
Incidents/20181104-s8-replication

== Summary== 
Several s8 replicas (db1101, db2045, db1092, db1116, db1104, db1099, db1109) had replication stopped because of error: Error Duplicate entry 745452474-1295751 for key PRIMARY on query. Default database: wikidatawiki. Manual intervention was needed as deleting the extra row from the database.
The table where the duplicate entry showed up was: wikidatawiki.revision_comment_temp the row can be found with: 
select * from revision_comment_temp where revcomment_rev=745452474 and revcomment_comment_id=1295751;
The next day a table level comparison was run between the s8 master (db1071) and one of the previously broken slaves (db1104) to see if there are any more extra rows exist on the replicas which aren't on the master, but we didn't find any. 
== Timeline ==
(The times are in UTC)
* 23:25: The errors are start to show up in #wikimedia-operations (and pages probably sent out - I did not found them in the icinga event logs)
* 23:38: Jynus removes the extra row on db1104 and restarts replication
* 23:39: The host db1104 reports the replication is working
* 23:42: Jynus removes the extra row on the other servers
* 23:44: The rest of the servers reports working replication
== Conclusions ==
There could be hidden data inconsistencies in the rarely-written parts of the database clusters. We have a theory that the ongoing maintenance scripts on s8 (ongoing comment migration) caused the issue.
== Links to relevant documentations ==
https://phabricator.wikimedia.org/T208695
== Actionables ==
There's an ongoing project for continuous monitoring of the possible data drifts across the databases (https://phabricator.wikimedia.org/T207253)
If that service gets implemented these kind of errors could be detected before they cause any outages.
Incidents/2019-01-10 WDQS

== Summary ==
WDQS update process started failing on wdqs1007 and shortly after on wdqs1008. No further updates were possible though read queries proceeded normally. 
Further details on Phabricator: https://phabricator.wikimedia.org/T213210
== Timeline ==
* Jan 7 22:57:10 wdqs1007 started producing errors on writes
* Jan 8 around 03:00:00 wdqs1008 started producing errors on writes
* Jan 8 the cause identified as Blazegraph allocator limit (see below)
* Jan 9 12:02:42 wdqs1007 and wdqs1008 database was reloaded from wdqs1004 and full functionality restored
== Conclusions ==
Blazegraph has a limit of 256K allocators (see below), which has been hit on these database. It is unclear why these databases hit it while others did not, but due to the way that Blazegraph allocates things (e.g. literal/term dictionary entries are never removed even if triples containing them are deleted) it may depend on the workload and previous history. If the database at the limit for allocators, it will not be able to create new memory allocations, and thus writes will fail.
We need to readjust how we use the database to preclude exhausting the number of allocators.
== Links to relevant documentation ==
* Discussion on Blazegraph issue tracker: https://github.com/blazegraph/database/issues/114
* https://github.com/blazegraph/database/wiki/FixedAllocators
* Phabricator umbrella task: https://phabricator.wikimedia.org/T213210
== Actionables ==
* We need to take some measures to ensure we're not hitting allocator limits now or in the future:
** categories namespace has non-trivial size, splitting it to separate database (Blazegraph instance) would probably free up some allocations and thus alleviate allocator problem for a while
** We are using so called "raw record" mode, which means memory for literals is allocated directly. This consumes a lot of allocators. Turning it off would allocate literals together with index pages, reducing allocator numbers. 
** We may consider inlining more URIs - such as values and references - which switch them from allocated space to in-index space. While it may not be significant savings in terms of storage space, it would make them not consume allocators either.
* We may want to add some monitoring to allocator counts (available under status?dumpJournal) to ensure that when we're nearing a dangerous place we at least get warned about it.
* wdqs1006 also has low number of available allocators, we may want to reload its database as well.
Incidents/2019-01-17 RESTBase-Parsoid

== Summary ==
On 2019-01-17, Mobile-Content-Service has been deployed to bump a content version to 1.3.10. Due to a bug in RESTBase, it caused all requests for summaries for page previews that fall out of Varnish to get re-rendered in an incorrect attempt to update the version. Since the requests were coming from the public internet, they have included Accept-Language header for languages that support variants (sr, zh etc), which was forwarded to MCS and back to RESTBase for a proper variant of HTML. RESTBase doesn't store variants of HTML, so it relies on Parsoid for transformations, thus all the requests started hitting Parsoid to transform the variants, which in turn over-loaded Parsoid and made it page. The overload of Parsoid was probably due to a bug in Parsoid regarding work assignments. Undeploying MCS quickly mitigated the issue since no attempts to incorrectly upgrade the content version were made anymore by RESTBase.
== Timeline ==
All times UTC.
* 21:10: bsitzmann deploys new mobileapps https://gerrit.wikimedia.org/r/#/c/mediawiki/services/mobileapps/deploy/+/484766/ 
* 21:10: restbase begins prodigously logging errors and warnings, which increase over time https://logstash.wikimedia.org/goto/a34dd8ace7c46e09426ed41e18b8b596
*03:16: first problem report from Icinga: <+icinga-wm> PROBLEM - Restbase edge ulsfo on text-lb.ulsfo.wikimedia.org is CRITICAL: /api/rest_v1/transform/wikitext/to/html{/title}{/revision} (Transform wikitext to html) timed out before a response was received
* 03:39: first page from Icinga: <+icinga-wm> PROBLEM - LVS HTTP IPv4 on parsoid.svc.codfw.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds
* 04:02: cdanis restarts parsoids in codfw via scap
* 04:06: scap restarts complete, parsoid still returning errors
* 04:10: Pchelolo reverts mobileapps deployment
* 04:12: Services begin to come back up
* 04:18: All services back to nominal.
== Conclusions ==
* Monitor log files after mobileapps deploys. Updated mobileapps deployment procedure page.
* RESTBase has a bug in ensure_content_type.js version checking that caused RESTBase to request all /page/summary responses to be re-rendered. (/page/summary requests Parsoid content.)
**Filed phab:T214056.
* During the outage, Parsoid has seen steady increase in language variant conversion requests that more than doubled the rate (< 20 reqs/s to >70 reqs/s) and at some point the rate spiked (>140 reqs/s) which increased Parsoid request latencies for these requests. The reason for the request spike is not yet understood, but thecurrent theory is that some big template on a wiki with language variants was edited that caused a massive amount of pages to drop out of the cache. Yet to be understood.
**Filed phab:T214099
* There's no graphs showing rates/latencies of language variant transforms
* When upgrading the content, RESTBase must not forward Accept-Language - this would cause it to store content in non-primary variant.
== Actionables ==
* RESTBase should not pass through Accept-Language on content upgrades 
*Stress test Parsoid HTTP API 
*Instrument Parsoid language variant conversion 
*