nyuuzyou/wikis huggingface dataset:
Incidents/2020-06-19 Logstash

== Summary ==
Logstash indexes were temporarily stale with newer messages from Kafka not yet available. This was caused by TODO.
Impact: Monitoring alerts that are based on Logstash missed potential errors. Developers were briefly unable to see the latest messages in Kibana. They may've also been reduced availability in querying older messages from Logstash.
== Timeline ==
Write a step by step outline of what happened to cause the incident, and how it was remedied. Include the lead-up to the incident, and any epilogue.
Consider including a graphs of the error rate or other surrogate.
Link to a specific offset in SAL using the SAL tool: https://tools.wmflabs.org/sal/, For example: https://tools.wmflabs.org/sal/production?q=synchronized&d=2012-01-01
All times in UTC.
* 00:00 (TODO) OUTAGE BEGINS
* 00:04 (Something something)
* 00:06 (Voila) OUTAGE ENDS
* 00:15 (post-outage cleanup finished)
TODO: Clearly indicate when the user-visible outage began and ended.
== Detection ==
Write how the issue was first detected. Was automated monitoring first to detect it? Or a human reporting an error?
Copy the relevant alerts that fired in this section.
Did the appropriate alert(s) fire? Was the alert volume manageable? Did they point to the problem with as much accuracy as possible?
TODO: If human only, an actionable should probably be to "add alerting".
== Conclusions ==
What weaknesses did we learn about and how can we address them?
=== What went well? ===
* (Use bullet points) for example: automated monitoring detected the incident, outage was root-caused quickly, etc
=== What went poorly? ===
* (Use bullet points) for example: documentation on the affected service was unhelpful, communication difficulties, etc
=== Where did we get lucky? ===
* (Use bullet points) for example: user's error report was exceptionally detailed, incident occurred when the most people were online to assist, etc
=== How many people were involved in the remediation? ===
* (Use bullet points) for example: 2 SREs and 1 software engineer troubleshooting the issue plus 1 incident commander
== Links to relevant documentation ==
Add links to information that someone responding to this alert should have (runbook, plus supporting docs). If that documentation does not exist, add an action item to create it.
== Actionables ==
Create a list of action items that will help prevent this from happening again as much as possible. Link to or create a Phabricator task for every step.
* To do #1 (TODO: Create task)
* Create spiceracks cookbooks for restarting Logstash's Elastic cluster (we have ones for Cirrus' Elastic cluster) – https://phabricator.wikimedia.org/T255864
TODO: Add the #Wikimedia-Incident-Prevention Phabricator tag to these tasks.
Incidents/2020-06-25 caching-sessions

== Summary ==
See also <https://lists.wikimedia.org/pipermail/wikitech-ambassadors/2020-June/002316.html>
Documentation:
* User reports: T257522, T256395, T255156
* text-backend.inc.vcl.erb#72
== Timeline ==
Restricted document. 
== Actionables ==
* https://phabricator.wikimedia.org/T260098 Understand beresp.ttl=0 issues in Varnish
Incidents/2020-07-03 Lilypond down

== Summary ==
https://phabricator.wikimedia.org/T257062 Lilypond seemingly bypassing security restrictions.
== Actionables ==
* Audit all mismatched/unrecognised wmf-config settings. – https://phabricator.wikimedia.org/T257079
* Lint against unrecognised wmf-config settings in CI. – https://phabricator.wikimedia.org/T248866
Incidents/2020-07-05 eqsin-router-crash

== Summary ==
On Sunday 5th at 11:22UTC, the primary hard drive of cr3-eqsin (one of the two Singapore POP routers) crashed.
This caused the router to reboot into its second disk, containing only a factory default configuration. Everything failed over cleanly to the redundant router.
Impact: We lost at max ~15000 requests/s in a 7min window (see screenshot, and graph).
== Timeline ==
All times in UTC.
* 11:22 PROBLEM - Host cr3-eqsin is DOWN: PING CRITICAL - Packet loss = 100% (paging) OUTAGE BEGINS
* 11:25 SREs reports of connectivity issues to eqsin (too brief to trigger alerting)
* 11:27 Routing is done converging, no more reports of connectivity issue OUTAGE ENDS
* 11:35 DNS patch ready to depool eqsin (just in case, unused) - https://gerrit.wikimedia.org/r/c/operations/dns/+/609571/
Monday 06
* ~07:40 Router is brought back up on its backup disk Redundancy restored
== Detection ==
* Was automated monitoring first to detect it? Yes
* Did the appropriate alert(s) fire? Yes
* PROBLEM - Host cr3-eqsin is DOWN: PING CRITICAL - Packet loss = 100% (paging alert)
* Was the alert volume manageable? Yes, only relevant alerts fired
* Did they point to the problem with as much accuracy as possible? Yes, the router went down, and only the router down paging alert triggered
== Conclusions ==
* This outage showed that our hardware redundancy and failover are solid
* Juniper recently introduced a new feature: vmhost snapshot that would have prevented the lack of redundancy (but not the crash itself)
=== What went well? ===
* Everything failed over as expected to the redundant router
=== What went poorly? ===
* N/A
=== Where did we get lucky? ===
* N/A
=== How many people were involved in the remediation? ===
* 2 SREs investigating the issue, multiple SREs reported present to the page
== Links to relevant documentation ==
https://wikitech.wikimedia.org/wiki/Network_monitoring#host_(ipv6)_down
== Actionables ==
* cr3-eqsin disk 1 failure - https://phabricator.wikimedia.org/T257154
* Investigate Junos vmhost snapshot - https://phabricator.wikimedia.org/T257153
Incidents/2020-08-14 isp-unreachable

== Summary ==
Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites.
Caused by a misconfiguration in Jio's own RPKI records, which can only be fixed on their end. Their network engineers are aware and working on the issue.
As temporary workaround, Wikimedia changed its network advertisements to accept invalid RPKI to let Jio costumers reach our sites again.
== Actionables ==
* https://phabricator.wikimedia.org/T260449
* https://phabricator.wikimedia.org/T260452
Incidents/2020-09-08 Rack D hosts outage

== Summary ==
A power cable issue in asw2-d3-eqiad during a scheduled PDU upgrade, caused hosts in racks D1, 3, 4 to be unavailable (13:34 UTC). It rebooted into a different firmware image than the rest of the switches and had to be upgraded; once this happened, all hosts were again available (15:02 UTC).
User impact: Wikifeeds was not working properly; Horizon unavailability meant that creation and updating of WMCS images was broken; dns1002 was flapping, causing sporadic dns lookup failures for eqiad hosts; logins failed to stat1005, stat1006.
Other impact: kafka-jumbo1006 being unavailable meant that some data was lost for webrequest and EventLogging topics; some bacula full backups had errors and failed to run and the director had to be restarted; ferm was broken on about 30 hosts that were running puppet when DNS connectivity was lost/resolution failed and had to be manually restarted
== Actionables ==
* Create icinga hostgroup per rack row / Can we make row/rack visible in icinga tactical overview of hosts down somehow? (Akosiaris)
* Investigate why asw-d3-eqiad rebooted into a different boot partition with different JunOS version, causing it to be “inactive” in the VCF - and prevent this from happening again - https://phabricator.wikimedia.org/T262290
** Run “show system snapshot media internal ” and “request system snapshot slice alternate” on all EX switches
* Wikifeeds (and as a consequence, its endpoint for restbase) basically went down until the network issues in eqiad were resolved. Figure out what the hidden dependency is there. (Akosiaris)
* Investigate why routing around a missing switch member D3 did not work, hosts in D1, D2, D4 (at least) were also impacted - https://phabricator.wikimedia.org/T256112
* Check Anycast configuration as it seemed the flapping connections caused a few issues with dns1002 - https://phabricator.wikimedia.org/T262372
* Rsyslog deliveries to cengit trallog must fail independently - https://phabricator.wikimedia.org/T226703
Incidents/2020-09-09 mobileapps config change

== Summary ==
While reconfiguring all services to use our service proxy middleware to make remote procedure calls, a faulty configuration was deployed by yours truly for mobileapps at 08:40. This caused mobileapps to create mobile-html content with broken css and js links for pages regenerated during the day.
The issue was reported at 16:40 and the issue was quickly reverted. Then we needed a few hours to actually clear all the caching layers (RESTBase, edge caches). All pages affected were purged by 20:20.
== Actionables ==
* https://phabricator.wikimedia.org/T268484 Tracking task.
* https://phabricator.wikimedia.org/T262437 The bug caused by this incident (Page content service is deployed with localhost links).
* https://phabricator.wikimedia.org/T264340 Create test for api/rest_v1/page/mobile-html
* The biggest actionable is of course to always wait for validation from service owners before merging a patch - and the whole outage would've been avoided if that was done. Anything else listed here is purely a second-order actionable.
* While this deployment was the result of bad judgement, SRE need to be able to deploy a configuration change with confidence. The fact that the mobileapps spec tests all passed in staging lulled SRE into a false sense of security. The OpenAPI spec should be extended to include a test for the aforementioned URLs. (TODO: create task)
* We need staging to become a functional environment where we can test more than just a swagger spec test. Maybe linking it to restbase-dev, and making it possible to compare results of urls with production would help (TODO: create task)
Incidents/2020-10-06 cloud-vps

See also: https://lists.wikimedia.org/pipermail/cloud-announce/2020-October/000324.html
See also: https://phabricator.wikimedia.org/T264694
==Summary==
Sensitive credentials (authentication tokens) were leaked via publicly accessible, memcached servers, on a default port & without authentication.
Tokens can be used to escalate privileges and (at least) “bump quotas, create and destroy” arbitrary VMs in Cloud VPS/Toolforge. While the service runs in the “production” realm, no escalation paths to other production services and hosts are known so far.
These tokens are for access to the Keystone API, which is firewalled to the Cloud VPS and production networks, so they could not be used directly from the internet easily without changing signed cookies in Django/Horizon web UI somehow (which should require the secret key). Therefore, this exposure would theoretically benefit a malicious cloud user to the extent that they could manipulate the Keystone API from inside cloud according to the permissions of the particular captured token.
Access was blocked in eqiad at 02:21 UTC and in the development environment in codfw at 08:43 UTC.
Keystone auth tokens were manually rotated (the generally have a lifetime of 24 hours)
Investigation of potential privilege escalation within Cloud VPS (via leaked Keystone tokens) ongoing, additional mail to be sent to cloud users (see below).
==Actionables==
* Better understand and refine port-scanning tools in order to detect future vulnerabilities like this one. The daily network scan is in the process of being fixed to scan all ports (instead of a selection of 2000) and when that has happened we need to do a one off investigation of the findings to establish a base point so that all further runs can be studied as a diff against the base check.
* Defense-in-depth measures for cloudcontrol services. Per OpenStack recommendations, “[f]or production deployments, we recommend enabling a combination of firewalling, authentication, and encryption to secure it.”. The first of the three failed here, but it was all three that were missing and allowed this vulnerability.
* More fine-grained access control for the ferm ranges in the profile::memcached::instance class to allow to restrict access for cloudcontrol* further.
* Extend the daily network port scan to cover 64k ports instead of 2000 well-known, this requires some changes to the packet filters running on on our servers https://phabricator.wikimedia.org/T264888
* Meet with security to discuss threat model for cloud services
Incidents/2020-10-30 fr.wikipedia.org-slow-main-page

== Summary ==
The main page of the French Wikipedia was loading much slower than usual for some time on October 30th 2020. The page did load but took several seconds and up to 20 seconds to finish.
The incident lasted roughly 7 hours and the group of affected users were all logged in users using the main page of French Wikipedia during that time. Logged out users / general readers were served cached results and did not notice the incident. We can roughly estimate the number of affected users to about ~ 625, based on about 2500 unique editors per day.
The ones who did notice started reporting on the French Wikipedia Bistro page and a ticket on phabricator was created.
The FeaturedFeeds extension adds Atom feed <link> tags to the head of main pages, but internally it generates the entire feed, just to add the tags. Normally it reads from the cache so this wasn't noticed until the feed became too big to fit in memcached (ITEM_TOO_BIG error) and was being generated/parsed on every main page load, including action=history or action=edit.
The issue was resolved after the wmgFeaturedFeedsOverrides limit used by the FeaturedFeeds extensionwas reduced from 60 items per feed to the default of 10 items per feed that is used by other wikis.
This did not change rendering of the main page; the only lasting effect will be that users of fr.planet.wikimedia.org, the RSS feed aggregator for Wikimedia-related feeds, will see only 10 instead of 60 items from the fr.wikipedia featured feed.
The feed config and special case for frwiki had been added as part of back in 2017.
An additional mini report can be found in T266865#6592372.
== Timeline ==
* 12:22 UTC https://phabricator.wikimedia.org/T266865 is opened
* ~ 17:40 UTC SRE starts investigating
* 17:51 SRE confirms issue on ticket and notes it is not limited to esams but affects all data centers but only the French main page and not other pages
* 18:22 UTC SRE finds “Memcached error for key "WANCache:frwiki:featured-feeds:1:fr|#|v" on server "127.0.0.1:11213": ITEM TOO BIG”
* 18:27 UTC SRE reaches out to perf-team
* 18:35 UTC SRE finds that MW config has wmgFeaturedFeedsOverrides with a limit set to “60” for frwiki while other wikis use defaults
* 18:36 UTC volunteer suggests disabling featuredfeeds extension and testing on mwdebug1001 and does so. Confirms load time is much faster without it.
* 18:39 UTC SRE uploads config change to remove the custom limit of 60 items per feed.
* 18:50 Deployment of said patch
* 18:56 French WP user confirms issue looks resolved. (https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Le_Bistro#Temps_de_chargement_de_la_page_d'accueil)
This incident also happened again on 2020-12-21.
== Actionables ==
* reduce number of feed items from 60 to default (done)
* : FeaturedFeeds should not load all feed content just to output the feed URLs on the main page (done)
Incidents/2021-02-01 swift-codfw

== Summary ==
Today at 11:41 the icinga check for `ms-fe.svc.codfw.wmnet` timed out and thus paged (and recovered three minutes later):
11:41 -icinga-wm:#wikimedia-operations- PROBLEM - LVS swift-https codfw port 443/tcp - Swift/Ceph media 
 storage IPv4 #page on ms-fe.svc.codfw.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 
 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems
...
11:44 -icinga-wm:#wikimedia-operations- RECOVERY - LVS swift-https codfw port 443/tcp - Swift/Ceph media 
 storage IPv4 #page on ms-fe.svc.codfw.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 396 bytes in 
 0.140 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems
Users in codfw/ulsfo/eqsin have experienced ~15min of higher latency (possibly timeouts) for hit-local and miss requests (10-25% of the site's requests, depending on the site).
Specifically hitting /monitoring/backend timed out, this in turn meant that some of the backend server(s) where the monitoring container lives were slow/unhealthy.
Case in point ms-be2033.codfw.wmnet was reported as slow from /var/log/swift/server.log on e.g. ms-fe2006.codfw.wmnet:
Feb 1 11:44:29 ms-fe2006 proxy-server: ERROR with Object server 10.192.16.15:6000/sdk1 re: Trying to GET /v1/AUTH_mw/monitoring/backend: Timeout (10.0s) (txn: txe96767fb630b4828af04a-006017e993) (client_ip: 208.80.154.88)
The slowness was induced by an earlier swift rebalance () and the way we do rebalances at the moment means that such operations are generally noisy/impactful to the cluster (e.g. , ). Swift has been depooled internally from its discovery record (essentially anticipating ).
== Actionables ==
* Change /monitoring/backend to /monitoring/frontend (i.e. check the frontend itself) for icinga service check and pybal's proxyfetch 
* Consider depooling swift's discovery records during rebalances
Incidents/2021-02-26 sudo

== Summary ==
A patch was merged and deployed to all hosts containing a syntax error on the /etc/sudoers file. This meant sudo did not work for the period of time indicated below, affecting mostly nagios execution (alerting) and creating root mail spam. As a consequence, also mail delivery got overloaded/delayed.
== Timeline ==
* 08:50 666899 is merged, containing a syntax error in /etc/sudoers
* 08:51 People warn on IRC unable to sudo on db1107 due to a parse error (>>> /etc/sudoers: syntax error near line 6 <<<), and other hosts
* 08:52 100s of emails start to arrive to root@ with *** SECURITY information for <hostname>*** (sudo failures)
* 08:55 <jbond42> !log disabled puppet pending rollback of https://gerrit.wikimedia.org/r/666899
* 08:59 klausman merges 667110, containing a fix, and runs puppet-merge soon after.
* 09:00 Incident opened. jynus becomes IC.
* 09:06 Puppet reenabled
* 09:12 Start reenabling puppet fleetwide
* 09:23 Puppet run at 10%
* 09:37 Puppet run at 30%
* 09:50 Puppet run at 50%
* 10:17 Puppet run at 80%
* ~10:20ish UNKNOWN nagios alerts gone
* 10:33 puppet run finished
* 11:40: jbond took over ic
* 11:45: mx2001 queue has remain static at 4344 for 20 minutes
* 11:45: mx1001 queue reducing at between 0-3 msgs/sec
* 12:55: run `exiqgrep -i -o 7200 -y 10800 -f 'root@wikimedia.org' | xargs exim -Mrm` on mx servers
* 12:55: queue down to ~ 2000 (891 frozen) msgs on mx1001 and 500 (434 frozen) on mx2001 
* 13:02: Still receiving 450-4.2.1 from gmail for a number of recipients
* 13:30: reports of flood emails slowing down 
* 13:43: message queue excluding frozen messages on mx2001 is 0 (mx1001 ~ 800)
* 14:00 ran the following to push through the last few messages: `for i in $(sudo exiqgrep -f nagios@lists1001.wikimedia.org -i) ; do sudo exim -M ${i} ; sleep 1 ; done `
* 14:05: unfrozen queue is still at 784 however the queue looks normal 
* 14:05: Incident resolved
== Cleanup GMail ==
You can use this filter to find them all:
 from:(nagios@) SECURITY after:2021-02-25 
Look also in your spam folder.
== Remediation considerations ==
As Puppet runs as root and is triggered by a cron entry, an issue with sudo does not affect the capability of running Puppet and hence to fix the problem.
In addition as Cumin does SSH as root in all hosts, it's also possible to ssh into the cumin host, become root using the password in pwstore and perform an emergency fix via Cumin on all hosts. If for some reason it would not be possible to SSH directly into the Cumin host, it's also possible to connect to it via the management console and login from there directly as root.
== Actionables ==
* Add `validate_cmd` to sudoers file
** https://gerrit.wikimedia.org/r/667119
** https://gerrit.wikimedia.org/r/667120 
* Understand safe batch numbers for fleetwide puppet runs
* Prevent automatic email to get marked as spam by GMail
* Add exim queue metrics to grafana https://phabricator.wikimedia.org/T275867
* Investigate why the exim queue has many frozen messages for wiki at wikimedia.org and otrs at ticket.wikimedia.org
Incidents/2021-03-14 MediaWiki API

==Summary==
On March 14 2021 the MediaWiki API were overloaded and ran out of php-fpm processes. This caused an API outage on all API servers from 17:00 to 17:26 UTC. The root cause of the outage were queries against commons that caused database s4 on server db1144 to be overloaded. Db1144 also serves queries to contributions, recentchanges, watchlist and other MediaWiki features. Task: T277417
https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?viewPanel=46&orgId=1&from=1615734448378&to=1615746774986&var-datasource=eqiad%20prometheus%2Fops&var-cluster=api_appserver&var-method=GET&var-code=200
https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=37&orgId=1&var-server=db1144&var-port=13314&from=1615738747603&to=1615745074190
The queries against commons were analyzed for inefficiencies but seem to be well written and optimized SQL. See task: T277416
==Actionables==
*Improve traceability of commons queries: T193050 (filed in 2018)
Incidents/2021-03-30 Jobqueue overload

== Summary ==
An upload of 65 video 4k files via the server-side upload process caused high CPU/socket timeout errors on jobrunners (all jobrunner hosts are also videoscalers). This caused an increase in job backlog and unavailability on several mw-related servers (job queue runners, etc.). It seems that a combination of the files being 4k (and thus requiring many different downscales), long (averaging an hour in length), combined with the fact that the videos were uploads from a local server (mwmaint) with a fast connection to the rest of our infrastructure resulted in too much load being placed on the jobqueue infrastructure.
Halting the uploads and temporarily splitting the jobqueue into videoscalers and other jobrunners allowed the infrastructure to catch up. 
== Actionables ==
* Document that users should use --sleep to pause between files when running importImages.php (done)
* Rate limit the process to upload large files 
* Add rate limiting to the jobqueue videoscalers 
* Add alerting for Memcached timeout errors 
* Update Runboook wikis for the application and LVS servers 
* Have some dedicated jobrunners that aren't active videoscalers