nyuuzyou/wikis huggingface dataset:
Incidents/2022-06-21 asw-a2-codfw accidental power cycle

==Summary==
During regular maintenance, there was a (scheduled) loss of power redundancy on the codfw-A1 server rack around 14:32:00 UTC.
While the servers in this rack did not lose power (given a redundant power supply), they did fully lose network connectivity and thus effectively went down. This happened because the second power cable for the ASW network switch was not plugged all the way in, resulting in an unscheduled full loss of the switch for that rack, and hence the rack's network connectivity.
Happily, higher-level service redundancy worked as expected:
* regarding LVS, lvs2010 automatically took over from lvs2007, for CDN traffic to Codfw. There was a very temporary increase on response latency for on-the-fly Codfw requests until traffic stabilized.
* ns1 DNS server was automatically moved to Eqiad, should not have any user impact.
* Most A2 servers alerted about loss of power redundancy, but having 2 power supplies they didn't go down.
* App servers could have been affected more, latency-wise while they were automatically depooled, but they were not serving production traffic at this time as Eqiad is the primary DC.
After the secondary power cord was properly connected, connectivity recovered with no issues. Maintenance finished at 15:01.
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	yes	Not paging
People 	Were pages routed to the correct sub-team(s)? 	no 	Not paging
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	Not paging
Process 	Was the incident status section actively updated during the incident? 	no 	No working doc
Process 	Was the public status page updated? 	no 	not user facing
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	no 	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	yes	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	no 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	8
Incidents/2022-06-30 asw-a4-codfw accidental power cycle

==Summary==
Network connectivity for the A4 codfw server rack went down due to full power loss of its switch. This caused lots of alert spam, but otherwise it had very little to no impact on users due to services not being pooled on codfw or redundancy working as intended.
This was very similar incident to Incidents/2022-06-21 asw-a2-codfw accidental power cycle (bump wrong cable on switch side again). See that page for more details.
After the secondary power cord was properly connected, connectivity recovered with no known issues. Power maintenance on that rack finished at 15:50.
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	yes	Not paging
People 	Were pages routed to the correct sub-team(s)? 	no 	Not paging
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	Not paging
Process 	Was the incident status section actively updated during the incident? 	no 	No working doc
Process 	Was the public status page updated? 	no 	not user facing
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	no 	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	no 	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	no 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incidents/2022-07-03 shellbox request spike

==Summary==
An increase in Score requests (musical note rendering) from Parsoid overwhelmed the Shellbox service. This was mitigated by allocating more k8s pods to Shellbox. The overload took place from 11:17 to 11:33.
From the Grafana dashboard, we see that a majority of requests took over 5,000ms (5s) instead of less the usual 10ms (0.01s), and for two minutes 11:25-11:27 requests actually failed with HTTP 503 instead.
From the Logstash dashboard, we measure approximately 35,000 failed requests during this time. Of which 99.9% were from Parsoid, for de.wikipedia.org requests that render musical notes through the Score extension. 26K received HTTP 503, and 9K received HTTP 504. The remaining 0.1% were edits or pageviews calling Shellbox for syntax highlighting. 
See 2022-07-11 Shellbox and parsoid saturation for re-occurrence and follow up action items.
Documentation:
*Shellbox requests dashboard
==Actionables==
*T310557: Improving Shellbox resource management
*T312319: Reduce Lilypond shellouts from VisualEditor
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	no 	
People 	Were pages routed to the correct sub-team(s)? 	no 	
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	
Process 	Was the incident status section actively updated during the incident? 	yes	
Process 	Was the public status page updated? 	no 	
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	no 	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	no 	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	no 	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	7
Incidents/2022-07-10 thumbor

==Summary==
Due to a uptream bug introduced in a firejail update, Thumbor constantly restarting itself. This lead to increased error rates, and also increased delays from the HAProxy in front of it.
==Detection==
Write how the issue was first detected. Was automated monitoring first to detect it? Or a human reporting an error?
Copy the relevant alerts that fired in this section.
Did the appropriate alert(s) fire? Was the alert volume manageable? Did they point to the problem with as much accuracy as possible?
TODO: If human only, an actionable should probably be to "add alerting".
==Links to relevant documentation==
*Grafana dashboard: Thumbor 
==Actionables==
*…
Create a list of action items that will help prevent this from happening again as much as possible. Link to or create a Phabricator task for every step.
Add the #Sustainability (Incident Followup) and the #SRE-OnFIRE (Pending Review & Scorecard) Phabricator tag to these tasks.
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers?  Answer “no” if engineers were paged after business hours.		
Process 	Was the incident status section actively updated during the incident? 		
Process 	Was the public status page updated? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 		
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incidents/2022-07-11 FrontendUnavailable cache text

==Summary==
There was an increase in requests to the API cluster that resulted in reduction of availabe PHP workers and a database host (db1132) running out of available connections. This database host runs MariaDB 10.6 which is known to be sensitive to high load. This resulted in an increase in latency and errors returned to clients. The spike auto-recovered and it's not clear what the exact root cause was. 
Documentation:
*Appservers RED dashboard
==Actionables==
*https://phabricator.wikimedia.org/T311106 - investigate mariadb 10.6 performance regression during spikes/high loads.
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	no 	
People 	Were pages routed to the correct sub-team(s)? 	no 	
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	
Process 	Was the incident status section actively updated during the incident? 	no 	
Process 	Was the public status page updated? 	no 	
Process 	Is there a phabricator task for the incident? 	no 	
Process 	Are the documented action items assigned? 	yes	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	no 	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	yes	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	7
Incidents/2022-07-12 codfw A5 powercycle

==Summary==
During the scheduled maintenance to upgrade the PDUs in rack A5, CyrusOne flipped the incorrect breaker on the breaker panel, prior to pulling the PDU's power cord out from its circuit. This resulted in all servers in rack A5 losing power to both its primary and secondary power feeds. The affected hardware in rack A5 booted back up, once CyrusOne realized the mistake and flipped the breaker back on.
* 15:45 <+icinga-wm> PROBLEM - Host graphite2003 #page is DOWN: PING CRITICAL - Packet loss = 100%
* 15:45 <+icinga-wm> PROBLEM - Host maps2005 is DOWN: PING CRITICAL - Packet loss = 100%
* 15:55 <+icinga-wm> PROBLEM - MariaDB read only s8 #page on db2079 is CRITICAL: Could not connect to localhost:3306
* 15:56 <+icinga-wm> PROBLEM - MariaDB read only m1 #page on db2132 is CRITICAL: Could not connect to localhost:3306
* ..
* 16:00 <+icinga-wm> RECOVERY - MariaDB read only s8 #page on db2079 is OK
== Actionable ==
* As a remediation item, the remaining PDU maintenances at codfw will no longer be hot-swapped with live equipment. After 3 of 5 inadvertent incidents, Dc-Ops team deemed it would be safer to coordinate with the SREs for hard downtime for all the affected servers in each rack. This would result in the graceful shutdown of affected servers, and allow for a shorter duration to complete each PDU upgrade with a temporary hired contractor.
== See also ==
* Incidents/2022-06-30 asw-a4-codfw accidental power cycle
* Incidents/2022-06-21 asw-a2-codfw accidental power cycle
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	no 	
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	no 	
People 	Were pages routed to the correct sub-team(s)? 	no 	
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	
Process 	Was the incident status section actively updated during the incident? 	no 	
Process 	Was the public status page updated? 	no 	
Process 	Is there a phabricator task for the incident? 	yes	
Process 	Are the documented action items assigned? 	no 	
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	no 	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	yes	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	7
Incidents/2022-08-10 confd all hosts

==Summary==
A Puppet patch (change 817307) was merged which would accidentally install confd on a significant number of production hosts. The Puppet provisioning for these confd installations failed half-way due to having no corresponding Icinga checks defined. This in turn fired an Icinga alert:
 10:44 <icinga-wm> PROBLEM - Widespread puppet agent failures on alert1001 is CRITICAL: 0.1057 ge 0.01 
Engineers started work on reversing it, by cleaning up the inadvertent installs of confd via Cumin. Security posture was not compromised and there was no external user impact.
Documentation:
* puppet failures - https://logstash.wikimedia.org/goto/a5b60af08e257d90a469a78d12056ec2
*Confd
* Cumin
==Actionables==
* Git defaults to shows the author's date, not the commit date. Consider adding the following aliases to your git config as fix:
** lola = log --graph --pretty=format:\"%C(auto)%h%d%Creset %C(cyan)(%cr)%Creset %C(green)%cn <%ce>%Creset %s\" --all
** grephist = log --graph --pretty=format:\"%C(auto)%h%d%Creset %C(cyan)(%cr)%Creset %C(green)%cn <%ce>%Creset %s\" --all -S
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 	yes	Overlap of 3
People 	Were the people who responded prepared enough to respond effectively 	yes	
People 	Were fewer than five people paged? 	yes	
People 	Were pages routed to the correct sub-team(s)? 	yes	n/a
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.	yes	n/a
Process 	Was the incident status section actively updated during the incident? 	yes	
Process 	Was the public status page updated? 	no 	
Process 	Is there a phabricator task for the incident? 	no 	
Process 	Are the documented action items assigned? 	yes	action items limited to improved local bash aliases
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 	yes	
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.	yes	
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 	yes	
Tooling 	Did existing monitoring notify the initial responders? 	yes	
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 	yes	
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 	no 	
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above) 	12
Incidents/2023-07-19 enrich job outage

== Summary ==
== Description ==
An enriched message exceeding Kafka's max.request.size caused the application's Kafka producer to crash. This in turn resulted in a Flink Taskmanager shutdown. 
Full stacktrace is available at:
https://logstash.wikimedia.org/app/discover#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-k8s-1-1.11.0-6-2023.29?id=EeKfdIkB6U_kV85ADO_D
HA tried to restart the application but the restart strategy, failed more than restart-strategy.fixed-delay.attempts times.
A manual application restart was required. The offending message has seemingly been discarded.
A fix is proposed in 
==Timeline==
* 20:10:38 UTC: GModena opens thread to ACK the outage in #data-platform-engineering
* 20:20:24 UTC: TChin identifies the root cause of the issue (
* 20:24:00 UTC: GModena manually restarts the application
* 20:44:00 UTC: GModena silences Kafka Consumer lag alerts while the application catches up with queues messages.
==Detection==
GModena and TChin reacted to alerts triggered by degrading SLIs.
==Conclusions ==
* Application is running.
* We know what needs to be fixed.
===What went well?===
===What went poorly?===
===Where did we get lucky?===
*
==Links to relevant documentation==
==Actionables==
* increase max message size allowed by Kafka
* Filter out messages larger than the max allowed size; 
==Scorecard==
Increasing account creation threshold

There are cases when the Wikimedia Foundation or others will host events that have a tendency of creating a large amount of new account requests during a short window of time from a single IP or a few IPs. The IP can be effectively white listed to be excluded from rate limiting. This will also raise edit rate limits to the value used for autoconfirmed users (by default 90 edits/minute) instead of the limit for non-autoconfirmed users (8 edits/minute for all newbies coming from the same IP address).
== throttle.php ==
This can easily be done if you add an $wmgThrottlingExceptions array in 
$wmgThrottlingExceptions[] = [ // T185930
	'from' => '2018-03-10T11:00:00 -5:00',
	'to' => '2018-03-10T16:00:00 -5:00',
	'range' => '38.125.10.42',
	'dbname' => [ 'enwiki' ],
	'value' => 30 // 20 expected
];
IP and dbname values could be scalar values or arrays.
After this a backport deployer must sync the throttle with
sync-file wmf-config/throttle.php 
After this, if the event is in less than 72 hours, you should delete the memcached key with mwscript resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 1.2.3.4
The processing logic for throttle.php is in throttle-analyze.php.
If the IP is not known then you can also set $wgAccountCreationThrottle to a value higher then 0.
Category:How-To
Install2003

install2003 is an install server which consists of DHCP, TFTP, webproxy (Squid).
It replaced install2002. The APT repo was separated to another VM, apt2001.
As opposed to the previous install servers it has a private IP.
=== Services ===
* TFTP install-server
* DHCP server
* HTTP proxy for eqiad hosts
== See also ==
* install1003, install2002, carbon
Install server

An install server consists of DHCP, TFTP, webproxy (Squid) and apt.wikimedia.org (reprepro) servers.
The current servers are install1003 in eqiad and install2003 in codfw for DHCP/TFTP/webproxy.
APT repositories are currently on apt1001 and apt2001. 
carbon, install1002 and install2002 are decommissioned install servers used in the past.
== How to create an install server ==
Apply the puppet class role(installserver) to a node.
== Monitoring ==
There is Monitoring/atftpd to check if the TFTP process is running.
Category:SRE Infrastructure Foundations
Integration.wikimedia.org

integration.wikimedia.org is the Continuous integration dashboard for Wikimedia.
== Service ==
Currently hosted on contint.
The site itself is a mostly-static PHP microsite with a Zuul status page that provides real-time insight and filtering of which Gerrit changesets are being tested, and links to the relevant builds of Jenkins jobs.
The domain is also a proxy for:
* Jenkins, the Java application at at https://integration.wikimedia.org/ci/ that runs all the jobs and stores the build output.
* Zuul, a Python web frontend serving the Zuul status API at https://integration.wikimedia.org/ci/
Puppet provisioning: https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modules/contint/files/apache/integration.wikimedia.org.conf
Source code: https://gerrit.wikimedia.org/g/integration/docroot
== Deployment ==
Deploy updates to the microsite (integration/docroot) with Scap from the Deployment server.
== See also ==
* Jenkins
* Continuous Integration on mediawiki.org
* Continuous Integration/Jenkins on mediawiki.org
== External link ==
* https://integration.wikimedia.org
Category:Services