nyuuzyou/wikis huggingface dataset:
Incident documentation meeting/QR201407/group1/notes

../../../20140318-EventLogging/
* migrated to m2 shard, shouldn't have too many load issues in future
* analytics is responsible for responding to alerts is from analytics
* ops is responsible for generic looking database alerts
* EL can be down or lagging for up to 48 hours (weekends) - "Tier 2" support
** https://www.mediawiki.org/wiki/EventLogging/OperationalSupport
../../../20140328-DB-Queries/
* would have been good to have Ariel on the call
* greg to follow up on explicit next steps with Bryan and Reedy
* Add to next group's list
../../../20140509-EventLogging/
* all green :)
* seems all bases are covered here, any disagreement? :)
../../../20140526-m1/
* blog work, loop back with RobH re future of that box? HA? etc?
* how far away to get rid of blog?
../../../20140607-Elasticsearch/
* still need to create reproducible steps for this to be reported upstream
* still need to manually remove a sick node (on purpose)
../../../20140613-Videoscalers/
* Friday! :)
* need a bug for "Add monitoring for individual job types on single machines. "
* Should deploy https://gerrit.wikimedia.org/r/#/c/144612/ before hhvm goes to jobrunners
../../../20140619-parsercache/
* MediaWiki failed to stop trying to use the bogged down machine
** Greg: need to get this diagnosed and tracked
** HHVM's impact here?
* proposal 4 related to Rashomon?
../../../20140622-es1006/
* epicly awesome fix: http://ur1.ca/hpjpa ;)
../../../20140622-imagescaler/
* need the Swift bandwidth
* metrics: https://bugzilla.wikimedia.org/show_bug.cgi?id=67116 Please help :)
../../../20140625-CirrusSearch/
* Still have the feature request for scap here
** https://bugzilla.wikimedia.org/show_bug.cgi?id=67091
Incident documentation meeting/QR201407/group2

= Quarterly Review of post-mortems - 2014-07 =
== Questions we want to be able to answer ==
* Have all of the issues that came out of the post-mortem been addressed? If not, why not?
* Are we satisfied with the current state of that part of the infra? Are there further actions to take (upon further reflection)?
* anything else?
== Agenda ==
* Go through the post-mortems and their respective action items and make sure they have been followed up appropriately.
** If you have details that are relevant to the post-mortem in BZ/etc, please link from the post-mortem.
* Discuss if there is anything else that we learned from the situation and follow up to better inform future decisions.
* Notes written up by all, collaboratively, so that others in the organization will learn from these as well.
== Notes ==
HERE!
== The post mortems ==
=== ../../20140328-DB-Queries/ ===
Bryan, Reedy
=== ../../20140403-Deploy/ ===
Bryan, Reedy
=== ../../20140503-Thumbnails/ ===
Antoine, Faidon
=== ../../20140517-bits/ ===
Timo
=== ../../20140529-appservers/ ===
Ori
=== ../../20140608-Kafka/ ===
Andrew O
=== ../../20140612-Math/ ===
Greg
=== ../../20140618-Wikitech/ ===
Andrew B, Marc
Incident documentation meeting/QR201407/group2/notes

See also:
* https://wikitech.wikimedia.org/wiki/Incident_documentation/QR201407
* https://wikitech.wikimedia.org/wiki/Incident_documentation/QR201407/group2
=== ../../../20140328-DB-Queries/ ===
Bryan, Reedy
* PrivateSettings.php should be in a repo so we can be sure what's changed.
** (Not Gerrit obvs, maybe living same place as private ops repo or similar?) (reedy)
*** Local on tin in a sub directory I guess. Properly backed up (hashar)
** AdminSettings.php has the same problem bd808 thinks (bryan)
* Changes made should go out immediately as they do for all configuration files.
* Db user and password settings should perhaps go into PrivateSettings (and not be removed from AdminSettings until anyone relying on that file has converted their jobs).
** Migration
*** Move AdminSettings.php contents into PrivateSettings.php
*** Symlink AdminSettings.php to PrivateSettings.php (saves keeping 2 copies of passwords around, but keeps target file in place till known to be perfectly unused)
** Only (obvious) usage would seem to be snapshots - "Fixed" in https://gerrit.wikimedia.org/r/#/c/145017/ (stuff needs moving to PrivateSettings.php first)
reedy@ubuntu64-web-esxi:~/git/operations/puppet$ grep -R AdminSettings *
modules/snapshot/templates/wq.conf.media.erb:adminsettings=wmf-config/AdminSettings.php
modules/snapshot/templates/wikidump.conf.erb:adminsettings=wmf-config/AdminSettings.php
modules/snapshot/templates/addschanges.conf.erb:adminsettings=wmf-config/AdminSettings.php
modules/snapshot/templates/wq.conf.erb:adminsettings=wmf-config/AdminSettings.php
* Better coordination (dunno how to implement this one).
** Header comment in PrivateSettings to tell people to commit (when it's in a repo) and SYNC after changing
=== ../../../20140403-Deploy/ ===
Bryan, Reedy
(bd808) basically had to run scap twice. scap.py updated
* go back to the testwikipedia removal discussion (GREG)
** bug 43722
=== ../../../20140503-Thumbnails/ ===
Antoine, Faidon
* $wgSquidServersNoPurge needed to include the varnish hosts for images
* due to a change by Brandon ($wgSquidServersNoPurge now uses ranges instead of unique IP), the only times this should need to be updated will be with new rows/DCs/etc, hopefully
* we still need some sort of auto-graph-watching that notices anomalies/slope changes/etc
=== ../../../20140517-bits/ ===
Timo
* Tracking bug for the outage: bug 65424
* Monitor for anomalies/spikes in read failures of memcached
** bugzilla:67817
=== ../../../20140529-appservers/ ===
Ori, Reedy
* we really need to get rid of wikimedia-task-appserver debian package in favor of puppet
* Reedy will update status, if this even makes sense any more ;)
=== ../../../20140608-Kafka/ ===
Andrew O.
Most things are done. Two monitoring related RTs to resolve:
* https://rt.wikimedia.org/Ticket/Display.html?id=7828
* https://rt.wikimedia.org/Ticket/Display.html?id=7829
=== ../../../20140612-Math/ ===
Greg
* add schema change test on beta cluster bug
* still need an arch review?
=== ../../../20140618-Wikitech/ ===
Andrew B. / Marc-André
* Need RT filled to add an icinga monitoring of puppet status on wikitech host
** RT 7842
Incident homepage test

Looking for information about incidents or have to report an incident? This page will help you find the right resource!
== Is there an ongoing incident? ==
* First check the page and WikiMedia status page to check if someone else already reported it!
* If you need to report a new incident or don't know what to do next: follow the 
== Want to check past incidents? ==
* Start with the Incident_status page.
* See the Incident documentation list to view all incidents by date
== New SRE or need training on incidents? ==
* We have a Incident_response/Training page just for you!
Incident response/Lightweight report

A lightweight incident report may be used to document incidents that are sufficiently straightforward, well-understood, and low-impact that the effort of writing a full-length incident report isn't justified either by the expected readership or by the investigative writing process itself.
The lightweight format consists only of the sections most important to improving the reliability of production over time: a narrative summary of the incident, and a list of action items to prevent recurrence. The other sections of the full report, which are valuable for analyzing a particularly complex incident, are omitted to save substantial time for authors and reviewers in cases where that analysis isn't necessary.
To write one, submit the form at Incident documentation to create a new page from the lightweight template.
== Decision criteria ==
Don't write a lightweight report when the full-length report would actually be valuable. You should write a full-length report if the answer to any of these questions is "yes":
* Did user impact from the incident attract attention from the community or from WMF management? (This IR is valuable because it would have an interested audience.)
* Did the incident highlight—either positively or negatively—something unique or surprising about our stack, tools, or processes in a way that can help SREs to understand it better? (This IR is valuable for SRE education: we can use it to learn about our systems. This is different from "we discovered a problem and we have action items to fix it," which should be true for every incident.)
* After the incident is resolved, is there still uncertainty or disagreement about the chain of events? (In this case, the collaborative process of compiling the timeline and writing the IR can help answer those questions.)
* Can you imagine telling stories about the incident in six months? (The IR is valuable for historical perspective.)
* Has anyone said they’d like to read the full IR? That’s probably a good enough reason.
Incident response/Lightweight report template

== Summary ==
…
Summary of what happened, in one or two paragraphs. Avoid assuming deep knowledge of the systems here, and try to differentiate between proximate causes and root causes.
Documentation:
* …
Link to relevant source code, graphs, or logs
==Actionables==
* …
Create a list of action items that will help prevent this from happening again as much as possible. Link to or create a Phabricator task for every step.
Add the #Sustainability (Incident Followup) and the #SRE-OnFIRE (Pending Review & Scorecard) Phabricator tag to these tasks.
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours.		
Process 	Was the incident status section actively updated during the incident? 		
Process 	Was the public status page updated? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 		
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)
Incident response/Process improvement

ONgoing reFormulation of Incident Response Efforts (ONFIRE)
== The problem ==
Incident response has been raised repeatedly by the WMF SRE Team as an area for improvement in various ways. In January 2019, a discussion amongst SREs raised the following key pain points:
* “I don’t know if I need to do something”
* very unequal distribution of burden
* not clear what to do or how to do it or who to escalate to
* don’t have shared understanding of process and definitions. 
== Paging & Incident Response Working Group ==
At the January 2019 offsite, the SRE team agreed that a working group, with Joel facilitating/PMing, will identify and offer solutions for the problems with status quo incident reporting, with something to implement before June 2019. Meeting notes for the group are at Incident response/Process improvement/Meetings.
View our charter for more information.
Incident response/Process improvement/Definition of an incident

== Definitions ==
==== Alert ====
An alert is any notification from Icinga or other automated monitoring tool. It may be simply reported on IRC, or file a ticket in Phab. It might also be a page. 
The word alarm is usually synonymous with alert in this context; to avoid confusion, it should not be used.
==== Page ====
A page is an alert deemed serious enough for the automation to potentially wake up humans (currently implemented as sending SMS).
==== Incident ====
An incident is an outage, security issue, or other operational issue whose severity demands a human response. Any incident MUST have a followup incident report.
Incidents will not always begin with a page, but a page MUST always open an incident.
==== Major incident ====
A major incident is an incident of such a severity that it demands "all hands on deck": a DoS attack, attempts to hack into production, wikis being inaccessible in Europe, etc.
An ongoing major incident MUST have an incident coordinator.
Major incidents MUST only ever be declared by a human, and are considered ongoing until declared closed by the coordinator.
==== Incident Coordinator ====
As incidents scale in severity and in size of response, the most important concern becomes communication and coordination. This is the responsibility of the incident coordinator, who SHOULD NOT directly undertake technical measures to help resolve the incident; rather, their responsibility is to coordinate the work of others, ask questions, document what is being done, communicate status with other teams and staff, and ensure that the right people to do so are communicating externally with the community / with the world.
In an incident longer than a few hours, the title of coordinator SHOULD be handed off between people. There MUST always be an active coordinator until the incident is sufficiently mitigated to no longer require a large response team.
(TODO: link to incident severity list)
== Guidelines ==
Anyone on the SRE team, or who is otherwise involved with the operations of WMF sites (e.g. Security team, Performance team, Mediawiki deployers, etc), may declare an incident, if they believe they have discovered a problem severe enough to merit such. They should then pull in others to assist in responding, including manager(s).
Incident status

;Create a new incident report
# Deal with the issue by following Incident response/Runbook.
# Use this form to create a report based on the report template:
 type=create
 useve=1
 preload=Incident response/Full report template
 default=-- myservice
 placeholder=YYYY-MM-DD Service name here
 buttonlabel=Create report
 bgcolor=inherit
 break=no
 prefix=Incidents/
# Write up what happened, filling out the template. Use past incident reports for inspiration.
== Past incidents ==
This page tracks incident reports about Wikimedia Foundation properties, such as Wikipedia.
type=fulltext
prefix=Incidents/
break=no
width=30
searchbuttonlabel=Search
placeholder=Enter search term...
namespace=
=== ===
=== Incident history ===
Check Category:Incident documentation for older incident reports.
=== See also ===
* https://www.wikimediastatus.net (docs).
* Incident graphs by Krinkle.
* on Libera Chat
 
__NOTOC__ __NOEDITSECTION__
Incidents/20110713-PlannedMaintenance

Draft Report on the July 13, 2011 Planned Network Maintenance
== Summary ==
A service impacting network maintenance was scheduled on 7/13/11 from 13:00 UTC to 14:00 UTC but was later extented to 15:00 UTC.
The maintenance completed the replacement and migration to the new MX80 router and resolved several reported networking issues. There were complications with the NICs in those database servers when we had a switch reload, a bug we had seen before with the NIC drivers (Forcedeth). That caused about 25 minutes of rolling service interruption.
== Background ==
On Friday 7/8/11, Mark installed the new Juniper Router (MX80) in our Tampa data center. As some of you might remember, earlier we had issues with network performance, full CAM cache table size  and dropped multicasting. That router upgrade is to address these identified issues permanently. Mark executed the deployment successfully. However that upgrade is just part 1 of the plan. Part 2 requires more preparation and downtime - thus the 7/13/11 scheduled maintenance. After the deployment, Mark did notice multicasting was somewhat sent to servers even they were not supposed to receive them and IGMP snooping was not working correctly on the core switch - csw1-sdpta. (They are not service impacting issues however.) 
== Plan ==
Mark scheduled a Network Maintenance on 13th July to complete the migration to MX80 and his plan included:
* demote the then main Foundry router to be a switch
* reload csw1-sdtpa and asw-b-sdtpa
* fix the broad multicasting issue
* fix the IGMP snooping not working issue (not working on csw1-sdtpa)
== Maintenance period ==
The maintenance started as planned and took longer than expected because of longer preparation time. The maintenance started at 13:00 UTC and was mostly non-service impacting till about 15:00 when Mark reloaded the switch. The database servers did not recover from the brief lost network connectivity and each of their network software had to be reset. This is a known problem with the NIC forcedeth driver. Services started coming  back from 14:55 UTC to 15:18 UTC after the team intervened. Error logs are clean now and Mark is not seeing any problem with the multicasting traffic.
== Note ==
Mark will be installing the 2nd MX80 router (it will provide the needed redundancy) next month, if not earlier. We are waiting for some parts to arrive before we schedule that deployment.
Category:Incident documentation
Incidents/20110926-PowerCable

== What ==
Access switch power outage at 14:12 UTC for about 18 minutes.
==Cause==
Our Tampa data center contractor was replacing the broken management switch msw-d2-sdtpa in sdtpa rack D2. Our management network is completely unrelated to our production network, and doesn't carry any important, production traffic. However, the management switch is mounted directly under the production network access switch (asw-d2-sdtpa). During the replacement, at 14:12 UTC, the power cable of asw-d2-sdtpa was accidentally knocked loose as it was very close to the switch and cables being replaced. The technician didn't notice it because switch LEDs are on the other side.
==Impact==
As a result, the entire rack, consisting of about 40 MediaWiki application servers, lost network connectivity. Our monitoring tool, Nagios, complained about all servers in the rack, and due to all kinds of second order effects, many other services were (briefly) disrupted or overloaded, leading to many other Nagios downtime reports. Several Ops and Engineering members were looking at the downtime, but had difficulty pinpointing exactly what was going down, until Mark noticed that all application servers being reported down were located in the same rack D2 (when he checked in Racktables), and that asw-d2-sdtpa was reported down by Observium (e-mail to root@ and the web interface). A quick check revealed that indeed this switch had gone down. Mark got confirmation from the technician on IRC it was related to his work. As soon as the technician restored power, all network connectivity and services were restored within a minute at 14:30 UTC.
The main problem with a rack of application servers going down, besides the significant reduction in capacity, is the loss of memcached caching - meaning that MediaWiki on the remaining servers will be parsing/regenerating many pages and objects, resulting in a much higher load at reduced capacity for a while. memcached redundancy/resiliency is definitely something that could need improvement for this reason.
==Mitigation==
In our newer racks and data centers, we also have better redundancy of the network, including redundant power and uplinks for access switches, so this doesn't happen so easily. If we do not intend to move Tampa any time soon, we should look at retrofitting our older racks with that setup as well.
Category:Incident documentation
Incidents/20110926-s5switchover

==What==
Database master (S5) switchover caused outage at 22:41 UTC for about 10 minutes.
==Cause==
Our master switch scripts start off with the following:
# set the old (still current) master to read-only
# in the case of MasterSwitch.php, kill queries on the old master running for more than 10 seconds
# in the case of both sets of switch scripts, run "flush tables" against it, to ensure that pending transactions are committed. Unless sql_bin_log is disabled for a session, running "flush tables" on a master is replicated to the slaves.
When Asher was performing the S5 master db switchover, there were three particularly long running queries on db44, one of the two dewiki slaves. Each was a ~11 hour long query and those queries come with the same values in the 'where' clause. It should be returning the same 684 rows each time after apparently examining ~100 billion (!).
When flush tables ran on this host, it locked all tables and essentially hung there, waiting for the multi-hour temp table writes to complete. MySQL will still happily accepting connections and queries, which are queued up "waiting for table" status.
==Impact==
A large number of apache workers were tied up by sending blocked queries to this db, resulting in the actual site outage. A bug in MySQL prevented 'flush tables' cannot truly be killed directly (http://bugs.mysql.com/bug.php?id=44884). The site was restored only after sending to that host a sigterm, which prevented it from accepting additional incoming connections and killing everything off. MySql on db43 was then restarted, and the rest of the replication changes went smoothly.
==Mitigation==
Need to modify the switch scripts to prevent the "flush tables" statement on the master from replicating, as well as automated killing of long running queries on the slaves (instead of just the old master) should prevent a recurrence of this scenario.
Category:Incident documentation
Incidents/20110926-s7switchover

==What==
Database master (S7) switchover caused outage at 18:23 for about 9 minutes.
==Cause==
To prepare for MediaWiki 1.18 release, Asher has been implementing those needed schema changes to the slave database the art week. The final step was to make one of the slaves (of each cluster) to be the new master. This outage happened during the switch over of the S7 cluster. Usually this goes automatically without incidents, but cluster s7 also contains the CentralAuth database, used by all wikis. Before changing masters, Asher set cluster s7 to read-only in the MediaWiki db configuration. As soon as he changed the s7 master to a new database server at 18:23, the new master got bombarded with MediaWiki "slave lag" queries, i.e. MediaWiki instances trying to determine the replication lag of the slaves compared to the master, to select which db slave to use.
==Impact==
Normally, this check is disabled when MediaWiki is in database read-only mode, but as was discovered later, this does not apply to CentralAuth queries - possibly because these are also issued by wikis outside the s7 cluster. Therefore, the new master was immediately overloaded with these queries, and interrupted normal queries and replication traffic, bringing down the sites.
==Mitigation==
Asher investigated the situation, and killed all outstanding queries on the new master, which fixed the problem at 18:32. He also filed a bugzilla ticket #31170 to improve MediaWiki's behavior during db read-only situations. Additional measures to avoid this problem in the future could be to split off CentralAuth into its dedicated cluster on dedicated hardware (we could use some old DB servers for that.
Category:Incident documentation