Incidents/2023-05-19 videoscaler/jobrunner

== Summary ==
A large video-scaling job made server mw1469 so busy that its capacity was maxed out by ffmpeg processes. Since mw1469 was both a jobrunner and a videoscaler this led to alerts for both jobrunner and videoscaler services.
For the first part of the incident alerts could be seen on IRC but there had been no pages yet. Alerts were flapping, also on mw1469 specifically. Around 19:24 it eventually triggered a page. Dzahn and Aokoth were paged and
started looking at it and kept an eye on it for a while. Since it kept flapping the runbook was followed (Application servers/Runbook#Jobrunners.) and mw1469 was depooled from videoscaler, but pooled in jobrunner. The ffmpeg processes on mw1469 were killed. This protected the jobrunner which is much more important than the videoscaling (quoting runbook). Jobrunner alerts recovered. 
A little while later, server mw1495 was depooled from jobrunner and turned into a dedicated videoscaler. Videoscaler alerts recovered.
==Timeline==
Write a step by step outline of what happened to cause the incident, and how it was remedied. Include the lead-up to the incident, and any epilogue.
Consider including a graphs of the error rate or other surrogate.
Link to a specific offset in SAL using the SAL tool at https://sal.toolforge.org/ (example)
All times in UTC.
*19:04 flapping of IRC alerts begins
*19:24 page is sent
*19:25 Dzahn ACKs alert, starts investigating, watches the situation
*19:45 Since alerts are still flapping, Dzahn depools mw1469 from videoscaler, to protect jobrunner
*19:46 Dzahn kills ffmpeg processes on mw1469 (as instructed per runbook)
*19:47 alerts recover on mw1469
*20:23 alerts start on mw1495
*20:52 mw1495 is depooled from jobrunner, made dedicated videoscaler, so it can finish mmpeg processes eventually
*20:53 alerts for videoscaler recover
TODO: Clearly indicate when the user-visible outage began and ended.
==Detection==
First Icinga started reporting via icinga-wm on IRC, a little later SRE on duty got paged via Alertmanager.
==Conclusions ==
OPTIONAL: General conclusions (bullet points or narrative)
===What went well?===
* …
OPTIONAL: (Use bullet points) for example: automated monitoring detected the incident, outage was root-caused quickly, etc
===What went poorly?===
* …
OPTIONAL: (Use bullet points) for example: documentation on the affected service was unhelpful, communication difficulties, etc
===Where did we get lucky?===
* …
OPTIONAL: (Use bullet points) for example: user's error report was exceptionally detailed, incident occurred when the most people were online to assist, etc
==Links to relevant documentation==
* https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Jobrunners
* https://wikitech.wikimedia.org/wiki/Conftool#Modify_the_state_of_a_server_in_a_pool
==Actionables==
* reopen https://phabricator.wikimedia.org/T279100 ?
* dedicated alert for videoscalers https://phabricator.wikimedia.org/T338220
==Scorecard==
Incident Engagement ScoreCard
 	Question 	Answer
(yes/no)	Notes
People 	Were the people responding to this incident sufficiently different than the previous five incidents? 		
People 	Were the people who responded prepared enough to respond effectively 		
People 	Were fewer than five people paged? 		
People 	Were pages routed to the correct sub-team(s)? 		
People 	Were pages routed to online (business hours) engineers?  Answer “no” if engineers were paged after business hours.		
Process 	Was the "Incident status" section atop the Google Doc kept up-to-date during the incident? 		
Process 	Was a public wikimediastatus.net entry created? 		
Process 	Is there a phabricator task for the incident? 		
Process 	Are the documented action items assigned? 		
Process 	Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? 		
Tooling 	To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are
open tasks that would prevent this incident or make mitigation easier if implemented.		
Tooling 	Were the people responding able to communicate effectively during the incident with the existing tooling? 		
Tooling 	Did existing monitoring notify the initial responders? 		
Tooling 	Were the engineering tools that were to be used during the incident, available and in service? 		
Tooling 	Were the steps taken to mitigate guided by an existing runbook? 		
Total score (count of all “yes” answers above)	Total score (count of all “yes” answers above)