Incidents/20171120-Ext:ORES/minutes

== Post Mortem Meeting for train blocker - 20171120-Ext:ORES ==
=== Meeting Date: ===
December 7, 2017
=== Attendees: ===
JR, Adam W, Aaron H, Chad H, Greg G, Victoria C
=== For reference: ===
* Incident Report-https://wikitech.wikimedia.org/wiki/Incident_documentation/20171120-Ext:ORES
==== Main Task ====
* https://phabricator.wikimedia.org/T181006
==== Related Tasks ====
* https://phabricator.wikimedia.org/T181010
==== Resulting Tasks ====
* https://phabricator.wikimedia.org/T181191
* https://phabricator.wikimedia.org/T181183
* https://phabricator.wikimedia.org/T181071
* https://phabricator.wikimedia.org/T181067
* https://phabricator.wikimedia.org/T181187
== Topics: ==
=== Brief Summary of Problem(s) ===
* Code change that made the "threshold API" (used by RC, WL, Contribs dynamically)
* Fixed a crash and made it return "null" (in a good way)
** Didn't analyze what would happen when the ORES Ext would receive those values
* Extension starts recieving "null" and RC/WL/Contribs crashed for users
=== How problem(s) was/were discovered? ===
* RU/FR wikis threw an exception on RC/WL/Contribs
* awight heard about it on IRC (~45 minutes after deployment)
** note that deployments to the service can affect the extension
** In this case, the extension was not being monitored for a service deployment
* awight wasn't sure what the fix was (talked in #wikimedia-ai)
* rollback started 4 minutes later, but wasn't done correctly right away (related to deployments to a new experimental cluster)
** 2 rollbacks failed before success (which started 1 hour later)
*** Note that once the first failed rollback happen, the revid issue was apparent
** There were lots of upset people ("pressure") in #wikimedia-operations
** Before the rollback complete, we disabled ORES on FR/RU wiki
* Took 1-2 days to re-enable ORES (post rollback) and deploy a fix to ORES Ext.
=== How problem(s) were introduced? ===
* Why didn't this problem manifest in Beta?
** Large number of wikis that could have been affected.  Frwiki isn't even on beta cluster.  Still ruwiki was, so we could have reviewed all pages on the available beta wikis
** There was some configuration in Beta that prevented ORES from being enabled in a way in which it would crash there (would have only seen a working page)
*** TODO: (Adam) add documentation to ORES/Deployment explaining that the beta config must be carefully checked, and left empty once deployment to production is complete. (done)
** Beta is not a perfect mirror of production
* Collab and Scoring Platform share ownership of the ORES extension
** Configuration belongs to two teams.  Tried to split the code, but that was determined to be too expensive.
** Part of our deployment pattern involved work on the ORES Ext., later the Collab does more work on the ORES Ext. to make scores visible to users. 
=== What was learned? ===
==== How could problem(s) could have been avoided ====
* Graceful degradation ({{done}})
** And for the future -- a complete refactor of the extention by a more experience version of an engineer. 
* Monitoring extension graphs during deployments (and include certain common types of failures)
* Beta config
* Beta quality-assurance -- too difficult.  30 pages is too many.  
==== How could we detect this kind of problem before train/deployment ====
* low-traffic staging
==== What changes have been put in place or are planned to avoid similar scenario in future deployments. ====
* Covered in incident report and resulting tasks above.
=== Action Items ===
* ()greg/JR: reviews of extensions
** greg: send a note to Daniel Kinsler about extension review
* ()chad: proximity of service deployments to train deployments is a problem -- do something about that