Incidents/20160801-ORES

== Summary ==
Today, ORES went down for a one hour period during a deployment. This was due to some issues with how the redis password was read from the configuration and some regressions introduced in the V1 response format.
== Timeline ==
=== Short version ===
* 20:33 -- New code is deployed.
* 21:20 -- A hotfix is deployed (see ores:e8d2475)
* 21:37 -- Old code is reverted back to
=== Longer with context ===
* 20:33 < Amir1> !log deploying 624d777 to ores 
* ...
* 20:38 < Amir1> and we have problem with redis
* 20:38 < Amir1> the puppet change is not there 
* ...
* 21:08 < Amir1> https://phabricator.wikimedia.org/diffusion/1912/browse/master/
* 21:09 < Amir1> okay, something urgent. Can you update this manually
* 21:09 < Amir1> it's a mirror
* 21:09 < Amir1> and it's not updated 
* ...
* 21:11 < greg-g> Amir1: who's "you"?
* 21:11 < Amir1> the person who can update that
* 21:12 (in another channel) < greg-g> we need a root in -operations to help amir1
* 21:12 (in another channel) < greg-g> now
* 21:13 (in another channel) < greg-g> mutante: robh ^ 
* 21:14 < chasemp> I'm confused on what needs to be changed Amir1
* 21:14 < chasemp> are you saying you pull from diffusion and it's lagging behind has caused an issue?
* 21:14 < Amir1> yeah
* 21:14 < Amir1> chasemp: yup
* 21:14 < chasemp> diffusion is on a staggered update schedule based on change rate in a repo iirc
* 21:15 < chasemp> let me see here if I can force it 
* 21:17 < chasemp> !log iridium sudo -u phd /srv/phab/phabricator/bin/repository update 1912
* 21:20 < Amir1> !log deploying e8d2475 to scb nodes
** A hotfix is deployed (see ores:e8d2475)
* 21:23 < halfak> We're up!
* 21:23 < Amir1> it's up
* 21:23 < Amir1> yeah
* ...
* 21:27 < greg-g> still seeing the "no model available for {blah}" fatal
* 21:28 < Amir1> where?
* 21:28 < MaxSem> in exception log 
* ... unclear what is broken/if it is broken (halfak and amir discussing this)
* 21:33 < Amir1> let's rollback and check later
* 21:33 < Amir1> mutante: hey, can you revert that? 
* 21:33 < greg-g> halfak: the ORES extension is fataling in prod, it needs to be reverted now
* 21:34 < grrrit-wm> (PS2) Dzahn: Revert "ores: changes for configs for the refactor" [puppet] - https://gerrit.wikimedia.org/r/302352
* 21:34 < mutante> Amir1: ^ that?
* 21:34 < Amir1> yup 
* 21:37 < Amir1> !log deploying 6790ccb
* 22:41 < MaxSem> huh, ores is still broken?
* 22:41 < halfak> Oh... It shouldn't be.
* 22:41 < halfak> MaxSem, looks up to me
* 22:41 < MaxSem> RuntimeException from line 136 of /srv/mediawiki/php-1.28.0-wmf.12/extensions/ORES/includes/Cache.php: No model available for [361156268]
* ...
* 22:43 < MaxSem> not one error, flood of them
* 22:44 < MaxSem> resumed at 22:23 UTC 
* 22:55 < halfak> MaxSem, it looks to me like error you cited stop by 22:41
* 22:55 < halfak> The errors that occur after that are expected.
== Conclusions ==
The deployment to beta did not show issues for two reasons. 
# the redis server used in beta requires no password
# the ORES Extension on Beta uses the prod installation of the ORES service
== Actionables ==
* ores:e8d2475 fixes config issue
* ores:3d6f5dd fixes output format issue
* Set up password on ORES Beta redis server (Phab:T141823)
* Config beta ORES extension to use the beta ORES service Phab:T141825
* Review new service 'pre-deployment to production' checklist Phab:T141897
Category:Incident documentation