Netflix/asgard — pull request #498 by joesondow (2014-03-07 22:21:38+00:00)
Title: ASGARD-1287 Asgard's single health check should fail if caches are missing during startup but should pass if caches are missing later, after init

This change to the health check will enable us to have the desired behavior from ELBs at each of Asgard's lifecycle phases.

During startup, it's important that Asgard fail its health check while loading caches so that ELB traffic will not be directed to a non-ready Asgard.

After cache loading ends, it's important that ELB traffic continue to flow to the single canonical Asgard instance even when Amazon or Asgard has a problem that causes one of the caches to become temporarily empty. If the initial cache loading processes are done OR SKIPPED then the health check should liberally return 200 so traffic is allowed to reach the Asgard instance.

After careful consideration, I realized the existing logic for constantly checking whether the caches were full was not a useful health check for traffic decisions, at least until Asgard is sufficiently stateless that we can run many concurrent identical Asgard instances that all take traffic from the same source. When we get there, we can craft a better health check that regards more than just the default region, and possibly something better than whether a few caches have items in them. For now, we're better off with a load balancer health check that only fails during server initialization.