guardian/deploy — pull request #219 by rtyley (2014-12-19 15:11:46+00:00)
Title: Improved waiting for stabilisation in Auto-Scaling deploys

We had a failing deploy on the Membership team, and took some time to discuss it with @philwills :

https://riffraff.gutools.co.uk/deployment/view/582120b7-0d4e-4759-8bdd-21d334b98aae

These are the relevant lines:

```
[11:28:04] task DoubleSize Double the size of the auto-scaling group in Stage(PROD) for apps App(frontend)
[11:28:04] task WaitForStabilization Check the desired number of hosts in both the ASG and ELB are up and that the number of hosts match
[11:28:05] task HealthcheckGrace Wait extra 0ms to let Load Balancer report correctly
[11:28:05] task WaitForStabilization Check the desired number of hosts in both the ASG and ELB are up and that the number of hosts match
[11:28:05] task CullInstancesWithTerminationTag Terminate instances with the termination tag for this deploy
[11:28:08] Unhandled exception in task CullInstancesWithTerminationTag Terminate instances with the termination tag for this deploy
[11:28:08] Unhandled exception in deploy for Membership::frontend (build 3589) to stage PROD
```

You can see that the 1st and 2nd `WaitForStabilizations` executed almost instantly (and we haven't had a `HealthcheckGrace` duration configured for our deploy, thus the intervening `HealthcheckGrace` executed instantly too). So, all these checks executed before the AWS API could catch up and report the updated values from the `DoubleSize` step. Consequently the deploy proceeded with only 3 instances in our ASG & ELB, not the desired double-capacity of 6, and then the instance-culling failed.

Setting a HealthcheckGrace would alleviate our problem - but then the first `WaitForStabilization` becomes completely pointless. So this commit removes the 1st `WaitForStabilization` - and sets the default `HealthcheckGrace` to 3 seconds, which hopefully is both harmless & helpful :smile:

cc @jennysivapalan