cityindex/logsearch-development-flow — pull request #148 by sopel (2013-08-23 15:35:53+00:00)
Title: Add EC2 Auto Scaling support

This is early work in progress - since we hadn't discussed the desired/required stack composition changes in detail, I've applied the following assumptions:

* for separation of concerns, it makes sense to run every tier separately now in principle (not the least due to the imminent introduction of spot instances usage, this will in fact reduce cost, while still improving performance and availability on average)
* for a start the most obvious candidate is the logstash parser, which is also the current bottleneck in our running clusters, thus will benefit the most from horizontal scaling
* :exclamation: I'd still like to see *all* tiers running via Auto Scaling (even if it keeps to be one instance only) to gain automatic health checks, spot instance usage etc., but some tiers require load balancing in turn, which is a minor, yet additional complexity best handled via a separate issue once we've implemented the logstash tier

Accordingly, I've now factored out the logstash node(s) into a separate template as follows:

* it would still be possible to keep the new Auto Scaling group in the broker template, but given the success with decomposing the CloudFormation templates for maintenance and resulting parallel processing gains separation seemed natural (see separate tiers above)
* the separation suggest to default the Redis node to `m1.medium` now (same cost basically), because it is memory rather than CPU bound - the new logstash nodes retain `c1.medium` of course
* since Auto Scaling implies health checks, the explicit EC2 instance status check alarm is gone
* :information_source: I've still reused the `AlarmTopicArn` for the Auto Scaling notifications to keep track of how AS handles things (will be more relevant once we switch to spot) - in light of proper noise management for on call management, it will probably be appropriate to separate this down the road.
* I haven't surfaced the Auto Scaling `MaxSize` in the composite template yet, it currently defaults to 4 in the sub template until this has settled a bit
* I've removed the Elasticsearch configuration parts from the broker node, which doesn't need to interact with ES anymore I think (might be incomplete or incorrect)
* the logstash group `WaitCondition` is a bit demoted, insofar it only waits until the first instance in the group signals
* :information_source: while it it possible to provide a number of signals to wait for and this does work fine for initial stack creation, it runs out that `WaitCondition` doesn't support updates - in a way I'd argue that with Auto Scaling in place, this is really just about getting info about when the stack is starting to be usable, thus consider this unimportant.

@dpb587 - I had initially just converted the broker template to use Auto Scaling, which would make it more obvious that this requires only minor changes concerning the former instance (a local diff might still illustrate this), but due to this yielding a non working cluster, I've now tried to actually decompose the tiers as outlined. Still I've focused on the CloudFormation and Auto Scaling mechanics rather than the Logsearch functionality, so please review and test this thoroughly, I might have messed up the actual use case ;)