cityindex/logsearch — pull request #290 by dpb587 (2014-01-06 16:32:50+00:00)
Title: Basic Auto-Scaling of Elasticsearch (#282, #270)

At the core, we can now use AWS Auto-Scaling to manage our elasticsearch cluster. Instead of duplicating hefty chunks of code in our main CloudFormation template to add a node, we now simply update a single parameter to adjust the size of the cluster. While it's technically easy to automatically add or remove a node, it's logistically difficult to automatically rebalance the data across a new cluster size (so that's still a manual optimization step). It's pretty fun to be able to quickly add a new node and see it magically appear on an elasticsearch dashboard. The result is a new CloudFormation template called [group-elasticsearch-default.template](https://github.com/cityindex/logsearch/blob/cfn-ec2-as-elasticsearch/.build/aws/cloudformation/group-elasticsearch-default.template).

![screen shot 2014-01-03 at 3 30 59 pm](https://f.cloud.github.com/assets/207601/1851796/70674a88-76ef-11e3-91eb-ff4623fbc431.png)

This auto-scaling does present a new problem, however. Previously our main CloudFormation template created DNS records which pointed to one of the elasticsearch nodes. As an auto-scaling group, those IPs are no longer directly accessible from within CloudFormation. Basically, we need a reliable elasticsearch endpoint for:

* nginx to proxy kibana HTTP elasticsearch requests to
* logstash nodes to push parsed log messages to
* ssh tunneling to

I thought of a couple alternatives:

* **Elastic Load Balancer** - [AWS ELB](http://aws.amazon.com/elasticloadbalancing/) provides a fairly easy way to proxy ports (HTTP or just TCP) to a cluster. It would be easy to tie the AWS Auto-Scaling group with the ELB so new instances are automatically registered.
* **Dedicated Instance** - create a very lightweight, dedicated instance which runs elasticsearch in "proxy only" mode. Using elasticsearch's built-in cluster discovery capabilities, node changes would be immediately noticed.
* **AWS API Updates** - we could use a cron job and the AWS API to automatically update the DNS endpoint with the most correct IP address.
* **Elastic IP**, **VPC ENI** - while unsupported by auto-scaling, the API could be used to attach static IP addresses to instances once booted.

I picked the Dedicated Instance option as my preferred implementation though because:

* ELB is AWS-specific whereas this could run on any generic host;
* using elasticsearch's built-in discovery capabilities should be much quicker and more effective than the ELB heartbeat approach;
* API calls seem fragile and error-prone (given the way we'd have to implement it);
* this dedicated instance could also take the responsibility of hosting kibana and we could get rid of nginx/kibana on the other elasticsearch instances; and
* I've [mentioned](http://labs.cityindex.com/labs-team/2013/10/01/logsearch-thoughts-for-possible-improvements/) proxy instances before as something I want to move towards and investigate.

The result is a new CloudFormation template called [node-kibana-default.template](https://github.com/cityindex/logsearch/blob/cfn-ec2-as-elasticsearch/.build/aws/cloudformation/node-kibana-default.template) and minor adjustment to disable kibana on the auto-scaling template. I'm not a huge fan of adding an additional node for the reasons of cost, which is why I'm pushing for the related discussion on creating reliable spot instances and their significant cost savings.