lusis/chef-logstash — pull request #92 by jbrook (2013-03-02 11:29:23+00:00)
Title: Enable AWS discovery for Elasticsearch Output

Here is my first attempt at a recipe to enable AWS discovery. I have tested it on Ubuntu 12.04. At the moment the attributes to populate the elasticsearch.yml are completely independent of the Karmi Elasticearch recipe. I know that recipe does slightly more sophisticated things with these attributes, including checking a databag for the AWS access key and secret key.

This works for the embedded client in the output and I suspect it would also work for clustering fully embedded Elasticsearch.

It could probably easily be extended to include other ES plugins. I am not sure where static resources like HTML would need to be placed for embedded ES to serve them.

----

Added a recipe to enable the embedded client in the logstash output to automatically discover elasticsearch nodes running on Amazon EC2. The default mulicast discovery will not work on EC2.

* Download the cloud-aws plugin, unzip it and link it the logstash home directory. Uses the ark cookbook.
* Modified the logstash startup scripts to specify a classpath instead of the jar argument. Optionally include the cloud-aws plugin on the classpath.
* Place an elasticsearch.yml config file in the logstash home directory. This file contains the cluster name, AWS access key, etc.