cityindex/logsearch — pull request #309 by dpb587 (2014-01-14 04:33:05+00:00)
Title: Rollup - Elasticsearch Auto-Scaling + logstash-1.3.2

I created this branch in order to combine the elasticsearch auto-scaling and logstash update branches together. Assuming the mentioned integration tests are sufficient, we'll merge this before tomorrow's deploy. From this branch...

I successfully ran the tests on a clean instance to verify logstash-1.3.2 is still working as expected:

ubuntu@ip-10-49-127-242:/app/app$ rake logstash:test:all_types
==> All tests completed successfully :)
ubuntu@ip-10-49-127-242:/app/app$ echo $?
0

I successfully deployed the CloudFormation stack with the command:

rake deploy_aws_cloudformation_stack[logsearch-dpb587-`date +%Y%m%dT%H%M%S`,ci-logsearch,config/logsearch/cluster,composite-ci-r53,"ParameterKey=ElasticsearchSnapshotId;ParameterValue=snap-ce7f062c"]

I ensured elasticsearch successfully loaded the (now three) nodes:

![screen shot 2014-01-13 at 9 05 44 pm](https://f.cloud.github.com/assets/207601/1907553/3f3f286c-7cd1-11e3-92b2-3a3bb9bc5475.png)

I successfully ran several load tests against the branch stack (primarily to verify the `t1.micro` frontend would not be a bottleneck) and the results matched my recollection of my previous, similar tests:

![screen shot 2014-01-13 at 9 01 15 pm](https://f.cloud.github.com/assets/207601/1907535/913b40fc-7cd0-11e3-849f-6f05a93e6a57.png)

Out of curiosity, I subsequently ran the exact same load test against the running cluster:

![screen shot 2014-01-13 at 9 28 33 pm](https://f.cloud.github.com/assets/207601/1907614/6507579c-7cd4-11e3-90ec-3b82a679ca58.png)

The running cluster has noticeably different CPU trends between the elasticsearch nodes which I would guess is due to the active write parsing, but otherwise the results are sufficiently similar. The new cluster *averaged* 30 ms longer in request response time.