logstash/logstash — pull request #708 by jordansissel (2013-10-08 18:34:14+00:00)
Title: Add 'workers' setting to output plugins.

This allows you to run multiple workers for a single output in cases
where CPU or network IO are bottlenecks in a single output. The
original use case was to help improve elasticsearch_http performance.

elasticsearch_http blocks until the index request gets a response, so
simply doing more requests in-flight at any given time will improve
performance assuming processing or network-time is your bottleneck,
which in most cases it is.

In testing this, for 1,000,000 events, it took 150 seconds with 1
worker, 100 seconds with 2 workers, and 70 seconds with 4 workers. The
test was done on an EC2 m3.2xlarge logstash node writing to a two-node
elasticsearch 0.90.5 cluster on the same instance type (3 nodes total)