rackerlabs/blueflood — pull request #361 by dlobue (2014-08-13 22:48:00+00:00)
Title: Dominic/indexing perf enhancements

Some indexing performance and safety enhancements.

Do not use abort rejection handler for DiscoverWriter threadpool

Do not use abort rejection handler for DiscoverWriter threadpool. The
abort rejection handler throws an exception and causes work to be lost
if no worker is available for a job.

We do NOT want to throw away work, so use the default rejection handler
which effectively ensures all work is done, and effectively throttles
ingestion.

Configure DiscoveryWriter threadpool size by itself

The DiscoveryWriter threadpool does not need to be very big. Create
configuration new config options specific to DiscoveryWriter TP size for
maximum control when tuning.

Do not re-index existing metrics in ES

Use the `CREATE` optype to use the "put-if-absent" functionality in
Elasticsearch. This will cut down on extra and unnecessary indexing
operations and IO in elasticsearch.