elasticsearch/logstash — pull request #1885 by haoch (2014-10-13 05:18:54+00:00)
Title: Balance problem about logstash redis output with multi-hosts

Logstash redis output plugin supports multi-hosts now.

~~~ruby
config :host, :validate => :array, :default => ["127.0.0.1"]
~~~
__BUT__ it just creates one connection per thread according `@workers`'s count and rely on `@host.shuffle!` to choose which host to connect randomly in certain thread.
~~~ruby
if @shuffle_hosts
@host.shuffle!
end
~~~
So in fact there is high probability that only __part__ of redis hosts in the hosts array are really selected and some are even __duplicated__ in different threads, where the worst situation is that all threads may use one same redis host only. If logstash is configured to use `congestion_threshold` which will limit the cached key size in __connected__ redis instances, it may even cause these connected redis instances are overused and corresponding logstash agent threads are __blocked__, while other un-used redis instances are free but wasted.

**The problem will reduce balance and performance of the whole redis cluster and throughput of the logstash agent badly**

This PR is with one of the simplest solutions (not including connection pool under consideration that redis connection is cheap) and mainly to consult for your suggestions.

How would you think about it then?

Regards,
Hao