Increase Logstash ES output max_inflight_requests.

The default value for the Logstash ElasticSearch max_inflight_requests
parameters is 50. Bump that up to 512, to have a bigger firehose to
ElasticSearch. ElasticSearch is the piece of the Logstash puzzle that
doesn't seem to have trouble with bigger firehoses.

Change-Id: I3eeaa036c68b00a8daa0896a4256b30043f04aaa
Reviewed-on: https://review.openstack.org/30769
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Allocate more memory to ElasticSearch indexing.

* modules/logstash/files/elasticsearch.yml: By default ElasticSearch
allocates 10% of its memory space to indexing. Bump this up to 40% to
help ensure indexing does not block on needing more memory.

Change-Id: Ic2b862235c12467213ab9916e7f85d92559efc37
Reviewed-on: https://review.openstack.org/30796
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Scale out logstash indexing to multiple hosts.

Logstash performs filtering in a single thread so it does not scale up
very well. Work around this by scaling Logstash out to multiple indexer
hosts.

Current plan is to have a small (2GB) kibana web front end host that
does nothing but talk to elasticsearch, three 4GB logstash indexers that
will run a single log-pusher.py + logstash indexer with some partition
of the logfiles assigned to each indexer, and finally the existing large
elasticsearch node.

Eventually properly load balancing log processing across the worker
nodes would be great, but the current partition method should work well
enough with little additional effort.

Change-Id: Ifc6396560934314ffd6a7c47eb2acff9e9c2a7af
Reviewed-on: https://review.openstack.org/30573
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins