uken/fluent-plugin-elasticsearch — pull request #67 by mattheworiordan (2014-08-30 12:59:42+00:00)
Title: Connection reset & retry when connection is lost

On all my servers in EC2, after some time (ranging from a few hours to a few days), I get a Faraday error stating that this plugin cannot connect to my Elasticsearch server anymore. The odd thing is that:

* The Elasticsearch server is healthy and receiving logs from other machines at the time
* A restart of fluentd immediately resolves the issue
* An internal IP address that I have no idea where it comes from is listed as the server it is trying to connect to, see log pasted below.

I have no idea why the plugin is trying to connect to an internal IP address with the default port 9200 when the connection details I provided were based on a DNS name and on port 443. However, in order to mitigate this, I have added some better recovery features:

* If a connection fails, create a new connection and retry
* Do not ping the ES server for every request, this is unnecessary and adds lag & load
* Log information about connection state and failures

I have rolled this out into production and so far it works, although I will need to give it a day or two to see if it actually resolves the issue I found where logs simply stop being sent.

```
2014-08-29 20:03:03 +0000 [info]: following tail of /logs/syslog
2014-08-30 04:20:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2014-08-30 04:20:40 +0000 error_class="Faraday::ConnectionFailed" error="Failed to connect to 172.17.0.5 port 9200: Connection refused" instance=32026160
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/patron-0.4.18/lib/patron/session.rb:223:in `handle_request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/patron-0.4.18/lib/patron/session.rb:223:in `request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/faraday-0.9.0/lib/faraday/adapter/patron.rb:33:in `call'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139:in `build_response'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/faraday-0.9.0/lib/faraday/connection.rb:377:in `run_request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/base.rb:187:in `call'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/base.rb:187:in `perform_request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/client.rb:92:in `perform_request'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/elasticsearch-api-0.4.11/lib/elasticsearch/api/actions/ping.rb:19:in `ping'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/bundler/gems/fluent-plugin-elasticsearch-5281b7b20d98/lib/fluent/plugin/out_elasticsearch.rb:55:in `client'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/bundler/gems/fluent-plugin-elasticsearch-5281b7b20d98/lib/fluent/plugin/out_elasticsearch.rb:138:in `send'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/bundler/gems/fluent-plugin-elasticsearch-5281b7b20d98/lib/fluent/plugin/out_elasticsearch.rb:133:in `write'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/fluentd-0.10.52/lib/fluent/buffer.rb:296:in `write_chunk'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/fluentd-0.10.52/lib/fluent/buffer.rb:276:in `pop'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/fluentd-0.10.52/lib/fluent/output.rb:310:in `try_flush'
2014-08-30 04:20:39 +0000 [warn]: /etc/fluent/vendor/bundle/gems/fluentd-0.10.52/lib/fluent/output.rb:132:in `run'
```