fluent/fluentd — pull request #468 by y-ken (2014-11-05 05:42:22+00:00)
Title: Flush all buffer by USR1 signal

I found a problem that does not flush all of buffer with USR1 signal.
That's why, the variable of `@next_retry` has very future timestamp which caused by output error.
So, there are no way to re-send buffer immediately expect restarting fluentd which used file buffer .

This pull request makes flushing all of buffer with USR1 signal.

## the sample of fixed behavior

Using this configuration.

```
$ cat td-agent.conf

type forward

# installed fluent-plugin-elasticsearch

type elasticsearch
host localhost
port 9200
logstash_format true
type_name example
request_timeout 30s
flush_interval 10s
# set very long interval for test
retry_wait 1h

```

At first, send message to dead cluster

```
$ sudo service elasticsearch stop
$ echo '{"message":"test"}' | /usr/lib64/fluent/ruby/bin/fluent-cat test.es
```

Raised exception due to the connection issue. (next_retry=2014-11-05 13:55:48 +0900)

```
$ sudo tailf /var/log/td-agent/td-agent.log
2014-11-05 12:48:30 +0900 [warn]: temporarily failed to flush the buffer. next_retry=2014-11-05 13:55:48 +0900 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9200, :scheme=>\"http\"})! couldn't connect to host" plugin_id="object:3fb5e2a8ebd4"
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:63:in `rescue in client'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:60:in `client'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:164:in `send'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:157:in `write'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/buffer.rb:296:in `write_chunk'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/buffer.rb:276:in `pop'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/output.rb:319:in `try_flush'
2014-11-05 12:48:30 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/output.rb:137:in `run'
```

Force flushing with USR1 Signal. But this case still has connection issue.

```
$ sudo kill -USR1 `cat /var/run/td-agent/td-agent.pid`
$ sudo tailf /var/log/td-agent/td-agent.log
2014-11-05 12:48:55 +0900 [info]: force flushing buffered events
```

If it still in trouble connection, increasing next_retry time (next_retry=2014-11-05 14:34:41 +0900)

```
2014-11-05 12:48:56 +0900 [warn]: temporarily failed to flush the buffer. next_retry=2014-11-05 14:34:41 +0900 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9200, :scheme=>\"http\"})! couldn't connect to host" plugin_id="object:3fb5e2a8ebd4"
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:63:in `rescue in client'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:60:in `client'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:164:in `send'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:157:in `write'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/buffer.rb:296:in `write_chunk'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/buffer.rb:276:in `pop'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/output.rb:319:in `try_flush'
2014-11-05 12:48:56 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/output.rb:137:in `run'
```

Recovering elasticsearch and send USR1 Signal to fluentd.

```

$ sudo service elasticsearch start
$ sudo kill -USR1 `cat /var/run/td-agent/td-agent.pid`
$ sudo tailf /var/log/td-agent/td-agent.log
2014-11-05 12:52:34 +0900 [info]: force flushing buffered events
```

Retry immediately and success.

```
2014-11-05 12:52:35 +0900 [info]: Connection opened to Elasticsearch cluster => {:host=>"localhost", :port=>9200, :scheme=>"http"}
2014-11-05 12:52:36 +0900 [warn]: retry succeeded. plugin_id="object:3fb5e2a8ebd4"
```

On troubling connection, it counting up next_retry again.

```
$ sudo service elasticsearch stop
$ echo '{"message":"test"}' | /usr/lib64/fluent/ruby/bin/fluent-cat test.es
$ sudo tailf /var/log/td-agent/td-agent.log
2014-11-05 12:52:58 +0900 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. couldn't connect to host
2014-11-05 12:53:00 +0900 [warn]: temporarily failed to flush the buffer. next_retry=2014-11-05 14:00:13 +0900 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9200, :scheme=>\"http\"})! couldn't connect to host" plugin_id="object:3fb5e2a8ebd4"
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:63:in `rescue in client'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:60:in `client'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:164:in `send'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.6.1/lib/fluent/plugin/out_elasticsearch.rb:157:in `write'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/buffer.rb:296:in `write_chunk'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/buffer.rb:276:in `pop'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/output.rb:319:in `try_flush'
2014-11-05 12:53:00 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.56/lib/fluent/output.rb:137:in `run'
```