elasticsearch/logstash — pull request #1646 by driskell (2014-08-19 10:48:47+00:00)
Title: Fix redis plugin crashes on timeouts and other errors (LOGSTASH-1475 et al.)

Reproduce timeout crash (LOGSTASH-1475 et al.)
1. Start redis and logstash
2. Stop redis and immediately start a blackhole listener to create symptoms of a connect timeout that raises the same exception as a regular timeout (e.g. `service redis stop; nc -kl 6379 >/dev/null`)
3. Watch logstash logs. "Failed to get event from redis" appears and the plugin crashes and is restarted. This occurs on every timeout occurrence.

Fix for timeout crash:
Capture all redis related errors and not just connection errors.
Also clear redis and reconnect again - it cuts down on the EVAL errors due to batch script missing and is a proper reconnect.

Reproduce subscribe crash and stop: (noticed it while fixing above)
1. Start redis and logstash with data type channel
2. Stop redis and immediately start a blackhole listener to create symptoms of a connect timeout that raises the same exception as a regular timeout
3. Watch logstash logs. "Failed to get event from redis" appears and the plugin crashes and is restarted.
4. However, it then doesn't appear to restart properly sometimes and ends up dead

Fix for subscribe crash and stop:
Same as for timeout crash but also protect plugin teardown from crashing. It seems it calls "unsubscribe" which could throw a connection error ("quit" never will) which crashes the plugin restart process.
Also clear redis in teardown so we force a reconnect on startup - otherwise we try to use same connection and rely on redis-rb to reconnect, which is not a true "restart" and results in extra EVAL errors due to batch script missing.