ryanlecompte/redis_failover — pull request #57 by arohter (2013-10-15 05:12:45+00:00)
Title: Ensure free_client() method is called on connection retries.

We need to explicitly call free_client() within the retry loop, as the ensure section is called only once, on rescue block exit.

Every time a redis command is sent, client_for() is called, adding the client connection object to a stack. Under normal usage, a subsequent free_client() call is made, removing this client object from the stack. However, on connection failure, with retries enabled (the default), the rescue::retry loop triggers client_for() multiple times, without matching free_client() calls (ruby ensure sections are only called at final rescue block exit). This results in duplicate copies of the client object to remain in stack, which effectively caches the client connection forever, resulting in perpetual stale server selection, breaking failover completely.

We ran into this almost immediately when testing failure handling with a simple rpush+blpop ping/pong test.
I'm pretty sure https://github.com/ryanlecompte/redis_failover/issues/42 is a manifestation of the same problem, and why using .clear instead of .pop happened to "fix" things for them.