python-zk/kazoo — pull request #41 by cgordon (2012-12-10 21:37:05+00:00)
Title: ConnectionDropped bug fix.

Fixing a bug that causes the writer thread/greenlet to exit on dropped connections. In the _send_request method there are three places in which the _submit method is called. Two of them happen in the body of the "try", while the third happens in the exception handler for self.handler.empty exceptions.

If the connection is dropped in either of the first two calls to _submit, it triggers the "except Exception" handler, which causes the method to return False, thereby triggering the writer thread/greenlet to shut down. The other call to _submit is already in an exception handler, so if it raises a ConnectionDropped it is handled in the _connect_loop, which is what should happen. The straightforward fix is to just forward ConnectionDropped exceptions for the other two cases as well.

It isn't very easy to test this case, as it requires a ConnectionDropped to occur while in a write. We will be testing this change on our production servers over the next couple weeks to see if it fixes the (rare) issue.