python-zk/kazoo — pull request #77 by mgius (2013-04-30 21:36:06+00:00)
Title: Tweaks to make KazooClient.stop() less likely to raise an exception

Three changes, all of which are intended to make the "Writer still open"
exception less likely to fire.

1) Adjust _safe_close timeout to match session_timeout, so that clients
which have adjusted the master timeout higher won't stop() timeout while
an inner timeout is still sleeping.

2) Add a breakout point to _connect_loop so that if a client has been
stopped, we don't keep waiting on a dead socket

3) Set a socket timeout on the initial connect to remote endpoints so
that we don't sit around waiting a ton of time for a dead endpoint to
respond. The modifier on session_timeout was chosen arbitrarily.

Context
=======

Kazoo clients handle remote zookeeper hosts which have died in a very
bad way. To reproduce the issues we've been seeing, start up a
Zookeeper server on a virtual machine, connect using Kazoo, then
pull the power plug on the virtual machine. Now call stop() on the
kazoo client. It will always raise the "Writer still open" exception,
because the Connection instance is stuck either in the _connect_loop
select loop or in the _connect waiting for a socket connection to
something that will never come around. As far as I could tell, the
underlying Connection never really recovered and cleaned itself up, and
further calls to stop() on the client are no-ops because the client
stopped flag is still set.

Fixes 2 and 3 allow me to call stop() at any point while trying to
connect to an invalid ip / previously valid IP and have the stop()
return successfully within the timeout. Fix #1 I put in because for some
very large timeouts (60s+?) you could have an inner timeout that is
longer than 10s, so you'd pretty much never be able to call stop()
successfully if you weren't actively connected to Zookeeper.