boundary/ordasity — pull request #16 by jdanbrown (2012-09-14 03:28:36+00:00)
Title: Session disconnect doesn't shutdown work

In [Cluster.connectionWatcher](https://github.com/boundary/ordasity/blob/master/src/main/scala/com/boundary/ordasity/Cluster.scala#L133) a `KeeperState.Disconnected` doesn't trigger `forceShutdown`, which is responsible for calling `listener.shutdownWork` and `listener.onLeave`. My guess as to why this is is that it's waiting for a `KeeperState.Expired`, which does trigger `forceShutdown` and is otherwise identical (modulo logging).

Based on my recent experience and the [zk session state-transition docs](http://zookeeper.apache.org/doc/r3.3.3/zookeeperProgrammers.html#ch_zkSessions), this allows the following behavior:

1. Client partitions from zk cluster
2. Client-side zk session timeout triggers, client receives `KeeperState.Disconnected`
3. Clients stays partitioned from network for arbitrarily long, but ordasity continues running its previously-claimed work
4. Client eventually rejoins network and regains route to zk cluster
5. Client attempts to reconnect to the zk cluster, cluster says no, client receives `KeeperState.Expired`, ordasity finally shuts down work
6. Client establishes new session, ordasity claims new work

This is harmful in my application since I want work ownership to be best-effort exclusive, i.e. nodes should minimize their overlap in work.

Is this a bug, or was ordasity intentionally designed to _maximize_ this kind of overlap when nodes partition? I can imagine that being a useful—or at least not harmful—behavior in some settings. If so, maybe I can elaborate this proposed change to include a config option to maximize vs. minimize oblivious work overlap?