Netflix/curator — Issue #3: possible failure mode in locking
Author: sbridges
Created: 2011-10-22 15:51:59+00:00
State: open

In LockInternals I think this line is problematic,

client.create().withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(path, new byte[0]);

The problem is that the create call may succeed on the server, but the server may crash before it can send the response to the client. If that happened, an ephemeral node would exist in zookeeper, but the client would see an error, and not realize that it had acquired the lock.

This can occur without the session becoming invalid. For example if a client is talking to server A, A crashes, and the client reconnects to server B. The clients session is still valid, and the ephemeral node exists, but the client does not know it exists. What is worse, the client does not even know the path of the node to delete since the node is created with EPHEMERAL_SEQUENTIAL.

What we do is embed a uuid in path, with the uuid being unique for every lock attempt. On failure to create, the client schedules a getChildren to be performed when it reconnects, and deletes the node based on the uuid.

The zk tree looks like,

/path/to/lock-uuid-1
/lock-uuid-2
/lock-uuid-3