rackerlabs/otter — pull request #675 by manishtomar (2014-07-21 20:37:29+00:00)
Title: Delete lock znode after group delete

After the group is deleted, `/locks/` znode in Zookeeper remains wasting space as group is never going to be locked. Hence, deleting the znode and its children (if any) after successful group delete. You would wonder if this can cause problem when a server becomes active after the group is deleted and lock is acquired. This is fine since kazoo locking recipe will just re-create the znode. Unfortunately, the newly created znode will become orphaned when all the servers related the group become active. This in reality should be less number assuming not many customers will do this. However, many tests do this. But it is still better than all orphaned znodes.

There are currently many orphaned znodes in all DCs because of this. ORD `zkCli.sh` is not able to display the list complaining the packet to too large :/ (`java.io.IOException: Packet len4449193 is out of range!`). I'll probably have to write a separate script that checks the znodes with all groups in CASS and delete unwanted ones. This script can be run again sometime later.