csfrancis/zk-lock — pull request #1 by csfrancis (2013-11-20 14:51:38+00:00)
Title: Fix deadlock on shutdown

This fixes the following deadlock that can sometimes occur on shutdown:
```
(gdb) info threads
Id Target Id Frame
3 Thread 0x7fb93144a700 (LWP 3075) "ruby" 0x00007fb9303fa763 in select () from /lib/x86_64-linux-gnu/libc.so.6
2 Thread 0x7fb92fa19700 (LWP 3076) "ruby" 0x00007fb93101c89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
* 1 Thread 0x7fb93143c700 (LWP 3072) "ruby" 0x00007fb931017148 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) where
#0 0x00007fb931017148 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb92fa22f7d in connection_data_free () from /home/deploy/scott/zk-lock/ext/zklock.so
#2 0x000000000041e02d in run_final (obj=37554320, objspace=) at gc.c:3033
#3 finalize_list (objspace=0x23968a0, p=0x23d0890) at gc.c:2055
#4 0x0000000000422e70 in rb_objspace_call_finalizer (objspace=0x23968a0) at gc.c:3145
#5 rb_gc_call_finalizer_at_exit () at gc.c:3080
...
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fb92fa19700 (LWP 3076))]
#0 0x00007fb93101c89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) where
#0 0x00007fb93101c89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb93101a6b0 in pthread_cond_broadcast@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007fb92fa245d2 in set_lock_state_and_signal () from /home/deploy/scott/zk-lock/ext/zklock.so
#3 0x00007fb92fa24d44 in cb_zk_delete () from /home/deploy/scott/zk-lock/ext/zklock.so
#4 0x00007fb92fa2869c in deserialize_response (type=, xid=1384956119, failed=0, rc=0, cptr=0x7fb928010fd0, ia=) at src/zookeeper.c:2078
#5 0x00007fb92fa28a03 in process_completions (zh=0x7fb928002aa0) at src/zookeeper.c:2119
#6 0x00007fb92fa2b748 in zookeeper_process (zh=0x7fb928002aa0, events=) at src/zookeeper.c:2295
#7 0x00007fb92fa2c41e in wait_for_close (zh=0x7fb928002aa0) at src/zookeeper.c:2515
#8 zookeeper_close (zh=0x7fb928002aa0) at src/zookeeper.c:2566
#9 0x00007fb92fa235d6 in connection_data_worker () from /home/deploy/scott/zk-lock/ext/zklock.so
...
```
I'm not entirely sure but I *believe* that either the mutex or the condvar for the lock have already been deleted, which causes the deadlock. My solution is to add an `exiting_` flag that will be checked before attempting to signal the condvar in `set_lock_state_and_signal`. It's also checked in `connection_data_worker` to early exit.

Because of Ruby `Test::Unit` fuckery (it executes tests in an `at_exit` handler), I had to register ZKLock's at_exit handler in `connection_initialize`, so it will be invoked after `Test::Unit`'s handler which kicks off the test run.