Fix race on thread shutdown causing deadlock

This patch and the next are somewhat a revert of 318e9f2, but the previous
fix didn't quite close the race. This only happens when we create threads
for a backstore that turns out to be invalid, which we then tear down.

See https://bugzilla.redhat.com/show_bug.cgi?id=848585 .

This is occurring because there's still a window where a thread misses
seeing info->stop == 1 but is not yet in cond_wait so it misses the
broadcast:

thread_close:              thread_worker_fn:
                           info->stop is seen as 0
info->stop = 1
pthread_cond_broadcast     -- misses broadcast
                           pthread_cond_wait
pthread_join (hangs)

I believe the solution is to go back to using pthread_cancel. We can call
it before pthread_cond_wait is called (or after) and it will do the right
thing: pop out and exit. The only tricky bit is we need to use the
pthread_cleanup_push mechanism to properly release info->pending_lock.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Remove startup_lock

The purpose of the startup lock was to hold threads at the startup_lock
so if a thread creation fails, it's still possible to have them
end themselves by looking at info->stop. Since we are back to using
pthread_cancel instead of info->stop, we don't need this any more.
The only wrinkle is we need to only send pthread_cancels to
successfully-created threads, so now check this in the destroy_threads
reverse-loop.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Replace if()/goto with while()

Is equivalent and nicer-looking.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>