redding/qs — pull request #37 by jcredding (2014-12-26 17:42:42+00:00)
Title: Add redis connection and worker pool logic to `Daemon`

This adds the logic for a `Daemon` to build a redis connection and
worker pool and use them to process jobs. This will allow a daemon
to pull jobs off of redis lists and add them to its worker pool.
The worker pool can then build and run payload handlers as workers
are available.

The daemon now builds a redis connection and a worker pool when
started. It also clears the redis list for handling signals so it
doesn't react to stale signals. Once a daemon beings its work loop
it will check to make sure a worker is available before pulling
any work out of redis. This is to avoid emptying redis into
memory and having to manage that when restarting. The daemon will
only pull jobs off of redis as it has workers to handle them.

The daemon now also properly waits for its worker pool's queue to
empty when its stopped without a shutdown timeout. Previously, it
would repeatedly sleep for 5 seconds which works but causes a
slower shutdown as it will always wait for 5 seconds. Now it will
immediately wakeup once its worker pool's queue is empty and
finish shutting down.

Finally, this also required updating the stop/halt logic to be able
to wake up the daemon thread from different states. There are 3
states that can cause the daemon thread to sleep: waiting for a
worker, waiting for work or waiting for its queue to empty when
shutdown. The first 2 occur while the daemon is doing its normal
processing of jobs. The last one only occurs when shutting down and
only allows halting after stopping, which will force the daemon to
immediately exit.

@kellyredding - Ready for review.