coleifer/huey — pull request #20 by collinwat (2013-03-27 17:41:46+00:00)
Title: Disable worker thread

This issue is more informational than an actual bug fix. I was hoping to get some feedback and/or support to see if what I'm doing makes sense.

Basically, I ran into a problem during a restart of my consumer process. When the consumer has delegated work to the queue and the worker_pool has delegated the task to a worker thread, it appears that the work is lost when the shutdown method is called.

I've added the `synchronize` example to test.

Without the fix (8a3c255) you can reproduce the issue by starting `consume.py` and `produce.py` in different terminals. While consuming, Ctrl-C the `consume.py` process to stop work. Then, restart the process.

You should observe that the `produce.py` process does not get a result and raises a `huey.exceptions.DataStoreTimeout`. Also, the `consume.py` logs should show that not all work is complete. There should be 0-10 test queries to yahoo and 0-10 test queries to google but depending on where you stopped the consumer, the request is not made.

With a change (45b4d9a) and following the same procedure as above, you should see that all work is completed and the `produce.py` process completes normally.

The change adds an option to disable thread workers. This means that work is completed within the worker_pool thread and that `self._pool` is acquired after `shutdown` is called.

I'm not sure if this is the best approach but it solves the issue I'm having.

Thoughts, ideas, changes?