binarydud/pyres — Issue #85: BLPOP slows down processing
Author: joeshaw
Created: 2011-10-24 17:50:24+00:00
State: open

Using `BLPOP` when queues are empty is considerably slower than `LPOP` because the entire timeout interval must expire before the next queue is checked.

Let's say that you are checking `queue1`, `queue2`, and `queue3`, and there is one job in `queue3`. The others are empty. Right now, the code does something like this:

while True:
for q in queues:
job = redis.blpop(q, timeout=5)
if job: job.perform(...)

Since `queue1` is empty, it takes 5 seconds before the `BLPOP` times out. Ditto for `queue2`. The job is popped off `queue3` immediately and executed, then the loop runs. Total time for the 3 queues to be traversed: **10 seconds** plus the time it took the job to be performed. The second time, with `queue3` empty it will take **15 seconds**. You can generalize this to **interval time x number of empty queues**.

With `LPOP`, the code was structured like this:

while True:
for q in queues:
job = redis.lpop(q)
if job: job.perform(...)
time.sleep(5)

This meant that every time through the list of queues took **5 seconds**, plus the time for the worker, regardless of the number of queues.

This could be solved by running a `BLPOP` across all the queues simultaneously and rotating the order of the queues to prevent affinity. The downside to this is that `BLPOP` does not return which queue an item was popped off of. We would have to add this to the serialized job representation, which might cause compatibility issues between pyres versions (not to mention resque). If that's not an option, I think we should revert to the `LPOP` implementation. (FWIW, upstream resque still uses `LPOP`)