backupify/resque — pull request #1 by jzaleski (2013-05-20 23:25:56+00:00)
Title: Use blocking pop when looking for work

Use blocking pop (blpop) when looking for work to do. This is a effectively a backport from resque/resque [master] (as this is now the default behavior).

We were seeing issues w/ idle workers hammering redis (at ~90K requests/sec -- assuming 13 worker machines in operation). This change will lower that average rate by two orders of magnitude to about 900 requests/sec by using the redis blpop method (http://redis.io/commands/blpop). As an added bonus, when the queues are empty and begin to fill again, when the scheduler runs, the jobs will be picked up a little bit faster (nice, but nothing to write home about).

Added new tests to check this functionality and verified via rake and in manual testing in staging.