defunkt/resque — pull request #731 by tarcieri (2012-10-23 00:36:23+00:00)
Title: Multiple failure queues

At LivingSocial we deal with lots of queues and lots of failures. To that end we've had an internal branch of Resque which contains a number of UI changes which make dealing with failures easier. One of the big things we added was support for multiple failed queues which makes it easier to drill down on particular errors.

This adds a new Resque::Failure::RedisMultiQueue backend (while still using Resque::Failure::Redis by default for backwards compatibility). It also adds a FAILURE_BACKEND environment variable which can be used to select the backend.

Resque::Failure::RedisMultiQueue works like Resque::Failure::Redis except instead of lumping everything into a single "failed" queue, it uses a separate queue (i.e. Redis list) for each Resque queue. This makes it easier to browse, query, and otherwise compartmentalize failures that happen in different parts of the system. We regularly encounter runaway jobs flooding the "failed" queue with tens of thousands of failed jobs (or more) which renders the resque-web UI broken. By breaking the failed queues apart into separate lists, we can ensure that such events only impact one failed queue instead of breaking resque-web for everyone.

Screenshots
-----------------
![failed overview](http://i.imgur.com/LkhxD.png)
![individual queue](http://i.imgur.com/tsZIi.png)
![overview](http://i.imgur.com/Hkgml.png)