nvie/rq — pull request #336 by negz (2014-04-22 04:34:09+00:00)
Title: Allow a queue to remember all extant jobs that have passed through it.

Disclaimer:
I'm not entirely convinced this is a good idea, so consider this an RFC. :)

Motivation:
I'm considering using RQ to broker server fleet management jobs - power cycles, installs, etc. These jobs would complete in ~minutes to ~an hour depending on the type. I'd like to have a queue for each job type, with a TTL of ~48 hours. This functionality allows me to easily expose recently completed jobs on a per-queue basis in some pretty interface without resorting to `KEYS *`. Basically a poor-man's best-effort result store.

Changes:
* Setting `remember_past_jobs=True` on a `Queue` will cause `q.jobs` to return any extant job that has passed through that queue, in addition to those currently queued. This is implemented using a Redis set for each queue, so enqueing a job requires an extra `SADD`. `q.count` remains unchanged - it still counts jobs waiting in the queue.
* Reading `q.jobs` causes a garbage collection of sorts when `remember_past_jobs=True`, which I hope is made faster by Lua, because Lua is terrible and scary.
* `q.get_jobs()` pagination is ignored when `remember_past_jobs=True`, since it uses a set.
* `q.remove(job)` now deletes the Job's Redis hash when it removes a job from the queue. I also considered setting the job status to a new `Status.ORPHANED`, since AFAICT jobs that have been removed from their queue will sit around waiting to be re-added to a queue?