Throttling on arbitrary resources.

Implements throttling for arbitrary resources.
https://github.com/seomoz/qless-core/issues/27

Currently running in our production system with 14k worker processes spread across 15 redis processes.

Created a throttle object model in qless-core which consists of a active job set, a pending job set, and a maximum value. The throttle controls how many jobs that can run simultaneously.

Modified the put command to attempt to throttle new jobs. This is to prevent wasting pops to move jobs into the throttled state.

Modified the pop command to check if all the throttles for the job are available and if not to add it to pending queue for one of the throttles.

Modified the complete, fail, retry commands to release all throttles associated with the job.

When a throttle is released the throttle checks its pending job set and inserts up to (maximum - active) jobs into the waiting queue to be popped.

Let me know of any questions, changes that need to be made etc.


Interesting, I originally thought that the expiration value was being applied to each job (allowing it to only hold the throttle lock for that amount of time).  Instead, I'm seeing here that it only applies to the throttle as a whole, and the throttle is expired (deleted) from redis when the expiration time has elapsed.

Is the idea to support a throttle that you only want to be active for X minutes or hours?  And the expiration will take care of removing it after that period, without you having to manually do that?