mperham/sidekiq — pull request #264 by wjossey (2012-06-20 22:42:05+00:00)
Title: Support Pushing in Batches

Hi Mike,
In an effort to make sidekiq a bit faster on the push side, I've added the ability for coders to pass in a bulk array of arguments which leverages the fact that rpush can take a list of values and place them on the tail of the list (or queue, in this instance).

An example use case would be a user wanting to send a message to 100 of their friends. Rather than calling perform_async 100 times with only slightly different messages (in this case, just the target user would change), we can call perform_bulk_async once with an array of messages, and save ourselves a bunch of round trips over the wire.

Some issues I already see and would like your feedback on how you'd like me to proceed.
1) Scheduling won't work with this model, because zadd doesn't take a list of values. We can call zadd multiple times; however, that causes issues with validating that all jobs were properly enqueued.
2) It's unclear to me whether UniqueJobs should be updated. If we're trying to prevent identical bulk-requests from processing, then it's fine in its current state. If we'd like each individual job within that bulk message to be unique, then it'll need to be tweaked.

Thanks for all your hard work on Sidekiq. It's an awesome tool that I'm already using all over the place.

-Wes