mperham/sidekiq — pull request #843 by bdurand (2013-04-11 00:49:25+00:00)
Title: More atomic poller operations

I recently had an issue where my application lost a very large number of jobs. The problem was with future scheduled jobs being popped from the scheduled queue but never pushed onto one of the worker queues. Here are the steps that happened:

1. We performed some redis maintenance which took the server down for a few minutes.
2. Due to a bug in scheduled.rb (since fixed see issue #309) the poller stopped querying the scheduled queue.
3. When redis came back online, sidekiq started processing all workers executed with perform_async. Any workers called with perform_in were put on the scheduled queue but never popped off due to the stopped poller.
4. Because sidekiq was processing some jobs just fine, monitoring did not pick up that there was a problem.
5. After almost 20 hours sidekiq got restarted. This started the poller again.
6. At this point there were over 2 million jobs in the scheduled queue.
7. The poller on one of the sidekiq processes would have tried to pop all 2 million jobs at once off the scheduled queue. It than would have tried to insert them one at a time on to the appropriate worker queues. Something happened in the middle of this that aborted the process some where between the pop and finishing the pushes. It could have been an out of memory error or it could have been monitoring thinking there was a run away process due to the ballooning memory.

The end result was that most of the jobs were irretrievably lost. This had also happened two weeks earlier during some other redis maintenance.

This pull request changes the logic in Poller to pop messages one at a time from the retry and schedule queues and immediately push them to the appropriate worker queue. The new logic is:

1. Get the next item in the queue if it's score (time to execute) is <= now
2. If an item exists, then try to remove the item from the queue
3. If item successfully removed, then add it to the appropriate worker queue
4. If item not successfully removed, then try again (it was likely removed by another process)
5. Repeat until no items have a score that matches <= now