alphagov/whitehall — pull request #402 by tekin (2013-03-28 12:58:04+00:00)
Title: Make scheduled publishing more robust

Tracker: https://www.pivotaltracker.com/story/show/46889987

The primary aim here is to stop scheduled editions from failing to be published at their allotted time. The problem occurs because the rake task that publishes due scheduled editions is run on *all* backend servers (for redundancy reasons), and thus we occasionally end up with MySQL deadlocks. This patches up the existing mechanism by allowing it to retry up to 5 times before giving up and reporting the failure.

The better solution would be to use a robust job queuing system that supports scheduled jobs and retries out-of-the-box (e.g. sidekiq or resque). We'd be able to simply queue a job to be run at correct time and let the job queue take care of the rest. With this in mind, the unit of work for publishing a scheduled edition has been written as a worker class that is compatible with the APIs of sidekiq and resque, which should make it easy to swap things out in the future if/when we switch to this model.