alphagov/whitehall — pull request #1310 by tekin (2014-03-10 17:07:38+00:00)
Title: Don't rescue all exceptions in edition services

This is dangerous as it can mask all sorts of issues with our edition workflow. It also doesn't handle exceptions in listeners very gracefully - if a given listener raises an exception, no further listeners will get notified and will be lost to the wind. Instead of rescuing all exceptions on all service objects, we now only rescue in the `ScheduledEditionPublisher`, which is where we are most concerned about failure to publish. We also log the exception to errbit so we know when it happens.

Further work should be done so that any listeners that register themselves for publishing notifications are able to handle common exceptions themselves. Once this is the case, the rescue should be removed altogether. Also, the current `EditionServiceCoordinator` should be enhanced to use a more robust notification system/message bus that is able to continue notifying all listeners if one should fail. It currently uses `ActiveSupport::Notifications::Fanout` which is pretty rudimentary.

I've currently got the code set up so that exceptions are only handled in the production env so that we see still see them in test and development mode. I'm on the fence about this. On the one hand, I think we should see when things are broken in our tests and in dev mode, but on the other hand, this will mean that any scheduled publishing performed in dev mode will fail without a working panopticon endpoint. Thoughts?