alphagov/email-alert-service — pull request #31 by rboulton (2014-11-24 08:14:51+00:00)
Title: Exit the process on uncaught exceptions

Before this change, we're seeing the consumers sometimes getting stuck;
they accept a message from Rabbit, but never ack it, so never get sent
another one to process. I believe this is most likely to be because
they trigger an exception somewhere, which doesn't get caught in the
processor's `process` method. Bunny then catches the exception in its
threadpool code, and handles it itself (but doesn't log anything).
Critically, bunny doesn't ack or nack the message (by design; there's no
requirement to ack or nack immediately).

I tested the old code's behaviour if an exception is raised from
`process`: the exception is swallowed, and no logging happens, which
matches the symptoms we see in production. I also investigated Bunny's
implementation, and this looks like expected behaviour.

The simplest fix for this is to exit the process if a `process` method
raises an uncaught exception. Since Bunny catches the exception, we
can't simply do this by setting the global "Thread.abort_on_exception"
property; instead we have to manually call exit().

I have implemented this fix, and improved logging to give us a chance to
see what exceptions are being thrown. In particular, we now log, and
send a message to errbit, any uncaught exception from a `process`
method, other than SignalException. We exit with an error status code
for any exception other than SignalException.

Story: https://www.pivotaltracker.com/story/show/83204672