mokshaproject/moksha — pull request #16 by ralphbean (2014-06-27 03:30:50+00:00)
Title: Catch and log errors from consumers.

It used to be that if a consumer raised an error, that would bubble up
to Twisted's callback stuff and get logged, but then things would
proceed smoothly with the next message.

Now, with the threaded consumer stuff, if a consumer raised an uncaught
exception, it would break the consumer thread out of its ``_work()``
loop and cause it to hang dead. Messages would pile up. Most recently,
this happened to our irc bot a few nights in a row.

This wraps a try block around the hand-off in that ``_work()`` method so
that if a consumer hits an exceptional condition, we can recover and
keep processing.

For fedmsg, ``self.log.error`` and ``self.log.exception`` will send
emails to members of sysadmin-datanommer... so we get to know about
these things when they happen.