Add robust handling of RabbitMQ connections

I replace `.connection` method calls on the broker with kombu's own
retry method calls and wrapped them with appropriate exception handlers
and logging messages.

I also added a SystemExit handler to the core loop to control exit from
the loop and handle appropriate exceptions with logging.

When faced with a socket timeout we attempt to re-establish the
connection and will log and exit if that retry mechanism fails to
establish a new connection.

Fixes DHC-2262
I was going to suggest we put configuration in a separate PR

> On Nov 26, 2014, at 1:05 PM, Ryan Petrello <notifications@github.com> wrote:
> 
> In akanda/rug/notifications.py:
> 
> > @@ -151,7 +157,18 @@ def listen(host_id, amqp_url,
> >          virtual_host=conn_info.path,
> >          port=conn_info.port,
> >      )
> > -    connection.connect()
> > +    try:
> > +        connection.ensure_connection(
> > +            errback=_handle_connection_error,
> > +            max_retries=10,
> 
> I agree, though I feel like these are reasonable defaults and better than what we have now (no connection retries). I'm okay with us making a ticket and adding config options for this later (which also involves updating chef recipes, etc...)
> 
> —
> Reply to this email directly or view it on GitHub.
>