Gracefully handle consumer cancel notifications

With mirrored queues and clustered rabbit nodes a queue is still
mastered by a single rabbit node. When the rabbit node dies an
election occurs amongst the remaining nodes and a new master is
elected. When a slave is promoted to master it will close all the
open channels to its consumers but it will not close the
connections. This is reported to consumers as a consumer cancel
notification (CCN). Consumers need to re-subscribe to these queues
when they recieve a CCN.

kombu 2.1.4+ reports CCNs as channel errors. This patch updates
the ensure function to be more inline with the upstream kombu
functionality. We now monitor for channel errors as well as
connection errors and initiate a reconnect if we detect an error.

Change-Id: Ie00f67e65250dc983fa45877c14091ad4ae136b4
Partial-Bug: 856764
Slow down Kombu reconnect attempts

For a rationale for this patch, see the discussion surrounding Bug

When reconnecting to a RabbitMQ cluster with mirrored queues in
use, the attempt to release the connection can hang "indefinitely"
somewhere deep down in Kombu.  Blocking the thread for a bit
prior to release seems to kludge around the problem where it is
otherwise reproduceable.

DocImpact

Change-Id: Ic2ede3046709b831adf8204e4c909c589c1786c4
Partial-Bug: 856764
Merge "Gracefully handle consumer cancel notifications"
Merge "Slow down Kombu reconnect attempts"