Swallow exceptions from connection destroy for now to catch the case
where it fails when the broker is down.