holoirc: don't remove logging manager until we know we're totally disconnected