Cleanup listener and driver when they die arbitrarily.