Make sure that closed connections are properly cleaned up

Otherwise we can't connect to them again...