Abandon connect loop if something has called Close()