Ensure that socket becomes closed when no more retries left