If retries equals one, then we should retry exactly once, so we
actually try to use the socket twice.