Only reconnect if we have any retries left.