Add ability to cancel retries