Remove the retry options from the worker

in preperation for manual retries