Add custom retry delay

Allow custom retries to be defined per-worker as discussed in #1013

@mperham how does this look?