Allow retries to be templatable

Fixes #5865