Task retry manager with a certain back off time.