Since the jobs are stateful (mainly around this method), it has to be reset before the test.  Running the entire suite without this causes failures because `@on_failure_retry_hook_already_called` gets set in some test before this (the test on its own passes fine).

Could make a new job class just for this test, but that doesn't really solve the problem.  Happy to do that if you'd prefer we not break encapsulation like this, though