CE/improve runs

Should we pull this out as a constant and call it `retryTolerance` or something like that?