Bail as soon as possible if we aren't supposed to retry