only check once if there is a fail low/high

This little tweak gives a nice speedup in default bench, at least on my laptop (should be less important at higher depth, but safe anyway).