Enforce minimum value of rtol

This prevents a possible infinite loop.