Fix ulp test with respect to nan and inf