Better fix for ULP on NaN

Use a proper NaN-safe max and add a test for the scenario
in the max_ulp test.