adjust test for machine numerical precision difference