basic test for numeric precision