Correct use of false method in g3dmath

In isNaN(float x) the method nan() is called. This causes an warning
that the converstion from double to float is not lossless. The correct
function that already exists is fnan() I canged that to prevent the
warning.
I was unable to locate the original branch for that code so I assume the
error (if not fixed) will be in the next versions to (if there are any).