Fix method call in g3dmath + typo

In isNaN(float x) the method nan() is called. This causes an warning
that the converstion from double to float is not lossless. The correct
function that already exists is fnan() I canged that to prevent the
warning.
I was unable to locate the original branch for that code so I assume the
error (if not fixed) will be in the next versions to (if there are any).