Fix float precision issues and mixed usage of Math.sin, MathUtils.sin

The mixed usage of both the high-precision and the faster trig functions
can leads to very different results and problems due to the nature of the
faster methods.

Also, the MathUtils implementation produces some slightly different results
when, for example, rotating a matrix on an arbitrary axis.