Fix rotation matrix decomposition in J3DIMath

The rotation decomposition in J3DIMath used incorrect formulas that only worked for special cases and were sensitive to spec-compliant differences in trigonometric function approximation in different browsers. This made the MatrixTest demo flicker noticeably in recent versions of Google Chrome.

Some issues might still remain in J3DIMath code, particularly comments on row/column majorness. The code seems somewhat confused about what are rows and what are columns in a matrix, fixing this properly would require a larger overhaul.