math: fixed rotation matrix singularity bug