[math] fix resolution in ltp_def rotation matrix