check rotation matrix