test for rotation matrix