arcball rotation, rotation matrix