computing coordinate frame error between rotation matrices