Avoid some numeric issues when computing rotation error.