Fix numerical stability of rotation_matrix_to_axis_angle