Fix test for transformation.

Set tolerance for error in determinant of a 3D rotation matrix to 1e-10.