Relax rotation threshold

Relax threshold for orthogonality of rotation matrix.

I propose this pull request to replace https://github.com/nipy/nibabel/pull/157 and https://github.com/nipy/nibabel/pull/156