BF: Make orthogonality test for rotation matrix less stringent

Fixes issue with some data reported in dcmstack: https://github.com/moloney/dcmstack/issues/10