Fix quanternion unittest for x64.
Problem was both with reduced precision and not testing for values near zero.