YafaRay/Core — pull request #58 by DavidBluecame (2013-04-02 10:33:26+00:00)
Title: Fixes for Black Dots

For your information, I have found the cause of many of the black dots problems. I've tested this with the "horror gallery" scenes and it seems to work fine with both Linux 32 bits and Linux 64 bits

The problem is located in the acos() function. The acos() is the arccosine and its valid domain is [-1.0, +1.0]

However, sometimes it receives values "slightly" outside the domain, as for example +1.000000000000000001

That causes the acos() function to generate an invalid NaN result that is propagated over all the math until you get the black NaN dots in the render.

The solution is to create a custom fAcos() function which does a domain check before calling the actual acos(), ensuring that the values are within the domain. If a value is outside the domain, it's "forced" inside.

The underlying causes of the "values outside the domain" are quite complex and surprising. It also explains why this is different in Linux 32 bits and Linux 64 bits, for example.

Causes:
* The compilation flags are "unsafe". This is good for speed, but many floating point operations are not guaranteed to be done correctly. However, we have to keep the unsafe flags or we would have a much slower yafaray, so we do the extra checks manually in yafaray.

* Linux x86 (32 bits) compiles by default by using the CPU x87 floating point operations. Those operations are not IEEE 754 compliant. In fact, the x87 floating point operations use ALWAYS 80 bits for the x87 internal registers. That's why even with float 32bits numbers you may get some "garbage" digits where you would not expect them. Doing the manual domain checks should mitigate this, or even resolve it.

* Linux x86_64 (64 bits) compiles by default by using the CPU SSE floating point operations and NOT the x87 for floating point operations. These are IEEE 754 complaint and their internal registers work as expected (32bits with floats and 64 bits with doubles). Therefore, the floating point operations may work differently in Linux with 64 bits than 32 bits causing the strange differences in the Horror Gallery. In any case, even using SSE you still have sometimes values outside the valid domain, so the manual domain checks should mitigate or resolve this.

It's possible to force the compilation to use x87 or SSE instructions. For example, in Linux 32 bits, you can get the same results you get in a Linux 64 bits by compiling with the gcc flags: -msse2 -mfpmath=sse

I hope this helps. Best regards!!!