YafaRay/Core — pull request #60 by DavidBluecame (2013-04-03 13:24:06+00:00)
Title: Fixes for Black and White Dots

For your information, I have found the cause of many of the black and white dots problems. I've tested this with the "horror gallery" scenes and it seems to work fine with both Linux 32 bits and Linux 64 bits

BLACK DOTS (NaN)
----------------
The problem is located in the acos() function used in several places of the program. The acos() is the arccosine and its valid domain is [-1.0, +1.0]

However, sometimes it receives values "slightly" outside the domain, as for example +1.000000000000000001

That causes the acos() function to generate an invalid NaN result that is propagated over all the math until you get the black NaN dots in the render.

I have created a custom fAcos() function which does a domain check before calling the actual acos(), ensuring that the values are within the domain. If a value is outside the domain, it's "forced" inside.

The underlying causes of the "values outside the domain" are quite complex and surprising. It also explains why this is different in Linux 32 bits and Linux 64 bits, for example.

Some of the causes:

- The compilation flags are "unsafe". This is good for speed, but many floating point operations are not guaranteed to be done correctly. However, we have to keep the unsafe flags or we would have a much slower yafaray, so we do the extra checks manually in yafaray.

- Linux x86 (32 bits) compiles by default by using the CPU x87 floating point operations. Those operations are not IEEE 754 compliant. In fact, the x87 floating point operations use ALWAYS 80 bits for the x87 internal registers. That's why even with float 32bits numbers you may get some "garbage" digits where you would not expect them. Doing the manual domain checks should mitigate this, or even resolve it.

- Linux x86_64 (64 bits) compiles by default by using the CPU SSE floating point operations and NOT the x87 for floating point operations. These are IEEE 754 complaint and their internal registers work as expected (32bits with floats and 64 bits with doubles). Therefore, the floating point operations may work differently in Linux with 64 bits than 32 bits causing the strange differences in the Horror Gallery. In any case, even using SSE you still have sometimes values outside the valid domain, so the manual domain checks should mitigate or resolve this.

WHITE DOTS (Inf or very big values)
-----------------------------------
* In the pdf1D class in sample_utils.h, sometimes one array index was -1 causing an access to array out of bounds and therefore causing eventually invalid values, NaNs, Infs, etc. This is the line causing the problem

int index = (int) (ptr-cdf-1);

So I have added this fix:

int index = (int) (ptr-cdf-1);
if(index<0) index=0; <-- Fix to make sure the index never goes to negative values outside the array.

However, this should be investigated in the future to see why the index goes to -1 in the first place.

* Other type of white dots problem happens when the CMake option FAST_MATH=ON

One of the reasons is that the math simplifications were written only for float 32bit numbers, but in the rest of the program they sometimes were used to calculate double 64bit numbers, resulting in bad calculations and white dots.

* Also, sometimes the fSin() function generated return values outside its valid range [-1.0,+1.0] also causing other white dots.

I have created the double 64bit versions of all optimizations. I have also added checks to make sure the fSin() return value is in the correct range.

However, I think that we need to refine a bit better the double 64bit versions of the optimizations, but it seems to work fine now anyway.

For your reference, it's possible to force the compilation to use x87 or SSE instructions. For example, in Linux 32 bits, you can get the same results you get in a Linux 64 bits by compiling with the gcc flags: -msse2 -mfpmath=sse

I suppose these fixes will not solve all the black/white dots problems, but I hope they help in most cases.

Best regards! David Bluecame.