Floating point number comparison fix for testsuite

Both commits fix approximate equality checks for floating point numbers.

The first one converts approximate a <= b check into approximate equality check that is probably intended there in the same manner as in several other files in the testsuite, so the case when the return values significantly differ and the first one is less than the second one is handled as test failure. The `fabs()` usage requires a flag such as `-lm` but it seems that it is already in use there. `fabsl()` usage requires C99, I hope it is not a problem.

The second commit replaces the call to integer `abs()` (which is probably a typo in a floating point context) by a `fabs()` call.