tests: Fix tests on various compilers

Some compilers don't handle NaNs properly.
Some compilers don't implement fmod in a IEEE-compatible way.
Some compilers have exception handling codegen bugs (DMC...).