Fixed rounding in TransitionFunction.

This fix is in a code path that was not used by the regular quad
extraction pipeline.
Added handling for Visual Studio regarding x87 extended internal precision.