Fix floating-point in affine transformation on x86 vs v86-64
Merge pull request #1726 from mdboom/affine_transform_math

stackplot_test_baseline has different results on 32-bit and 64-bit platforms