Remove --ffast-math for all builds

Due to a bug in anaconda's libm support for linux, fast-math is
unusable.  And I don't want to try to hack a way to decide if it's
usable on things other than linux, because it's just one more thing to
break.