fix flaky test EXPECT_EQ code, using EXPECT_FLOAT_NEAR per Jeff
fix flaky math functions, remove unnecessary instantiations.