fix flaky test EXPECT_EQ code, using EXPECT_FLOAT_NEAR per Jeff
fix flaky math functions, remove unnecessary instantiations.
Merge pull request #1264 from Yangqing/dev

Clean flaky code