build/core: Revert -fast-math optimizations

Change-Id: Iadecc42aed9eb4663bb9d95f00ef81bcb55fe36f