BUILD: Remove simulator compiler flags

Not sure the need for these and they make the binaries very large.