Remove flags that could interfere with -march=native