Fix building on ARM by removing some unneeded x86-specific compiler flags