Only specify -m32/-m64 for archs where the variants exist.