Only specify -m32/-m64 for archs where the variants exist.

@redstar: This applies to PPC as well, right?