Revert "Always build all architectures"

This reverts commit fbc0ffcdac92de0ce14f46071985623a5ad18942.