Revert "Always build for all architectures"

This reverts commit 89da441d65e300b261fb08ecb108e60f9641b1f7.