Revert "Always build all architectures"

This reverts commit f63c40abe84391e345a2bb5f6217bb30f6a4d7c8.