Revert "Always build all architectures"

This reverts commit ce184c6b5fb169a6cfae87f3829cf6ffe64d1b1a.