Revert "Fixup armv8-a building, and make multiarch builds work"

This reverts commit b9c750dbf8d30c2dbe71871892eab7f495372548.