Repository: chainguard-dev/apko
Pull request #56 (2022-03-04 20:17:10)

Multiarch support
`apko publish` produces a multiarch image index. By default, this includes six common archs, which can be overridden with `archs:` in the image config YAML, or values in the `--arch` flag (in increasing order of preference).

An attempt is made to accept both apk-style arch strings ("x86_86") and OCI-style arch strings ("amd64"), but more could probably be done. There's a TODO to support armv7 and armhf (arm/v7 and arm/v6 respectively, in OCI terms).

apk keys are extracted from arch-specific locations (e.g., `/usr/share/apk/keys/s390x/*`) and placed in the standard location (`/etc/apk/keys/*`).

`apko build` and `apko build-minirootfs` both only produce a single arch's tarball, for the current runtime arch. If the image config specifies archs in this case, a warning is logged to avoid confusion.

kaniini: In Alpine parlance, I think all platforms would be:

- x86
- x86_64
- aarch64
- armv7
- ppc64le
- s390x
- riscv64

But other distributions support more or less platforms.  However, I think most will be using this with Alpine-based distributions for now, so defaulting to those 7 seems reasonable.

Maybe @rvs has some input here?  I think EVE is the biggest consumer of multiarch Alpine images.
imjasonh: Hmm, this seems to reliably fail when building `examples/nginx.yaml` and `nginx-rootless.yaml`:

```
ERROR: unable to select packages:
  /bin/sh (virtual):
    provided by: busybox
    required by: nginx-1.20.2-r0[/bin/sh]
                 s6-2.11.0.1-r0[/bin/sh]
                 alpine-baselayout-3.2.0-r20[/bin/sh]
Error: failed to build layer image: failed to fixate apk world: failed to run /sbin/apk fix --root /tmp/apko-2109078717 --no-cache --update-cache --arch aarch64: exit status 4
```

If I move `amd64` to the end of `AllArchs`, it works for `amd64`/`aarch64`, but then fails for whatever's now second. Is there something else I have to do to sequester these builds from each other?
kaniini: > Hmm, this seems to reliably fail when building `examples/nginx.yaml` and `nginx-rootless.yaml`:
> 
> ```
> ERROR: unable to select packages:
>   /bin/sh (virtual):
>     provided by: busybox
>     required by: nginx-1.20.2-r0[/bin/sh]
>                  s6-2.11.0.1-r0[/bin/sh]
>                  alpine-baselayout-3.2.0-r20[/bin/sh]
> Error: failed to build layer image: failed to fixate apk world: failed to run /sbin/apk fix --root /tmp/apko-2109078717 --no-cache --update-cache --arch aarch64: exit status 4
> ```
> 
> If I move `amd64` to the end of `AllArchs`, it works for `amd64`/`aarch64`, but then fails for whatever's now second. Is there something else I have to do to sequester these builds from each other?

Each of these need to be built in their own roots, but I'm not seeing any code to loop over the arch list and build images individually.
imjasonh: > Each of these need to be built in their own roots, but I'm not seeing any code to loop over the arch list and build images individually.

Thanks, that was exactly it!

The code to iterate archs and build arch-specific images is hereish: https://github.com/chainguard-dev/apko/pull/56/files#diff-dd36f582aabe1c6547458d485dd6cc334bb6cdeba1de577a1d0670007efba268R109

The problem was I was passing through the same workDir for all archs 💥 
imjasonh: Hmm, this one I can't seem to reproduce myself. 🤔 

I can run `apko publish examples/nginx.yaml imjasonh/apko-multi` for all the archs just fine: https://gist.github.com/imjasonh/5c74e01862b1ce30a3d3430a8a58beb5

In both cases each `apk fix` happens in a different root, e.g., `working directory: /tmp/apko-2357205812/aarch64`
kaniini: The tekton task needs to be updated to use proot.  In the meantime, we should restrict it to `--arch=x86_64`.
imjasonh: > The tekton task needs to be updated to use proot. In the meantime, we should restrict it to `--arch=x86_64`.

This got CI to pass at least 🎉 

> Also, the proot command line needs to add the `-q /usr/bin/qemu-$ARCH` option when emulation is necessary.

I started down this rabbithole but I kinda think I should stop and do it in another change. The default `alpine:latest` base image doesn't seem to include `proot` by default, so there's more to do there, and this change is already getting kinda big.
kaniini: > I started down this rabbithole but I kinda think I should stop and do it in another change. The default alpine:latest base image doesn't seem to include proot by default, so there's more to do there, and this change is already getting kinda big.

Agreed!  Lets follow it up outside of this MR :)
kaniini: Looks okay except for the two things I pointed out and also the question about s390x root being used for aarch64 arch.

Patch:
diff --git a/config/task.yaml b/config/task.yaml
index 272a0b1..25546e8 100644
--- a/config/task.yaml
+++ b/config/task.yaml
@@ -44,6 +44,7 @@ spec:
       args:
       - publish
       - --image-refs=/tekton/results/IMAGES
+      - --arch=x86_64
       - $(params["path"])
       - $(params["dev.mink.images.target"])
 
diff --git a/examples/alpine-base-rootless.yaml b/examples/alpine-base-rootless.yaml
index 5d15d33..3d302d9 100644
--- a/examples/alpine-base-rootless.yaml
+++ b/examples/alpine-base-rootless.yaml
@@ -1,12 +1,6 @@
 contents:
   repositories:
     - https://dl-cdn.alpinelinux.org/alpine/edge/main
-  keyring:
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-61666e3f.rsa.pub
   packages:
     - alpine-base
 
diff --git a/examples/alpine-base.yaml b/examples/alpine-base.yaml
index fd14187..bd83425 100644
--- a/examples/alpine-base.yaml
+++ b/examples/alpine-base.yaml
@@ -1,12 +1,6 @@
 contents:
   repositories:
     - https://dl-cdn.alpinelinux.org/alpine/edge/main
-  keyring:
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-61666e3f.rsa.pub
   packages:
     - alpine-base
 
diff --git a/examples/nginx.yaml b/examples/nginx.yaml
index 70199ea..141bd6d 100644
--- a/examples/nginx.yaml
+++ b/examples/nginx.yaml
@@ -1,12 +1,6 @@
 contents:
   repositories:
     - https://dl-cdn.alpinelinux.org/alpine/edge/main
-  keyring:
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
-    - /etc/apk/keys/alpine-devel@lists.alpinelinux.org-61666e3f.rsa.pub
   packages:
     - alpine-baselayout
     - nginx
diff --git a/go.mod b/go.mod
index 1b5877b..e84de14 100644
--- a/go.mod
+++ b/go.mod
@@ -6,6 +6,7 @@ require (
 	github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20220216180153-3d7835abdf40
 	github.com/chrismellard/docker-credential-acr-env v0.0.0-20220119192733-fe33c00cee21
 	github.com/dominodatalab/os-release v0.0.0-20190522011736-bcdb4a3e3c2f
+	github.com/google/go-cmp v0.5.7
 	github.com/google/go-containerregistry v0.8.1-0.20220223122423-dd8d514a9b24
 	github.com/hashicorp/go-multierror v1.1.1
 	github.com/maxbrunsfeld/counterfeiter/v6 v6.4.1
diff --git a/pkg/build/apk.go b/pkg/build/apk.go
index efda6f6..fdd3f41 100644
--- a/pkg/build/apk.go
+++ b/pkg/build/apk.go
@@ -30,8 +30,6 @@ import (
 	"golang.org/x/sync/errgroup"
 )
 
-var systemKeyringLocations = []string{"/etc/apk/keys/"}
-
 // Programmatic wrapper around apk-tools.  For now, this is done with os.Exec(),
 // but this has been designed so that we can port it easily to use libapk-go once
 // it is ready.
@@ -42,16 +40,18 @@ var systemKeyringLocations = []string{"/etc/apk/keys/"}
 func (bc *Context) InitApkDB() error {
 	log.Printf("initializing apk database")
 
-	return bc.Execute("apk", "add", "--initdb", "--root", bc.WorkDir)
+	return bc.Execute("apk", "add", "--initdb", "--arch", bc.Arch.ToAPK(), "--root", bc.WorkDir)
 }
 
 // loadSystemKeyring returns the keys found in the system keyring
 // directory by trying some common locations. These can be overridden
 // by passing one or more directories as arguments.
-func (*Context) loadSystemKeyring(locations ...string) ([]string, error) {
+func (bc *Context) loadSystemKeyring(locations ...string) ([]string, error) {
 	var ring []string
 	if len(locations) == 0 {
-		locations = systemKeyringLocations
+		locations = []string{
+			filepath.Join("/usr/share/apk/keys/", bc.Arch.ToAPK()),
+		}
 	}
 	for _, d := range locations {
 		keyFiles, err := os.ReadDir(d)
@@ -75,9 +75,9 @@ func (*Context) loadSystemKeyring(locations ...string) ([]string, error) {
 				log.Printf("%s has invalid extension (%s), skipping...", p, ext)
 			}
 		}
-		if len(ring) > 0 {
-			return ring, nil
-		}
+	}
+	if len(ring) > 0 {
+		return ring, nil
 	}
 	// Return an error since reading the system keyring is the last resort
 	return nil, errors.New("no suitable keyring directory found")
@@ -144,7 +144,7 @@ func (bc *Context) InitApkKeyring() (err error) {
 			}
 
 			// #nosec G306 -- apk keyring must be publicly readable
-			if err := os.WriteFile(filepath.Join(bc.WorkDir, element), data,
+			if err := os.WriteFile(filepath.Join(bc.WorkDir, "etc/apk/keys", filepath.Base(element)), data,
 				0644); err != nil {
 				return fmt.Errorf("failed to write apk key: %w", err)
 			}
@@ -190,7 +190,7 @@ func (bc *Context) InitApkWorld() error {
 func (bc *Context) FixateApkWorld() error {
 	log.Printf("synchronizing with desired apk world")
 
-	args := []string{"fix", "--root", bc.WorkDir, "--no-cache", "--update-cache"}
+	args := []string{"fix", "--root", bc.WorkDir, "--no-cache", "--update-cache", "--arch", bc.Arch.ToAPK()}
 	if bc.UseProot {
 		args = append(args, "--no-scripts")
 	}
diff --git a/pkg/build/build.go b/pkg/build/build.go
index 1695d5d..0e6074b 100644
--- a/pkg/build/build.go
+++ b/pkg/build/build.go
@@ -35,6 +35,7 @@ type Context struct {
 	Assertions         []Assertion
 	WantSBOM           bool
 	SBOMPath           string
+	Arch               types.Architecture
 }
 
 func (bc *Context) Summarize() {
@@ -44,6 +45,7 @@ func (bc *Context) Summarize() {
 	log.Printf("  use proot: %t", bc.UseProot)
 	log.Printf("  source date: %s", bc.SourceDateEpoch)
 	log.Printf("  SBOM output path: %s", bc.SBOMPath)
+	log.Printf("  arch: %v", bc.Arch.ToAPK())
 	bc.ImageConfiguration.Summarize()
 }
 
@@ -215,3 +217,11 @@ func WithImageConfiguration(ic types.ImageConfiguration) Option {
 		return nil
 	}
 }
+
+// WithArch sets the architecture for the build context.
+func WithArch(arch types.Architecture) Option {
+	return func(bc *Context) error {
+		bc.Arch = arch
+		return nil
+	}
+}
diff --git a/pkg/build/oci/oci.go b/pkg/build/oci/oci.go
index 853eee8..4ff48a2 100644
--- a/pkg/build/oci/oci.go
+++ b/pkg/build/oci/oci.go
@@ -18,7 +18,6 @@ import (
 	"fmt"
 	"io/ioutil"
 	"log"
-	"runtime"
 	"time"
 
 	"chainguard.dev/apko/pkg/build/types"
@@ -35,7 +34,15 @@ import (
 	v1tar "github.com/google/go-containerregistry/pkg/v1/tarball"
 )
 
-func buildImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, created time.Time) (v1.Image, error) {
+var keychain = authn.NewMultiKeychain(
+	authn.DefaultKeychain,
+	google.Keychain,
+	authn.NewKeychainFromHelper(ecr.NewECRHelper(ecr.WithLogOutput(ioutil.Discard))),
+	authn.NewKeychainFromHelper(credhelper.NewACRCredentialsHelper()),
+	github.Keychain,
+)
+
+func buildImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, created time.Time, arch types.Architecture) (v1.Image, error) {
 	log.Printf("building OCI image from layer '%s'", layerTarGZ)
 
 	v1Layer, err := v1tar.LayerFromFile(layerTarGZ)
@@ -79,8 +86,8 @@ func buildImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, created
 
 	cfg = cfg.DeepCopy()
 	cfg.Author = "github.com/chainguard-dev/apko"
-	cfg.Architecture = runtime.GOARCH
-	cfg.OS = runtime.GOOS
+	cfg.Architecture = string(arch)
+	cfg.OS = "linux"
 
 	if ic.Entrypoint.Command != "" {
 		cfg.Config.Entrypoint = []string{"/bin/sh", "-c", ic.Entrypoint.Command}
@@ -100,8 +107,8 @@ func buildImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, created
 	return v1Image, nil
 }
 
-func BuildImageTarballFromLayer(imageRef string, layerTarGZ string, outputTarGZ string, ic types.ImageConfiguration, created time.Time) error {
-	v1Image, err := buildImageFromLayer(layerTarGZ, ic, created)
+func BuildImageTarballFromLayer(imageRef string, layerTarGZ string, outputTarGZ string, ic types.ImageConfiguration, created time.Time, arch types.Architecture) error {
+	v1Image, err := buildImageFromLayer(layerTarGZ, ic, created, arch)
 	if err != nil {
 		return err
 	}
@@ -119,41 +126,88 @@ func BuildImageTarballFromLayer(imageRef string, layerTarGZ string, outputTarGZ
 	return nil
 }
 
-func publishTagFromImage(image v1.Image, imageRef string, hash v1.Hash, kc authn.Keychain) (name.Digest, error) {
+func publishTagFromImage(image v1.Image, imageRef string, hash v1.Hash) (name.Digest, error) {
 	imgRef, err := name.ParseReference(imageRef)
 	if err != nil {
 		return name.Digest{}, fmt.Errorf("unable to parse reference: %w", err)
 	}
 
-	if err := remote.Write(imgRef, image, remote.WithAuthFromKeychain(kc)); err != nil {
+	if err := remote.Write(imgRef, image, remote.WithAuthFromKeychain(keychain)); err != nil {
 		return name.Digest{}, fmt.Errorf("failed to publish: %w", err)
 	}
 	return imgRef.Context().Digest(hash.String()), nil
 }
 
-func PublishImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, created time.Time, tags ...string) (name.Digest, error) {
-	v1Image, err := buildImageFromLayer(layerTarGZ, ic, created)
+func PublishImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, created time.Time, arch types.Architecture, tags ...string) (name.Digest, v1.Image, error) {
+	v1Image, err := buildImageFromLayer(layerTarGZ, ic, created, arch)
 	if err != nil {
-		return name.Digest{}, err
+		return name.Digest{}, nil, err
 	}
 
 	h, err := v1Image.Digest()
 	if err != nil {
-		return name.Digest{}, fmt.Errorf("failed to compute digest: %w", err)
+		return name.Digest{}, nil, fmt.Errorf("failed to compute digest: %w", err)
+	}
+
+	digest := name.Digest{}
+	for _, tag := range tags {
+		log.Printf("publishing tag %v", tag)
+		digest, err = publishTagFromImage(v1Image, tag, h)
+		if err != nil {
+			return name.Digest{}, nil, err
+		}
 	}
 
-	kc := authn.NewMultiKeychain(
-		authn.DefaultKeychain,
-		google.Keychain,
-		authn.NewKeychainFromHelper(ecr.NewECRHelper(ecr.WithLogOutput(ioutil.Discard))),
-		authn.NewKeychainFromHelper(credhelper.NewACRCredentialsHelper()),
-		github.Keychain,
-	)
+	return digest, v1Image, nil
+}
+
+func PublishIndex(imgs []v1.Image, tags ...string) (name.Digest, error) {
+	var idx v1.ImageIndex = empty.Index
+	for _, img := range imgs {
+		mt, err := img.MediaType()
+		if err != nil {
+			return name.Digest{}, fmt.Errorf("failed to get mediatype: %w", err)
+		}
+
+		h, err := img.Digest()
+		if err != nil {
+			return name.Digest{}, fmt.Errorf("failed to compute digest: %w", err)
+		}
+
+		size, err := img.Size()
+		if err != nil {
+			return name.Digest{}, fmt.Errorf("failed to compute size: %w", err)
+		}
+
+		cfg, err := img.ConfigFile()
+		if err != nil {
+			return name.Digest{}, fmt.Errorf("failed to get config file: %w", err)
+		}
+		plat := v1.Platform{
+			OS:           cfg.OS,
+			Architecture: cfg.Architecture,
+		}
+
+		idx = mutate.AppendManifests(idx, mutate.IndexAddendum{
+			Add: img,
+			Descriptor: v1.Descriptor{
+				MediaType: mt,
+				Digest:    h,
+				Size:      size,
+				Platform:  &plat,
+			},
+		})
+	}
+
+	h, err := idx.Digest()
+	if err != nil {
+		return name.Digest{}, err
+	}
 
 	digest := name.Digest{}
 	for _, tag := range tags {
 		log.Printf("publishing tag %v", tag)
-		digest, err = publishTagFromImage(v1Image, tag, h, kc)
+		digest, err = publishTagFromIndex(idx, tag, h)
 		if err != nil {
 			return name.Digest{}, err
 		}
@@ -161,3 +215,16 @@ func PublishImageFromLayer(layerTarGZ string, ic types.ImageConfiguration, creat
 
 	return digest, nil
 }
+
+func publishTagFromIndex(index v1.ImageIndex, imageRef string, hash v1.Hash) (name.Digest, error) {
+	ref, err := name.ParseReference(imageRef)
+	if err != nil {
+		return name.Digest{}, fmt.Errorf("unable to parse reference: %w", err)
+	}
+
+	err = remote.WriteIndex(ref, index, remote.WithAuthFromKeychain(keychain))
+	if err != nil {
+		return name.Digest{}, fmt.Errorf("failed to publish: %w", err)
+	}
+	return ref.Context().Digest(hash.String()), nil
+}
diff --git a/pkg/build/types/types.go b/pkg/build/types/types.go
index 7e299d6..8640a3f 100644
--- a/pkg/build/types/types.go
+++ b/pkg/build/types/types.go
@@ -14,6 +14,8 @@
 
 package types
 
+import "sort"
+
 type User struct {
 	UserName string
 	UID      uint32
@@ -44,4 +46,66 @@ type ImageConfiguration struct {
 		Users  []User
 		Groups []Group
 	}
+	Archs []Architecture
+}
+
+// Architecture represents a CPU architecture for the container image.
+type Architecture string
+
+const (
+	amd64   Architecture = "amd64"
+	arm64   Architecture = "arm64"
+	ppc64le Architecture = "ppc64le"
+	s390x   Architecture = "s390x"
+	_386    Architecture = "386"
+	riscv64 Architecture = "riscv64"
+	// TODO: armv7 and armhf (av6)
+)
+
+// AllArchs contains the standard set of supported architectures, which are
+// used by `apko publish` when no architectures are specified.
+var AllArchs = []Architecture{amd64, arm64, ppc64le, s390x, _386, riscv64}
+
+// ToAPK returns the apk-style equivalent string for the Architecture.
+func (a Architecture) ToAPK() string {
+	switch a {
+	case _386:
+		return "x86"
+	case amd64:
+		return "x86_64"
+	case arm64:
+		return "aarch64"
+	default:
+		return string(a)
+	}
+}
+
+// ParseArchitectures parses architecture values in string form, and returns
+// the equivalent slice of Architectures.
+//
+// apk-style arch strings (e.g., "x86_64") are converted to the OCI-style
+// equivalent ("amd64"). Values are deduped, and the resulting slice is sorted
+// for reproducibility.
+func ParseArchitectures(in []string) []Architecture {
+	uniq := map[Architecture]struct{}{}
+	for _, s := range in {
+		a := Architecture(s)
+		switch s {
+		case "x86":
+			a = _386
+		case "x86_64":
+			a = amd64
+		case "aarch64":
+			a = arm64
+		}
+		uniq[a] = struct{}{}
+	}
+	archs := make([]Architecture, 0, len(uniq))
+	for k := range uniq {
+		archs = append(archs, k)
+	}
+	sort.Slice(archs, func(i, j int) bool {
+		return archs[i] < archs[j]
+	})
+	return archs
 }
diff --git a/pkg/cli/build-minirootfs.go b/pkg/cli/build-minirootfs.go
index a46683f..1b7bd53 100644
--- a/pkg/cli/build-minirootfs.go
+++ b/pkg/cli/build-minirootfs.go
@@ -19,8 +19,10 @@ import (
 	"fmt"
 	"log"
 	"os"
+	"runtime"
 
 	"chainguard.dev/apko/pkg/build"
+	"chainguard.dev/apko/pkg/build/types"
 	"github.com/spf13/cobra"
 )
 
@@ -65,6 +67,12 @@ func BuildMinirootFSCmd(ctx context.Context, opts ...build.Option) error {
 		return err
 	}
 
+	arch := types.Architecture(runtime.GOARCH)
+	if len(bc.ImageConfiguration.Archs) != 0 {
+		log.Printf("WARNING: ignoring archs in config, only building for current arch (%s)", arch)
+	}
+	bc.Arch = arch
+
 	log.Printf("building minirootfs '%s'", bc.TarballPath)
 
 	layerTarGZ, err := bc.BuildLayer()
diff --git a/pkg/cli/build.go b/pkg/cli/build.go
index 5f36646..d92a0ce 100644
--- a/pkg/cli/build.go
+++ b/pkg/cli/build.go
@@ -19,9 +19,11 @@ import (
 	"fmt"
 	"log"
 	"os"
+	"runtime"
 
 	"chainguard.dev/apko/pkg/build"
 	"chainguard.dev/apko/pkg/build/oci"
+	"chainguard.dev/apko/pkg/build/types"
 	"github.com/spf13/cobra"
 )
 
@@ -70,6 +72,11 @@ func BuildCmd(ctx context.Context, imageRef string, outputTarGZ string, opts ...
 	if err != nil {
 		return err
 	}
+	arch := types.Architecture(runtime.GOARCH)
+	if len(bc.ImageConfiguration.Archs) != 0 {
+		log.Printf("WARNING: ignoring archs in config, only building for current arch (%s)", arch)
+	}
+	bc.Arch = arch
 
 	log.Printf("building image '%s'", imageRef)
 
@@ -79,7 +86,7 @@ func BuildCmd(ctx context.Context, imageRef string, outputTarGZ string, opts ...
 	}
 	defer os.Remove(layerTarGZ)
 
-	if err := oci.BuildImageTarballFromLayer(imageRef, layerTarGZ, outputTarGZ, bc.ImageConfiguration, bc.SourceDateEpoch); err != nil {
+	if err := oci.BuildImageTarballFromLayer(imageRef, layerTarGZ, outputTarGZ, bc.ImageConfiguration, bc.SourceDateEpoch, arch); err != nil {
 		return fmt.Errorf("failed to build OCI image: %w", err)
 	}
 
diff --git a/pkg/cli/publish.go b/pkg/cli/publish.go
index 6194e1b..12f271f 100644
--- a/pkg/cli/publish.go
+++ b/pkg/cli/publish.go
@@ -16,12 +16,17 @@ package cli
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"log"
 	"os"
+	"path/filepath"
 
 	"chainguard.dev/apko/pkg/build"
 	"chainguard.dev/apko/pkg/build/oci"
+	"chainguard.dev/apko/pkg/build/types"
+	"github.com/google/go-containerregistry/pkg/name"
+	v1 "github.com/google/go-containerregistry/pkg/v1"
 	"github.com/spf13/cobra"
 )
 
@@ -30,6 +35,7 @@ func Publish() *cobra.Command {
 	var useProot bool
 	var buildDate string
 	var sbomPath string
+	var archstrs []string
 
 	cmd := &cobra.Command{
 		Use:   "publish",
@@ -41,7 +47,8 @@ in a keychain.`,
 		Example: `  apko publish <config.yaml> <tag...>`,
 		Args:    cobra.MinimumNArgs(2),
 		RunE: func(cmd *cobra.Command, args []string) error {
-			if err := PublishCmd(cmd.Context(), imageRefs,
+			archs := types.ParseArchitectures(archstrs)
+			if err := PublishCmd(cmd.Context(), imageRefs, archs,
 				build.WithConfig(args[0]),
 				build.WithProot(useProot),
 				build.WithTags(args[1:]...),
@@ -58,11 +65,12 @@ in a keychain.`,
 	cmd.Flags().BoolVar(&useProot, "use-proot", false, "use proot to simulate privileged operations")
 	cmd.Flags().StringVar(&buildDate, "build-date", "", "date used for the timestamps of the files inside the image")
 	cmd.Flags().StringVar(&sbomPath, "sbom-path", "", "generate an SBOM")
+	cmd.Flags().StringSliceVar(&archstrs, "arch", nil, "architectures to build for (e.g., x86_64,ppc64le,arm64) -- default is all, unless specified in config.")
 
 	return cmd
 }
 
-func PublishCmd(ctx context.Context, outputRefs string, opts ...build.Option) error {
+func PublishCmd(ctx context.Context, outputRefs string, archs []types.Architecture, opts ...build.Option) error {
 	wd, err := os.MkdirTemp("", "apko-*")
 	if err != nil {
 		return fmt.Errorf("failed to create working directory: %w", err)
@@ -74,17 +82,53 @@ func PublishCmd(ctx context.Context, outputRefs string, opts ...build.Option) er
 		return err
 	}
 
+	if len(archs) == 0 {
+		archs = types.AllArchs
+	}
+	if len(bc.ImageConfiguration.Archs) == 0 {
+		bc.ImageConfiguration.Archs = archs
+	}
+
 	log.Printf("building tags %v", bc.Tags)
 
-	layerTarGZ, err := bc.BuildLayer()
-	if err != nil {
-		return fmt.Errorf("failed to build layer image: %w", err)
-	}
-	defer os.Remove(layerTarGZ)
+	var digest name.Digest
+	switch len(bc.ImageConfiguration.Archs) {
+	case 0:
+		return errors.New("no archs requested")
+	case 1:
+		bc.Arch = bc.ImageConfiguration.Archs[0]
+		layerTarGZ, err := bc.BuildLayer()
+		if err != nil {
+			return fmt.Errorf("failed to build layer image: %w", err)
+		}
+		defer os.Remove(layerTarGZ)
 
-	digest, err := oci.PublishImageFromLayer(layerTarGZ, bc.ImageConfiguration, bc.SourceDateEpoch, bc.Tags...)
-	if err != nil {
-		return fmt.Errorf("failed to build OCI image: %w", err)
+		digest, _, err = oci.PublishImageFromLayer(layerTarGZ, bc.ImageConfiguration, bc.SourceDateEpoch, bc.Arch, bc.Tags...)
+		if err != nil {
+			return fmt.Errorf("failed to build OCI image: %w", err)
+		}
+	default:
+		var imgs []v1.Image
+		workDir := bc.WorkDir
+		for _, arch := range archs {
+			bc.Arch = arch
+			bc.WorkDir = filepath.Join(workDir, arch.ToAPK())
+			layerTarGZ, err := bc.BuildLayer()
+			if err != nil {
+				return fmt.Errorf("failed to build layer image for %q: %w", arch, err)
+			}
+			defer os.Remove(layerTarGZ)
+
+			_, img, err := oci.PublishImageFromLayer(layerTarGZ, bc.ImageConfiguration, bc.SourceDateEpoch, arch)
+			if err != nil {
+				return fmt.Errorf("failed to build OCI image for %q: %w", arch, err)
+			}
+			imgs = append(imgs, img)
+		}
+		digest, err = oci.PublishIndex(imgs, bc.Tags...)
+		if err != nil {
+			return fmt.Errorf("failed to build OCI index: %w", err)
+		}
 	}
 
 	// If provided, this is the name of the file to write digest referenced into


Test patch:
diff --git a/pkg/build/types/types_test.go b/pkg/build/types/types_test.go
new file mode 100644
index 0000000..b6220d6
--- /dev/null
+++ b/pkg/build/types/types_test.go
@@ -0,0 +1,52 @@
+// Copyright 2022 Chainguard, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package types
+
+import (
+	"testing"
+
+	"github.com/google/go-cmp/cmp"
+)
+
+func TestParseArchitectures(t *testing.T) {
+	for _, c := range []struct {
+		desc string
+		in   []string
+		want []Architecture
+	}{{
+		desc: "empty",
+		in:   []string{},
+		want: []Architecture{},
+	}, {
+		desc: "sort",
+		in:   []string{"riscv64", "amd64"},
+		want: []Architecture{amd64, riscv64},
+	}, {
+		desc: "dedupe",
+		in:   []string{"amd64", "amd64", "arm64"},
+		want: []Architecture{amd64, arm64},
+	}, {
+		desc: "dedupe w/ apk style",
+		in:   []string{"x86_64", "amd64", "arm64"},
+		want: []Architecture{amd64, arm64},
+	}} {
+		t.Run(c.desc, func(t *testing.T) {
+			got := ParseArchitectures(c.in)
+			if d := cmp.Diff(got, c.want); d != "" {
+				t.Fatal("Got diff (-got,+want):", d)
+			}
+		})
+	}
+}