Change:
container: .container-$(ARCH) .container-$(ARCH): bin/$(BIN)-$(ARCH) <del> docker build --pull -t $(IMAGE_WITH_ARCH):$(TAG) --build-arg ARCH=$(ARCH) . </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --pull --platform linux/$(ARCH) -t $(IMAGE_WITH_ARCH):$(TAG) --build-arg ARCH=$(ARCH) . </ins> touch $@ push: .push-$(ARCH)

Discussion:
What happened: Non-amd64 images of claim to be for amd64. Obviously, those images are executing just fine but the image configs claim to be for the amd64 architecture. We just started noticing the issue as the library has some more rigorous checks now. What you expected to happen: Each image config should contain the correct architecture. How to reproduce it (as minimally and precisely as possible): A more detailed analysis can be found here: Anything else we need to know?: I suggest to update the 3.1 images. Although most runtimes/container engines are happily eating the image, the configs still claim to be for a different architecture. We haven't noticed the issue since the os/arch validity checks are fairly new in the containers/image library. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
cc
/sig release
cc
I think this is happening because we cross-build these docker images on x86 architecture and inspect shows amd64 for that particular image it is trying to pull:
Will have to figure out a way to tag the appropriate architecture while building docker image, I see an experimental flag flag been introduced which can be used to fix this issue but unfortunately it is not working properly when we build it image
Hey :wave:, I'm currently working on setting the arch/OS for a container image during image build (see ). We could think about building our images with buildah/podman if the feature gets accepted and fits our needs. WDYT?
Most of the build scripts are now using docker client but I'm open for using other tools if they are serving the purpose! wdys
as long as someone signs up to do the work and handle follow ups (AND cover all the scenarios that we do currently - like windows!) cc
ah! windows will be challenging I feel at the moment with buildah, will find a workaround for that!
Is buildah available on Mac or runnable in a docker container?
Ye, will look into it. But I think someone was already doing work on this, will have to check how it looks. And we can probably add the Image Promoter to this, including the Windows bits in a similar way we're doing with the test images.
It runs inside docker containers.

Can we create a list of requirements that I could prepare a demo container? What is the whole set of platforms which we have to support?
So, there used to be a PR that was adding support for Windows for this image: But a few things have changed since then, so things should be a bit different: we'll need multiple Windows OS version support: for 1809, 1903, 1909. we'll have to include wincat: - it is needed for a few kube-proxy usecases, if I remember correctly. As for building, we could probably do it using the Image Promoter? I currently have a few PRs that are adding the Windows bits to the k/k e2e test images, including the part in which the Image Promoter is using a few Windows Server build nodes to build the Windows container images (), so I could add the same bits to the pause image, if people want it. If so, I'll have to know first: is the Image Promoter in place for this image, or do I have to do it from scratch? :smile:
The problem is not just with windows images, most of the time the problem will be with cross-building images for non-amd64 archs like arm64, ppc64le, s390x platform using qemu.
Sorry, this is a bit unclear for me. the problem is that we're using qemu to cross-build the images for non-amd64 archs and we shouldn't?
I just upgraded to latest crio and it refuses to download/use the pause image for me since then. I assume this is from a change in container/libpod implementing more strict checks. In my point of view this is impossible to fix for all images, but at least the base cluster images (pause, static pods for kubernetes like apiserver, etcd) should be fixed. For reference here is the according crio issue I opened:
when we cross-build the images on the amd64, the image will contain the target architecture bits but the image will be signed with host arch(amd64) and this is what is creating issue while pulling the image by crio where they have a strict check of the image architecture and expects to match the host architecture(arm64 or ppc64le).

The solution here is to build all these images natively on the target architecture or feed the proper platform while cross-building(the above buildah PR is to achieve this)
works as expected with . I will prepare a trivial PR to fix this, but then we will need to decide how to roll out the new image...
/assign
sounds good to me,
Should be fixed in
Thanks a lot to all involved!
(whoops :|) building the images now, then we'll start updating references to it.
-the image is now pushed to staging after sorting out the manifest .. will file a promotion to prod now
actually updates to the new image, which should now be live at