Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: produces an error: What you expected to happen: Conformance tests should be launched. Examining image contents indicates executables are of the correct architecture. How to reproduce it (as minimally and precisely as possible): From aarch64 machine: Anything else we need to know?: Works with v1.17.4: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Architecture /sig Testing
/sig release /remove-sig architecture
any thoughts on this issue at all? We use this to test K3d and are blocked on releasing anything in the 1.18 line because of it.
I don't have expertise in the image release pipeline, I'd suggest reaching out to
Hi Did you run this test in an arm64 virtual machine? If so, I have fixed this issue in cadvisor side. This issue is caused by cadvisor. Currently, the version of cadvisor in kubernetes is v0.36.1-0.-. There is a defect in this version, it gets the wrong machine info and reports num_cores=0 on Arm64 guest. I have delivered the PR into cadvisor in 22 days ago. And this patch was merged in 10 days ago. Please see my pr as reference: Currently, I'm waiting a new tag of cadvisor, so that I can use this new tag in kubernetes and fix this issue in k8s side. Please see my issue: Thanks.
Currently we only have one Arm64 conformance test is alive, this conformance test was run on 2 AWS arm64 instances https://k8s- All other VM-based Arm64 conformance tests are down.
/area conformance
Thanks for taking a look just to be sure it is not a VM issue I fired up an a1-metal arm64 aws instance. Please note I am using sonobuoy for running tests rather than kubetest, I tried to find if kubetest is using the conformance image for testing and from a quick check maybe not. The error is not in the conformance test itself, but launching the conformance test image on arm64. The is . Tried again with with 1.18.2 and got the same error: still works ok for the v1.17 branch: It looks like the build chain for images has changed: So there are now six less layers of the conformance image for the 1.18 branch. Please also note how the arm64 images show amd64 as the architecture for both branches. This has been problematic for trying to air-gap arm64 properly and is a systemic issue for many of the projects on gcr (and elsewhere).
Hi I see. I'll check it. Thanks.
/reopen This issue is still unfixed as of today. here is a reproducer that helps doing the reverse test on a x8664 machine: So this is an image where the base os is actually arm64. starting with v1.18.0 and still as of today (v1.18.5-rc.0 and v1.19.0-beta.2) the uploaded -arm64 image is actually an amd64 image. There must be a typo somewhere. testable this way: This is supposed to generate the 'exec file error', but it doesn't, which indicates that it is actually an x8664 image (amd64).
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen x8664 # podman pull # podman run 66652b9ed7f16a9bc848038ddb00159f3e2d22d8c2bb1ba050913309daf101c9 cat /etc/os-release exec user process caused "exec format error" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC the fix was reverted due to not working in the release pipeline. I did say that would hurt :upsidedownface:
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten This is still blocking conformance testing on arm64, so I hope it can be considered open.
Looks like to me this is a very complicated issue which is related with GCB, Moby Buildkit, it's not easy for me to reproduce and some trouble shooting by myself since I am not able to access GCB. Just wanna know is there any follow up to make this PR merge again? , we do have some needs to run the conformance testing on ARM64 .
SIG testing does not use this image (we prefer not to run the test runner as a pod given our clusters may be unreliable since we're testing changes and we want to avoid e.g. the pod being evicted =data loss), however this is currently used by the CNCF conformance program (which is CNCF / SIG arch) and sonobuoy (which is a vmware project). A bunch of our images are shipping with buildkit / buildx currently at HEAD, something like number could probably be brought back if someone wanted to work on this. Currently I'm not going to have time myself.
/cc
cc for arm64 image :(
xref /assign
It looks like that the issue is not in k/release, since we only build/stage the images via . Then we push the images and annotate them to their corresponding architecture. Looking at the staged artifacts, for example for v1.20.1 arm64: Then we can see that also kube-proxy seems to have this issue: I assume that the fix is somewhere in the build scripts, where I'm now looking into. Beside that, all other architectures (arm, ppc64le, ...) are affected by this bug as well (for the same images only). Edit: Looks like the distroless images are fine, whereas the debian based images (proxy / conformance) are pointing to the wrong architecture. Conformance seems to always use , which might be the issue here. Edit: Done with my experiments. will change the already open PR according to our findings from the slack conversation (), whereas we have to change the used base images for kube-proxy to match the corresponding target architecture (like using for )

Discussion:
BASEIMAGE=debian:stretch-slim TEMP_DIR:=$(shell mktemp -d -t conformanceXXXXXX) <ins> #output format for docker buildx build [push, load] OUTPUT=--load </ins> all: build build:

cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile <del> docker build --pull -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR} rm -rf "${TEMP_DIR}" push: build docker push ${REGISTRY}/conformance-${ARCH}:${VERSION} </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --pull --platform linux/$(ARCH) $(OUTPUT) -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR} </ins> ifeq ($(ARCH),amd64) docker rmi ${REGISTRY}/conformance:${VERSION} 2>/dev/null || true <del> docker tag ${REGISTRY}/conformance-${ARCH}:${VERSION} ${REGISTRY}/conformance:${VERSION} docker push ${REGISTRY}/conformance:${VERSION} </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --pull --platform linux/$(ARCH) $(OUTPUT) -t ${REGISTRY}/conformance:${VERSION} ${TEMP_DIR} </ins> endif <ins> rm -rf "${TEMP_DIR}" push: OUTPUT=--push push: build </ins> .PHONY: build push all