Code change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --post-kubernetes-push-e2e-agnhost-test-images Image build unhappy on s390x arch: Building image for agnhost OS/ARCH: linux/s390x... [...] [ 3/10] RUN apk --update add bind-tools curl netcat-openbsd iproute2 ipe... 0.165 fetch http://dl- 0.725 fetch http://dl- 1.719 (1/35) Installing ncurses-terminfo-base (6.2p20200523-r0) 1.758 (2/35) Installing ncurses-libs (6.2p20200523-r0) 1.811 (3/35) Installing readline (8.0.4-r0) 1.851 (4/35) Installing bash (5.0.17-r0) 1.939 Executing bash-5.0.17- 1.945 ERROR: bash-5.0.17- script exited with error 1 1.945 (5/35) Installing fstrm (0.6.0-r1) 1.987 (6/35) Installing libgcc (9.3.0-r2) 2.013 (7/35) Installing krb5-conf (1.0-r2) 2.053 (8/35) Installing libcomerr (1.45.6-r0) 2.093 (9/35) Installing keyutils-libs (1.6.1-r1) 2.133 (10/35) Installing libverto (0.3.1-r1) 2.174 (11/35) Installing krb5-libs (1.18.3-r0) 2.334 (12/35) Installing json-c (0.14-r1) 2.376 (13/35) Installing libstdc++ (9.3.0-r2) 2.493 (14/35) Installing libprotobuf (3.12.2-r0) 2.667 (15/35) Installing libprotoc (3.12.2-r0) 2.807 (16/35) Installing protobuf-c (1.3.3-r1) 2.854 (17/35) Installing libuv (1.38.1-r0) 2.906 (18/35) Installing xz-libs (5.2.5-r0) 2.954 (19/35) Installing libxml2 (2.9.10-r5) 3.071 (20/35) Installing bind-libs (9.16.15-r0) 3.287 (21/35) Installing bind-tools (9.16.15-r0) 3.363 (22/35) Installing ca-certificates (-r4) 3.426 (23/35) Installing nghttp2-libs (1.41.0-r0) 3.456 (24/35) Installing libcurl (7.76.1-r0) 3.510 (25/35) Installing curl (7.76.1-r0) 3.561 (26/35) Installing iperf (2.0.13-r2) 3.607 (27/35) Installing libbz2 (1.0.8-r1) 3.649 (28/35) Installing fts (1.2.7-r1) 3.691 (29/35) Installing libelf (0.179-r0) 3.774 (30/35) Installing libmnl (1.0.4-r0) 3.814 (31/35) Installing libnftnl-libs (1.1.6-r0) 3.865 (32/35) Installing iptables (1.8.4-r2) 4.387 (33/35) Installing iproute2 (5.6.0-r0) 4.535 Executing iproute2-5.6.0- 4.540 ERROR: iproute2-5.6.0- script exited with error 1 4.540 (34/35) Installing libbsd (0.10.0-r0) 4.584 (35/35) Installing netcat-openbsd (1.130-r1) 4.627 Executing busybox-1.31.1-r20.trigger 4.632 ERROR: busybox-1.31.1-r20.trigger: script exited with error 1 4.632 Executing ca-certificates--r4.trigger 4.686 2 errors; 29 MiB in 49 packages ERROR: executor failed running [/dev/.buildkitqemuemulator /bin/sh -c apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/ && ln -s /usr/bin/iperf /usr/local/bin/iperf && ls -altrh /usr/local/bin/iperf]: exit code: 2 failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkitqemu_emulator /bin/sh -c apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/ && ln -s /usr/bin/iperf /usr/local/bin/iperf && ls -altrh /usr/local/bin/iperf]: exit code: 2 make: ** [Makefile:43: all-build-and-push] Error 1 ERROR ERROR: build step 0 "" failed: step exited with non-zero status: 2 ERROR: () build -78ae-4086-bc26- completed with status "FAILURE"
2021/04/30 16:58:31 Failed to run some build jobs: [error running [gcloud builds submit --verbosity info --config --substitutions PULLBASEREF=master,WHAT=agnhost,GITTAG=v20210430-v1.22.0-alpha.1-58-gf039f94a6fe --project k8s-staging-e2e-test-images --gcs-log-dir gs://k8s-staging-e2e-test-images-gcb/logs --gcs-source-staging-dir gs://k8s-staging-e2e-test-images-gcb/source gs://k8s-staging-e2e-test-images-gcb/source/-1c8a-4141-9bbb-]: exit status 1] Need this fixed to finish image version dance to fix conformance tests for 1.21.1:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This has failed 3 times consecutively now. All because s390x issues ... I've manually re-triggered it the fourth time. /sig testing but also /sig release because this is a major problem for continuing to ship this platform. cc We've seen this before with flakes across other image builds for this platform.
looks like the latest run passed -
there's still underlying flakiness i need to get to the bottom of
Also what latest run? Were new changes made without bumping the version? Or did someone else manually trigger it yet again?
I just saw a green run at and the image was pullable, so I thought it had succeeded
I think someone must have retried it again manually. Postsubmits only run on merge, and we've had nothing else merge
Guirish tried to see whether our local s390x environment has the same problem too, he'll update us here.
the issue seams to be intermittent caused while pulling respective binaries from Alpine repo. I tried to reproduce the issue locally, I am observing similar build failures while cross building the image for other ARCH too, so the issue is not specific to s390x and it is to do with alpine repo.
thanks i've filed a PR that may help mitigate please review!
looks like a long pending problem
Interesting. FWIW 4/4 consecutive failures appeared to be s390x in CI. We've had previous flakes relating to symlinks affecting this platform only.

Related discussion:
text: # - iproute2: includes ss used in NodePort tests # from iperf image # install necessary packages: iperf, bash <del> RUN apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/*  </del> <ins> RUN retry () { i=0; while [ $i -lt 9 ]; do "$@" && return || sleep 30; i="${i+1}"; done; "$@"; }  && retry apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/*  </ins> && ln -s /usr/bin/iperf /usr/local/bin/iperf  && ls -altrh /usr/local/bin/iperf