Change:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD Since v1.7.0-alpha.3, control plane images are only pushed for amd64. Compare: to This is a release-blocker and should be fixed ASAP. I guess the actual regression code-wise happened in test infra. cc
/assign
also cc
/remove-sig testing
Still looking, but "Only tag release containers if the name is different." makes me suspicious.
oh, this is a bug introduced by . fix should be easy.
I can open a new issue as well to track what I sig-testing for but I want us to set up the required testing to make sure this never happens again
the removal was under the guise of "sig testing doesn't control the cutting of bits", if there are tests that need to be put in place to ensure all the right things are cut/released, I would expect sig-release to track that eg: if there's code that needs to be changed in kubernetes/release, it's a sig-release issue; if there's code that needs to be changed in kubernetes/test-infra, it might be a sig-testing issue (or we can triage to the right group)
Sounds good
Details about the exact bug and the fix are in . The reason we didn't notice this sooner is that anago only if the required image doesn't exist - it does so because we don't support all arches for all release versions, and we didn't want to hard-code a list of expected versions into the release repo. (Also, anago automatically , so we were still pushing the legacy versions - we were only dropping the non-amd64 images with this bug.)
Can we add a after the warns? Otherwise even is the user of anago sees the warning, there's nothing he can do about it.
Anago is moving toward being less interactive. If any checks need to be done, they should be done at build time. 1) Determine what should be built 2) Stop right there if the build fails for whatever reason. Adding redundant checks comes with the cost of maintaining hardcoded lists in multiple places. The push step (and release infra in general) should be agnostic and just push what was built.
But in this issue, the build was successful, but some images were removed due to the bug. I'm worried that when building beta.2 this Wednesday, if a similar problem happens, i won't be able to stop anago from pushing the incomplete image sets.
I'm reopening this unless we are really sure that it actually works...
How can we create the testing required to stop this from happening again?
I'm doing a dry run today.
anago could have some sort of sanity checking, though I'm not sure how we'd do that without maintaining hardcoded lists, as notes. I guess the "real" fix is having some automated testing somewhere to consume the non-x86 docker images.
After anago dry run, i saw these images on my machine, so it looks fixed.
Thanks and :tada:!

Discussion:
"${DOCKER[@]}" rmi "${release_docker_image_tag}" 2>/dev/null || true "${DOCKER[@]}" tag "${docker_image_tag}" "${release_docker_image_tag}" 2>/dev/null fi <ins> else kube::log::status "Not a release, so deleting docker image ${docker_image_tag}" "${DOCKER[@]}" rmi ${docker_image_tag} 2>/dev/null || true </ins> fi <del> kube::log::status "Deleting docker image ${docker_image_tag}" "${DOCKER[@]}" rmi ${docker_image_tag} 2>/dev/null || true </del> ) & done