Change:
Which jobs are failing: Which test(s) are failing: BeforeSuite Overall Test Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
Taking an example from 1 of the runs, these failures seem to show up across all failures. These pods failed to reach a "Running" state after 10 minutes Then, while dumping events from the kube-system namespace these couple look interesting That kube proxy image seems to not exist (?), I tried pulling it from another machine and got this The commit that is part of the kube proxy tag, , does coincide with the k/k commit that went in right as this test broke and that commit range only introduces this PR The following CI tests are listed fro that specific commit (when looking at the commit history of k/k) ci-kubernetes-bazel-build ci-kubernetes-bazel-test post-kubernetes-bazel-build post-kubernetes-bazel-test These jobs dont show anything particularly interesting though.
looks like failed in /cc
I think it there may be a bad manifests used for this job. The kube proxy images are being built. If you look for in the logs of other ci jobs, i.e., you will see that an alpha 3 with a k/k commit is being used. For example, The thing is that, I think, there should be a mention of the ARCH in the container image, e.g., amd64 in the above example, where as teh GPU job doesnt have any arch.
Mmm though im not 100 sold on the image name being the problem. Checking out both kube-proxy and kube-proxy-amd64 both exist and both seem to be updated frequently
/assign I had similar issues trying to use the debian-iptables image, let me check if I'm able to solve it here too /cc FYI
this may help:
Seems the job is failing to install the nodes with an error related to the LICENSES folder LICENSES has a recent history of changes: I'm not able to correlate between the changes and job failures, you have better insights in the LICENSES folder, what's the best way to proceed here?
I think changed LICENSES in the root of the built artifacts from a file to a folder /assign
ARAGAGRAGAHHHHH !!!
FTR, I am looking at this, please don't revert my revert revert yet
Failed to repro, but I also get:
seems that this is only affecting the jobs
Yeah, I think I messed up the invocation. It should have run tests.
OK, I got it to run a test but it doesn't fail this way. I have another umask fix, let me back that out.
I can't get it to fail like this. On my run I see 27/27 pods running, but this run seems stuck with: that varies from 24 - 26. My personal kubetest invocation: And my run failed, but with:
Further more, I can't think of ANY way the old non-directory LICENSES file would be present unless there's some cache that is not invalidated when it needs to be and is leaking across tests
-- is this greenhouse/bazel?
I sincerely doubt greenhouse. I don't understand enough about the bazel build in this repo to say one way or the other. Why would it be consistently affecting this and no other job that runs in the same cluster, e.g.
and why started to fail this morning, if the patch was merged a few days ago?
Jordan figured it out. This node image has /home/kubernetes baked in.
2nd time I get to use this today:

Discussion:
mv "${src_dir}/kubelet" "${KUBE_BIN}" mv "${src_dir}/kubectl" "${KUBE_BIN}" <ins> # Some older images have LICENSES baked-in as a file. Presumably they will # have the directory baked-in eventually. rm -rf "${KUBE_HOME}"/LICENSES </ins> mv "${KUBE_HOME}/kubernetes/LICENSES" "${KUBE_HOME}" mv "${KUBE_HOME}/kubernetes/kubernetes-src.tar.gz" "${KUBE_HOME}" fi