Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: test-cmd runkubectlrun_tests Since when has it been failing: According to triage, the first failure was 2019-02-21 12:00 UTC Testgrid link: Reason for failure: Unknown Anything else we need to know: This may be a flake? I can't tell yet. The testgrid link is not a continuous failure, but it could be that not enough PR's have whatever change is causing the failure Triage link: It looks like it may be happening in CI as well
/sig cli /priority critical-urgent /milestone v1.14
/remove-kind failing-test /kind flake I can't see it consistently happening in CI, so it must be a flake?
What I got when running locally...
And I see the same thing in the build- for
Trying to track down the first occurrence. I think there was an XML parsing error that was obscuring this that was fixed recently. Backtracking through pull-kubernetes-integration failures and visually inspecting them. - runkubectlruntests - 2019-02-22 15:45 PST - runkubectlruntests - 2019-02-21 01:13 PST - runkubectlrun_tests - 2019-02-20 16:50 PST ... and then I don't see this particular test failing during 02-19 or 02-18
So going by that, let's cast a net in commits that landed just prior to the earliest PR failure, and N hours prior
Suspects - ~only because it touches nope, rolled back to right after this landed, no repro after 30 attempts - ~enables server side apply <-- this is my guess~ nope, rolled back to right after this landed, no repro after 30 attempts - reduce lock contention in watchcache
the test might be racy in the first place, maybe enabling the feature changes the time profile of the test.
Welp, now that I have what I think is a local repro loop for this, and I'm bisecting, it's looking like the whole commit range I suspected is fine Looking closer, this particular test case is overly optimistic, it deletes a job and immediately gets pods expecting there to be none. I'd still like to nail down what timing change surface this before I go change the to a or using a different set of flags for the call
Well, for some reason seems to be when the flakes started showing up
I realize that the flake is hard to reproduce especially since it's hard to run specific tests in . You can comment a lot of the tests by playing with some of these shell scripts so that they mostly just run the test you want to run. if you do so, you might be able to run 10 or 20 times the test in a row at this PR and right before. I'm curious what the difference would be in term of flake rate.
Yeah that's what I've been doing. It's and specifically looks to be . Which has nothing directly to do with pods, so I guess it's just a matter of this slowing the apiserver down just enough to start tripping a poorly written test.
FYI for the above. My plan for now is just to rewrite the test so it's more forgiving. But I wanted to confirm that it's expected / known that apiserver performance changed a bit as a result of this.
/assign /assign
What metric are you seeing suffer with that commit? There should be no impact.
A racy test goes from flaking 0% of the time to flaking 10-20% of the time Jordan Liggitt <:
flakes: It should probably use instead. The thing is, I can't get it to flake before. I hacked up test/cmd/legacy- to run just this, and none of the other test cases
It definitely should. I'll take a look at job-pod deletion propagation time before/after that commit, but there's nothing I can think of in that path a new networking API group would impact.

Discussion:
# Clean up kubectl delete jobs pi "${kube_flags[@]}" # Post-condition: no pods exist. <del> kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </ins> # Pre-Condition: no Deployment exists kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''