Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: all of the e2e jobs that run this test Which test(s) are flaking: ` Testgrid link: Reason for failure: mostly: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig apps xref:
approx:
my suspicion: /
I can't find anything else that seems relevant in the time frame, and it is the only change to the test file in that time frame.
merged recently, not sure if it's relevant
Not sure what the procedure it, but we can revert or try some other image instead of Pause, if you think it helps. I don't have a good explanation why the change would make the test flake like this.
I don't have a good explanation, just that it affects the test and is in the time frame that seems to fit the flake spike It may take a bit to pin down what exactly happened here :/ I can't look for a little bit.
I tried to reproduce this behaviour on a test cluster in AWS, but couldn't. I prepared a PR to revert the change, just incase.
Good news and bad news kind of thing... The same test fails in the revert PR also, with the previous EchoServer image:
I did an experiment reverting the second suspect in I couldn't reproduce the issue anymore even though I tried. You can check the results in the PR history.
I believe I have found the issue here. More information and fix incoming
Thanks for the update
I just about have a fix ready here. I was able to narrow the commit range to by looking at when this flake first occurred in It looks like was correct that this was introduced in . I believe what is happening is that in the test we are getting getting a with a set (the one that we previously successfully issued an eviction for) before waiting for 3 pods to be ready again in the namespace (). Before the aforementioned PR caused us to skip checking PDBs when is set we would still report back that eviction was not possible because our with did not meet the other criteria (i.e. was not , , or ), which caused us to still check PDBs, which would show us that we did not have the budget to delete (even though it was not really taking into account the that we were actually attempting to evict). I think there are a few things we can do to address this: we should check that not only is the , but also that it does not have set. above where we call so that we do not attempt to get a before all of the pods are ready. in that the does not have a . Just doing (3) by itself would address our current problem, but (1) and (2) also seem to make sense in the context of what we are trying to check here.
Thanks everyone and apologies for the false alarm :-)
/lifecycle active /priority critical-urgent (this is blocking merges of other fixes) /assign
/milestone v1.19

Discussion:
createReplicaSetOrDie(cs, ns, 3, false) ginkgo.By("First trying to evict a pod which shouldn't be evictable") <ins> waitForPodsOrDie(cs, ns, 3) // make sure that they are running and so would be evictable with a different pdb </ins> pod, err := locateRunningPod(cs, ns) framework.ExpectNoError(err) <del> waitForPodsOrDie(cs, ns, 3) // make sure that they are running and so would be evictable with a different pdb </del> e := &policyv1beta1.Eviction{ ObjectMeta: metav1.ObjectMeta{ Name:      pod.Name,

return jsonpatch.CreateMergePatch(oldData, newData) }) <ins> waitForPodsOrDie(cs, ns, 3) </ins> pod, err = locateRunningPod(cs, ns) // locate a new running pod framework.ExpectNoError(err) <del> waitForPodsOrDie(cs, ns, 3) </del> e = &policyv1beta1.Eviction{ ObjectMeta: metav1.ObjectMeta{ Name:      pod.Name,

ready := 0 for i := range pods.Items { pod := pods.Items[i] <del> if podutil.IsPodReady(&pod) { </del> <ins> if podutil.IsPodReady(&pod) && pod.ObjectMeta.DeletionTimestamp.IsZero() { </ins> ready++ } }

for i := range podList.Items { p := podList.Items[i] <del> if podutil.IsPodReady(&p) { </del> <ins> if podutil.IsPodReady(&p) && p.ObjectMeta.DeletionTimestamp.IsZero() { </ins> pod = &p return true, nil }