Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been flaking more frequently since 3/13. Testgrid link: Reason for failure: The issue seems to be a timeout cc /kind flake /sig node /priority important-soon /milestone v1.14
Latest PR to touch that test May be due to a readiness probe failing.
/remove-kind failing-test sending this your way for triage too - could you have a look please? I wonder if it's a node problem or if timeouts are originating elsewhere fwiw, it's failing about 10%-15% of the time in e2e tests:
It looks like updating the status after the patch to the readiness gate takes longer than expected. Looking at this failure: From the test logs: [It] should support pod readiness gates [NodeFeature:PodReadinessGate] ï¿½[1mSTEPï¿½[0m: submitting the pod to kubernetes ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true Mar 14 01:52:56.720: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:57.731: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:58.725: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:59.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:00.721: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:01.708: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:02.753: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:03.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:04.713: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.709: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.767: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false From the kubelet logs: The pod goes to Ready at 1:53:12, and we stopped checking at 1:53:05 This looks expected, based on the description of the parameter in the test. I don't think this should be release-blocking. /assign who owns this feature to determine how we can make the test less flaky. Feel free to close if this is already tracked somewhere else /unassign
Thanks, good to know. if it could be worked on, it would really help stabilize CI - which would be super appreciated, especially for release-blocking jobs.
When this tests was (2 or 3 release back), I remember the 10 second waiting timeout is enough for kubelet to react and update the pod ready condition. IIRC, kubelet reconciles pod status every 15s. But if pod readiness gate condition is updated, kubelet should reconcile ASAP skipping the 15s period. There might be a bug here.
Thanks - do you have time to take a look? It would help to know (especially if we need to include something in known issues).
I just did a few manual tests. Kubelet reacted very quickly. Not able to reproduce the update latency more than 10s.
I ran the test in a loop more than 50 times now on my own e2e cluster. Not a single failure. Is it because there are many tests running in parallel causing high load on kubelet, hence longer update latency?
Interesting, could well be the case Is there a way to validate that? And, is there something here we should raise in terms of performance/scalability (e.g. "update latency varies with pod load")?
That is a well-known result of the rate limiting of the kubelet. The kubelet can only send 5 qps of updates to the API Server, so the kubelet is limited in its ability to update statuses when there are lots of updates to send, which occurs when there is high pod churn.
This issue seems to not be a problem anymore. /close
Closing this issue. <detailsIn response to : This issue seems to not be a problem anymore. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Should this test be marked [Serial] then? (Or maybe flaky?) running into this again
Can you share the test grid link where it is flaky?
I've been testing extensively myself offline to re-evaluate tests to use in presubmit.
this sounds like inherently this test is not safe to run with lots of other tests in parallel, which would make it a candidate for
/reopen This is the flakiest test for PRs Triage link: Since this is a GA feature it should have conformance tests covering it, this specific test was prevented from promotion to conformance due to flakiness issues called out here One of the comments suggested rewriting to use an informer or poll for longer than 10s, and not use an explicit sleep. Tagging with [Serial] seems like it would work from a brute force perspective. Is the 10s number an important or documented SLO somewhere?
Reopened this issue. <detailsIn response to : /reopen This is the flakiest test for PRs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
another data point of sorts: IIRC I disabled this test in kind because it was too flaky, i wasn't aware that it was also flaky on GCE.
opened to deflake
Still seen after disabling auto flake retry:
top-ten flake according to
Confirming this flake is still top-ten after we merged Thanks for all your work here !
no longer top-ten, but still flaky -
back in the top ten according to
/assign assigning sig-node leads for visibility/routing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-priority critical-urgent /priority important-longterm Not in the top ten, but it still flakes occasionally
I'm pretty sure we still blacklist this test in most kind CI because it's too flaky. Aaron Crickenberger < :
/milestone v1.19
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Gonna push this with sig node /cc /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale It's still failing across a number of jobs but I can't tell that it's as painful a flake today
/milestone clear
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
working to reproduce outside CI - way easier to debug
still working my way towards a reproducer.
has been unable to reproduce locally. We are still seeing occasional timeouts in CI: Since this was the only timeout on that test run, we may want to consider tweaking the timeout thresholds?
I'm happy I have time again to work on this issue. It doesn't reproduce locally on BM, and this seems indeed to point to a too busy worker in CI and/or too short timeout. So let's start with the simplest fix:
Today in , it is the top 1 flake for "pr:pull-kubernetes-e2e-gce-ubuntu-containerd".
ok, time for me to prioritize this work
Do we have a testgrid link? I'm looking at to try to understand better the flakes and getting too little from here (but looks relatively good from here?)
More recent triage link: Appears to be failing in a number of containerd and crio jobs. /unassign /priority important-soon /remove-priority important-longterm
/milestone v1.22
Did we vet that this is fixed after ?

Discussion:
"k8s.io/client-go/dynamic" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/kubectl/pkg/util/podutils" </ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/kubelet" "k8s.io/kubernetes/test/e2e/framework"

} validatePodReadiness := func(expectReady bool) { <del> err := wait.Poll(time.Second, wait.ForeverTestTimeout, func() (bool, error) { podReady := podClient.PodIsReady(podName) </del> <ins> err := wait.Poll(time.Second, time.Minute, func() (bool, error) { pod, err := podClient.Get(context.TODO(), podName, metav1.GetOptions{}) framework.ExpectNoError(err) podReady := podutils.IsPodReady(pod) </ins> res := expectReady == podReady if !res { <del> framework.Logf("Expect the Ready condition of pod %q to be %v, but got %v", podName, expectReady, podReady) </del> <ins> framework.Logf("Expect the Ready condition of pod %q to be %v, but got %v (pod status %#v)", podName, expectReady, podReady, pod.Status) </ins> } return res, nil })