Change:
Appears related to I don't see this flaking in the but it appears to be quite flaky in presubmit CI, where this is the only test on the job that fails. /sig testing node /kind flake /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/cc
/remove-kind failing-test
I was debugging this test this morning the apiserver logs shows the kubelet events come too late, the test times out, is like they are queued in the kubelet the test fails at 10:24:22 but the event arrives at 10:24:27 but kubelet times doesn't add up, there are vents generated by kubelet at 10:23:07 if we check another job , where the test passes, there is only a few seconds delay in a working job I've analyzed the number of kubelet events on a job with the test passing against failing, and the results shows that the failing test have similar number of events, except the ones with the Reason FailedMount
I could've sworn we'd like banned conformance tests from depending on event delivery because it's not guaranteed to be reliable?
right now the test waits 60 seconds for the event since events are not synchronous, should we have a more reasonable period? EDIT saw Ben's comment after I commented, anyway, the test is failing because it takes more than 1 minutes, at least I couldn't find any failure related to no have any event at all
I'm not familiar with this test, but you're right that Events and Reason aren't things that a Conformance test can depend upon Conformance tests are intended to be stable and backwards compatible according to the standard API deprecation policies. Therefore any test that relies on specific output that is not subject to the deprecation policy cannot be promoted to conformance. Examples of tests which are not eligible to conformance: anything that checks specific Events are generated, as we make no guarantees about the contents of events, nor their deliveryanything that checks optional Condition fields, such as Reason or Message, as these may change over time (however it is reasonable to verify these fields exist or are non-empty)
/milestone v1.23
Did some digging. It looks like this has been tagged as Conformance since at least 1.2, ref: (sometime prior to 2015-10-07) So this appears to be a case where we missed going back through the existing corpus of conformance tests when we wrote the above doc. is where the rule originated (merged 2018-10-08) I think this test should be demoted from Conformance, since the very nature of what it tests is certain events being generated SIG Node is free to continue to treat this as release-blocking, but it's not something we can reasonable expect of any Kubernetes cluster
Code itself for corresponding test says 1.9: Git history shows the test basically hasn't changed in 1y:
I'll split out demotion into a separate PR, don't want to tangle this issue up
The first single instance of a timeout failure I see is on I'm going to see if I can try to bisect from there, probably merged recently.
to be clear: the test has been around and we should look at fixing it, I was just surprised to see the test name with , we should open a different thread/issue on that topic I am working on the kubernetes deb/RPM releases for the day or else I'd be taking a look.
,%20master%20(dev)%20[non-serial]&width=5 (IPv6),%20master%20(dev)%20%5Bnon-serial%5D&width=5
Opened to cover the conformance demotion
/priority critical-urgent I am proceeding as though we have a perf regression with kubelet events until proven otherwise. Others feel free to also TAL.
I can't reproduce locally on its own, I either will need to stress test it in parallel with some other stuff or I need a slower machine.
/reopen I don't want to just close this because the timeout was bumped if there's a possible regression
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Staring at notes: Looking at the journal logs I'm seeing some (mild, not unexpected) client-side throttling: So that looks to me like the kubelet thinks it successfully got the event out by 10:23:18 which is well within our 1m timeout. The timestamps in Antonio's post only look at POST, let me see if I can find the PATCH logs from apiserver
From That PATCH was received when the kubelet sent it so the event should have been updated and available... could there be an apiserver latency issue? /sig api-machinery
Possible repro on a just-rebased PR:
I see another PATCH at but I think that's just from the pod getting torn down, it matches the Kubelet logs
this is patching the pod status, probably setting the IPs or something like that, check the URI format is a patch on the pod objetct AFAIK those POST are creating the events , the URI format /api/v1/namespace/<namespace/events ,
latest failure 1:45 am CET fix merged 1:55 AM CET, let's keep it open one day more
I really don't think waiting from 1 -5m is an acceptable fix. This may be catching a performance regression. It has stopped the bleeding but I think the problem remains...
the events , by design, may take 6 mins , so it is completely legit that an event can take 5 mins, you cant' fail if take less (actually, by definition they are unreliable so it should be fine also to not receive it) Performance can not be tested this way, there has to be an explicit test for that, and that test can not use events of course
If the prior "median event lifecycle time" (in the context of this test) was previously < 1 min (and thus test flakes were rare) and now the "median event lifecycle time" has increased so as to trigger regular test flakes, that does seem like something interesting and worthy of prioritization. It just seems probable that there's some "there" there. This could also be explained by environmental changes, did anything change in the underlying test infra that correlates with the increased test failure rate?
We previously had issues where presubmits were failing when periodics were not, see
If the test was newly introduced and was flaking, I'd agree with you, but we've had this test at 100% green since like the 1.2 release, and it only started flaking in the past 48h. This suggests to me that we merged a regression, but since it landed during code freeze, it's very difficult to determine which of the dozens of PRs it was.
Fairly confident there have not been substantial changes to the infra in the past week. For blocking presubmits the infra is moved to public infra owned by the CNCF / SIG K8s infra and the configs are all in shows the CI code being updated regularly, but not the build/test cluster that executes against. KIND upgraded containerd/runc two weeks ago, but that doesn't fit. Given the timing it seems more like:
indeed, check my analysis, a failing job has 429 FailedMount events, a job passing the test only 277, the rest of the events are the same Is not the first time that we have a performance regression or an env problem that causes similar effects, I remember one because of runc, one issue with the DNS on CI machines (that elana linked), other with golang, ... but this is not uncommon at the end of the release when CIs are busier I don't disagree that there can be something, actually I totally agree, I disagree on using this test as a proof or measure of this "something", because: test verifies that, once you create a pod an event from the scheduler and from the kubelet is created. , looking at the code, can take 6 mins and by definition are not realiable , Tests should be clear about what they test , for this cases we need to define a threshold, in sig network we have a test for latency of services, maybe adding one for latency of pods? we can define a threshold there, sig-scalability has it
Latest in triage: It does appear that we had a bunch of CI flakes start showing up as well before increasing the timeouts in the test, so maybe not just a presubmit thing.
/assign
No, it hasn't failed since the timeout bump merged, but the bump suppressing the symptom and hence is making this much more difficult to root cause. I'd suggest looking at some runs before the timeout was bumped?
Per I'm wondering if some of the timing regressions we're seeing could be due to a perf regression having switched to CRI v1 with containerd. I can't find any corresponding failures on CRI-O. /remove-priority critical-urgent since test is no longer flaking
/unassign I'm not actively working this one
I think this is similar to in that mainly dockershim jobs are affected.
FTR the jobs in the OP are not dockershim based, kubeadm's kinder jobs do often use dockershim but kind (not kinder) does not support dockershim. is looking good though, fwiw
can we close this now?
single 1.21 failure. i'm going to call this closed for now.

Discussion:
var events *v1.EventList // Check for scheduler event about the pod. ginkgo.By("checking for scheduler event about the pod") <del> framework.ExpectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { </del> <ins> framework.ExpectNoError(wait.Poll(framework.Poll, 5*time.Minute, func() (bool, error) { </ins> selector := fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUID.UID),

})) // Check for kubelet event about the pod. ginkgo.By("checking for kubelet event about the pod") <del> framework.ExpectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { </del> <ins> framework.ExpectNoError(wait.Poll(framework.Poll, 5*time.Minute, func() (bool, error) { </ins> selector := fields.Set{ "involvedObject.uid":       string(podWithUID.UID), "involvedObject.kind":      "Pod",