Code change:
ci-kubernetes-node-kubelet-serial-containerd: ci-kubernetes-node-kubelet-serial: (This is just an example, most of the tests appear to be failing) /sig node /kind failing-test /milestone v1.23 since test is failing in a release-blocking job /priority critical-urgent /assign /cc I believe this stems from
/triage accepted These failures look weird - I don't understand why dockershim is sometimes failing after restarting the kubelet. I guess we should double check that timings match the old GPU test (I think I might've tightened them a bit, bc they were passing when I ran the device suite on the same nodes). What's weirder is that they passed on my PR and all of a sudden broke after merging
Discussed at today - the easiest solution now that 1.23 has been branched is to punt these tests out of the serial lane so we can get it green again for 1.23, since any changes at this point must be backported and we would prefer that this is not release-blocking.
/milestone clear should address this; we won't get a fresh serial run for another ~6 hours so let's confirm next business day that this is temporarily fixed.
/remove-priority critical-urgent /priority important-longterm
can we keep it in the milestone until we get a verified green run?
CI is green today after skipping DevicePluginProbe tests .
/assign Maybe related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Looks like serial has been stable for a while, closing: /close
Closing this issue. <detailsIn response to : Looks like serial has been stable for a while, closing: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen it is referred to here: Tests are skipped today
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you have a chance to look at this? I remember you were close to understanding on what is going wrong here
oh welp, missed that
I was (past tense) but now I need to go back and look at my notes. If someone else wants to take over, be my guest, I've been quite busy with other tasks lately. Otherwise I'll get back to this!
So! I'm able to resume working in this area. So I'm testing last k/k master branch tip and running with : and the tests seems to be running and passing quite reliably! (5 times in a row seems like a promising start :) ) To move forward I suggest to: a explicit s in . This will make more evident we have a pending issue in our camp, that's the most egregious exception we have and it's confusing the skip in the config: - this will enable us to do focused work in this area a PR to remove the skip and iterate over it running CI until it's good enough I can take care of all the above. If CI is still failing, we need to explore possible missing/wrong configuration, but then other e2enode tests should be failing, not just this one.
I agree, this plan sounds good. Thank you!
xref:
started to fail also locally - we have baremetal lanes in RH - in the project CI they skip the relevant test. Investigating a possible common root cause
after a very long troubleshooting and quite few attempts, is ready for review

Related discussion:
text: return &nodeList.Items[0] } <ins> // getLocalTestNode fetches the node object describing the local worker node set up by the e2e_node infra, alongside with its ready state. // getLocalTestNode is a variant of `getLocalNode` which reports but does not set any requirement about the node readiness state, letting // the caller decide. The check is intentionally done like `getLocalNode` does. // Note `getLocalNode` aborts (as in ginkgo.Expect) the test implicitly if the worker node is not ready. func getLocalTestNode(f *framework.Framework) (*v1.Node, bool) { node, err := f.ClientSet.CoreV1().Nodes().Get(context.TODO(), framework.TestContext.NodeName, metav1.GetOptions{}) framework.ExpectNoError(err) ready := e2enode.IsNodeReady(node) schedulable := e2enode.IsNodeSchedulable(node) framework.Logf("node %q ready=%v schedulable=%v", node.Name, ready, schedulable) return node, ready && schedulable } </ins> // logKubeletLatencyMetrics logs KubeletLatencyMetrics computed from the Prometheus // metrics exposed on the current node and identified by the metricNames. // The Kubelet subsystem prefix is automatically prepended to these metric names.