Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.
The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
t.Errorf("Failed to create node, err: %v", err) } } <ins> // Regularly send heartbeat event to APIServer so that the cluster doesn't enter fullyDisruption mode. // TODO(Huang-Wei): use "NodeDisruptionExclusion" feature to simply the below logic when it's beta. var heartbeatChans []chan struct{} for i := 0; i < nodeCount; i++ { heartbeatChans = append(heartbeatChans, make(chan struct{})) } for i := 0; i < nodeCount; i++ { // Spin up <nodeCount> goroutines to send heartbeat event to APIServer periodically. go func(i int) { for { select { case <-heartbeatChans[i]: return case <-time.Tick(2 * time.Second): nodes[i].Status.Conditions = []v1.NodeCondition{ { Type:              v1.NodeReady, Status:            v1.ConditionTrue, LastHeartbeatTime: metav1.Now(), }, } updateNodeStatus(cs, nodes[i]) } } }(i) } </ins> neededNode := nodes[1] if test.pod != nil { test.pod.Name = fmt.Sprintf("testpod-%d", i)

} } <ins> for i := 0; i < nodeCount; i++ { // Stop the neededNode's heartbeat goroutine. if neededNode.Name == fmt.Sprintf("node-%d", i) { heartbeatChans[i] <- struct{}{} break } } </ins> neededNode.Status.Conditions = test.nodeConditions // Update node condition. err = updateNodeStatus(cs, neededNode)

} cleanupPods(cs, t, []*v1.Pod{test.pod}) } <ins> // Close all heartbeat channels. for i := 0; i < nodeCount; i++ { close(heartbeatChans[i]) } </ins> cleanupNodes(cs, t) waitForSchedulerCacheCleanup(context.scheduler, t) })