Change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose? I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.

Discussion:
timeoutPermit       bool waitAndRejectPermit bool waitAndAllowPermit  bool <del> allowPermit         bool </del> cancelled           bool <ins> waitingPod          string rejectingPod        string allowingPod         string </ins> fh                  framework.FrameworkHandle }

}() return framework.NewStatus(framework.Wait, ""), 3 * time.Second } <del> if pp.allowPermit && pod.Name != "waiting-pod" { return nil, 0 } </del> if pp.waitAndRejectPermit || pp.waitAndAllowPermit { <del> if pod.Name == "waiting-pod" { </del> <ins> if pp.waitingPod == "" || pp.waitingPod == pod.Name { pp.waitingPod = pod.Name </ins> return framework.NewStatus(framework.Wait, ""), 30 * time.Second } <del> // This is the signalling pod, wait until the waiting-pod is actually waiting and then either reject or allow it. wait.Poll(10*time.Millisecond, 30*time.Second, func() (bool, error) { w := false pp.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { w = true }) return w, nil }) </del> if pp.waitAndRejectPermit { <ins> pp.rejectingPod = pod.Name </ins> pp.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { wp.Reject(fmt.Sprintf("reject pod %v", wp.GetPod().Name)) }) return framework.NewStatus(framework.Unschedulable, fmt.Sprintf("reject pod %v", pod.Name)), 0 } if pp.waitAndAllowPermit { <ins> pp.allowingPod = pod.Name </ins> pp.allowAllPods() return nil, 0 }

pp.timeoutPermit = false pp.waitAndRejectPermit = false pp.waitAndAllowPermit = false <del> pp.allowPermit = false </del> pp.cancelled = false <ins> pp.waitingPod = "" pp.allowingPod = "" pp.rejectingPod = "" </ins> } // newPermitPlugin returns a factory for permit plugin with specified PermitPlugin.

registry, prof := initRegistryAndConfig(permitPlugin) // Create the master and the scheduler with the test plugin set. <ins> // TODO Make the subtests not share scheduler instances. </ins> testCtx := initTestSchedulerForFrameworkTest(t, testutils.InitTestMaster(t, "permit-plugin", nil), 2, scheduler.WithProfiles(prof), scheduler.WithFrameworkOutOfTreeRegistry(registry))

permitPlugin.waitAndRejectPermit = test.waitReject permitPlugin.waitAndAllowPermit = test.waitAllow <del> // Create two pods. waitingPod, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "waiting-pod", Namespace: testCtx.NS.Name})) </del> <ins> // Create two pods. First pod to enter Permit() will wait and a second one will either // reject or allow first one. podA, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "pod-a", Namespace: testCtx.NS.Name})) </ins> if err != nil { <del> t.Errorf("Error while creating the waiting pod: %v", err) </del> <ins> t.Errorf("Error while creating the first pod: %v", err) </ins> } <del> signallingPod, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "signalling-pod", Namespace: testCtx.NS.Name})) </del> <ins> podB, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "pod-b", Namespace: testCtx.NS.Name})) </ins> if err != nil { <del> t.Errorf("Error while creating the signalling pod: %v", err) </del> <ins> t.Errorf("Error while creating the second pod: %v", err) </ins> } if test.waitReject { <del> if err = waitForPodUnschedulable(testCtx.ClientSet, waitingPod); err != nil { t.Errorf("test #%v: Didn't expect the waiting pod to be scheduled. error: %v", i, err) </del> <ins> if err = waitForPodUnschedulable(testCtx.ClientSet, podA); err != nil { t.Errorf("test #%v: Didn't expect the first pod to be scheduled. error: %v", i, err) </ins> } <del> if err = waitForPodUnschedulable(testCtx.ClientSet, signallingPod); err != nil { t.Errorf("test #%v: Didn't expect the signalling pod to be scheduled. error: %v", i, err) </del> <ins> if err = waitForPodUnschedulable(testCtx.ClientSet, podB); err != nil { t.Errorf("test #%v: Didn't expect the second pod to be scheduled. error: %v", i, err) } if !((permitPlugin.waitingPod == podA.Name && permitPlugin.rejectingPod == podB.Name) || (permitPlugin.waitingPod == podB.Name && permitPlugin.rejectingPod == podA.Name)) { t.Errorf("test #%v: Expect one pod to wait and another pod to reject instead %s waited and %s rejected.", i, permitPlugin.waitingPod, permitPlugin.rejectingPod) </ins> } } else { <del> if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, waitingPod); err != nil { t.Errorf("test #%v: Expected the waiting pod to be scheduled. error: %v", i, err) </del> <ins> if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, podA); err != nil { t.Errorf("test #%v: Expected the first pod to be scheduled. error: %v", i, err) } if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, podB); err != nil { t.Errorf("test #%v: Expected the second pod to be scheduled. error: %v", i, err) </ins> } <del> if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, signallingPod); err != nil { t.Errorf("test #%v: Expected the signalling pod to be scheduled. error: %v", i, err) </del> <ins> if !((permitPlugin.waitingPod == podA.Name && permitPlugin.allowingPod == podB.Name) || (permitPlugin.waitingPod == podB.Name && permitPlugin.allowingPod == podA.Name)) { t.Errorf("test #%v: Expect one pod to wait and another pod to allow instead %s waited and %s allowed.", i, permitPlugin.waitingPod, permitPlugin.allowingPod) </ins> } }

} permitPlugin.reset() <del> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{waitingPod, signallingPod}) </del> <ins> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{podA, podB}) </ins> } }

permitPlugin.timeoutPermit = false permitPlugin.waitAndRejectPermit = false permitPlugin.waitAndAllowPermit = true <del> permitPlugin.allowPermit = true </del> lowPriority, highPriority := int32(100), int32(300) resourceRequest := v1.ResourceRequirements{Requests: v1.ResourceList{

v1.ResourceMemory: *resource.NewQuantity(400, resource.DecimalSI)}, } <del> // Create two pods. </del> <ins> // First pod will go waiting. </ins> waitingPod := initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "waiting-pod", Namespace: testCtx.NS.Name, Priority: &lowPriority, Resources: &resourceRequest}) waitingPod.Spec.TerminationGracePeriodSeconds = new(int64) waitingPod, err = createPausePod(testCtx.ClientSet, waitingPod)

return w, nil }) <ins> // Create second pod which should preempt first pod. </ins> preemptorPod, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &resourceRequest})) if err != nil {