Change:
master-blocking: integration-master The only run that failed on master-blocking was: But shows this test to be failing across different jobs with an extremely high rate. The complete build log can be found here: There are numerous errors, highlighting some of them: is /sig scalability
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-scalability
can you take a look?
cc
/assign to put in my todo, but it likely takes some time for me to start working on (I'm about to go on vacation.) Someone feel free to take over meanwhile, if needed. Do we know when it's started? (Testgrid shows limited number of results)
I had a brief look and found mentioned at the top failed at DRA's test case, which is relatively new one in cc goes until the end, which looks like the cause.
I also checked two failures in ci-benchmark-scheduler-perf-master side, and all failures in those two were from DRA test cases.
/assign
TestScheduling/SchedulingWithMultipleResourceClaims/fast only failed once in ci-kubernetes-integration-master, right? I can't reproduce that locally. is in a much worse state. There are context cancellation errors, which implies that tests are timing out. We don't have enough history in test grid anymore to verify when that started. I remember that performance got worse when queuing hints were disabled, but I think tests were still passing at that time.
/unassign
Whether is it timeout-ed by per-test timeout or timeout for all? If the latter is true, probably we need to lengthen it as we new test cases.
I can't reproduce failures in the benchmarks either. I'm now doing performance comparisons to determine whether there has been a performance regression which could explain the timeouts.
Most likely it is this per-test timeout:
I'm not seeing a big drop between 1.30.0 and current master for :
As a first step, shall we make the per-test timeout longer?
A recent failure:
Hello! This Flake has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting 02:00 UTC Wednesday 10th July 2024 (about 4 weeks from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this is tagged for 1.31, is it still planned for this release?
Yes, it's under investigation.
/milestone v1.31
I've more logging to track down this problem in and also included several other changes there in the hope that they might help, like The underlying problem is that the following happens: pod gets created the PreEnqueue check fails for it because the claim for it has not been created yet (it's using a template) the claim controller creates the claim and stores its name in the pod status the scheduler detects an "Update event" for the pod, but PreEnqueue fails again because the claim has not been received yet the claim add event is received and processed, but does not trigger another scheduling attempt My theory is that the two events (pod modified and claim ) are getting processed so close together that they conflict somehow. Is that plausible? cc The "Update event" calls . The claim event calls (from ). An example log is Grep first for "at least" to get the pod name. Then grep for that pod and the claim created for it. Here the result with annotations: Any idea what to look at next?
I looked for reasons why (called by ) might not do anything for the pod without logging about it. I found and some log output to it. Indeed, that path is taken: What does it mean that the pod is gated? The docs say "Whether the Pod is scheduling gated (by PreEnqueuePlugins) or not.". Okay, so that means that returned false: But isn't the event exactly what may have changed the outcome of ? If it gets ignored, nothing else is going to wake up that pod. I don't understand the logic behind this: Incidentally, this was . This could be the recent change which caused this breakage. /assign
Removing this if check made work again for one test case in my trials. Let's try the revert with master:
/assign
Also, can you add a test case that reproduces that path in ?
I'm not sure I understand how that test works. Can you add it together with the fix?
can you remind me why this logic had to be in PreEnqueue? Let's say, as opposed to PreFilter. I saw and I don't quite like it, because it introduces something specific about SchedulingGates into the "generic" scheduler logic. OTOH, the Pod only gets stuck because PreEnqueue is depending on another object than the Pod itself. If PreEnqueue was depending only on the Pod, then it would have gotten (IIUC correctly). Was the point of PreEnqueue to be faster than PreFilter? If so, it's not quite fulfilling the purpose, because we have to rerun the plugin for every event. Maybe the PreEnqueue logic should be reserved for things that don't depend on factors external to the Pod. But I'd like to hear from as I think you use PreEnqueue as well, and probably depending on external factors. So, how have you dealt with scale?
Let me show our usecase too, to vote -1 for this point.. We also use PreEnqueue in (gang scheduling); PreEnqueue allows Pods to enter scheduling cycles iff all Pods in the group are now ready to retry scheduling. e.g., When gangA-pod1 is rejected by plugin-1 and when gangA-pod2 is rejected by plugin-2, we want to requeue all gangA Pods iff the scheduling queue observes some cluster events that could change the scheduling results of gangA-pod1 and gangA-pod2. (i.e., when QHints of plugin-1 and plugin-2 returns Queue for those Pods) Our points to use PreEnqueue, not PreFilter, are - A better scheduling throughput: By filtering out Pods at PreEnqueue, we can prevent waste scheduling cycles for those Pods, improving a whole scheduling throughput of the scheduler. Prevent too long backoff: If we rejected gang Pods at PreFilter, their backoff time would get too long very easily. PreEnqueue is the only possible way we can filter out Pods without demanding their (i.e., backoff). If we look at such PreEnqueue only, it's more time-consuming than what the scheduling gate does, but if we look at an overal scheduling efficiency, it could be more efficient actually. And, it's a users' responsibility to take a balance between a demerit of time-consuming PreEnqueue and a merit of rich filtering with PreEnqueue. So, I agree that the scheduling queue should be pure ideally, not having anything specific for some plugins though, to me, the reason doesn't make enough sense to limit PreEnqueue's usage as you said, forcing the usage not to depend on things apart from the pod itself. That'd be a big limitation, only considering the scheduling gate plugin, and requires an enough reason.
We have a similar approach to Our pre-enqueue is light weight and defaults to blocking a pod until we have done our processing. The default scheduler then is allowed to handle these pre-processed pods. Our primary goal is to prevent the backoff inside the default scheduler. We enforce quotas in the scheduler, that could cause longer wait times for a pod has until quota is available. If we do not have pre-enqueue those pods will be penalised. That backoff caused major issues for us. We no longer have that problem. When we run in plugin mode we cannot schedule more pods than the default scheduler can handle. We are able to "release" enough pods via our pre-enqueue to not have any impact on the default scheduler throughput. Our scheduling cycle to pre-process pods runs independently out of sync of the default scheduler.
I thing we agreed that it made sense at the time, but I am not sure anymore why. Probably something along the lines of what said. I am happy to go along with whatever guidance SIG Scheduling has for usage of PreEnqueue.
With DRA, all pods that use claim templates (the common case) end up as "cannot be scheduled yet" when created. Without PreEnqueue, all of them would fail once. Originally, avoiding that depended on checking the claims. Later we changed claim creation so that the name of the claim gets recorded in the pod status, so now DRA's PreEnqueue could be limited to depend only on pod changes. There's a small race (pod changed, but new claim not in informer cache yet), but that should be okay - I think... But that wouldn't help in the context of because the framework cannot tell whether some PreEnqueue implementation needs to react to other events than pod changes. That would have to be clarified.
Instruction is only instruction, we won't know how user will use that eventually, and based on the feedbacks, I think we should not constraint the preEnqueue behaviors. I thought about this weeks ago, the root cause here is scheduling is coupled with requeueing, specifically, the activeQ, scheduling cycle cares nothing about the other queues, so if we can make activeQ decoupled with two phases, the scheduling performance can improve a lot. Moreover, if requeueing becomes more and more complex in the future, it will occupy more time that the scheduling cycle should have. Well.. this beyonds the scope of preEnqueue extension point, but I think it values. So I drafted a model here: make activeQ a object store rather than a queue, it can be a map then, accelerate the GET/UPDATE/DELETE OP. maintain a new ToScheduleQ (heap like today's activeQ), this is for the real scheduling, we'll sync the elements from activeQ to ToScheduleQ via a block channel. Considering is quick, will not hold the lock for a long time, so synchronization performance will not be slow. Then in the before we'll lock the for the whole time of , but now, only for the , it's much shorter. The architecture looks like below: !
Ok, it sounds like we have to revert to the existing behavior of re-running PreEnqueue plugins for almost every event. Now, we need two fixes. The hotfix is probably just , as I cannot imagine any other equally simple fix. As a long term fix (not cherry-pickable), we have roughly two solutions, not necessarily exclusive: a distinction of PreEnqueue plugins that need to be rechecked for particular events. Kind of a generalization of but uses the list from instead of a hardcoded check. But definitely no scheduling hints here. was kind of going in that direction, but it still uses the hint, and it doesn't skip processing pods fast enough. cc processing PreEnqueue plugins without holding the lock of the . One option here is the proposal by above, but I'd like to see other proposals, if they are simpler. Any volunteers to explore these alternatives?
Can you elaborate it? We don't run PreEnqueue for every event already; we only run PreEnqueue when we're about to put pods in activeQ, which is either: when events in happen (since we move Pods to activeQ only when those events happen). pod itself is updated or pod is just created. I haven't come up with any improvements to reduce the number of calls to PreEnqueue further. (except QHint; QHint filters out events further and hence reduce the number of PreEnqueue calls at both two points above) +1 Not only PreEnqueue, but handling events and requeueing stuff in general should be as decoupled from activeQ as possible so that their slowdown doesn't block starting scheduling cycles. I mentioned such a general change in the past degradation issue, but haven't got time yet. I'd be pleased if it'd be taken over by someone as my recent time is eaten up mostly by reviewing, especially these days. otherwise I will come back myself after some time.
I'll jump in with my model design.
Thanks, do feel free to let me know at the time review is needed :) So, when would activeQ write Pods to ToScheduleQ? Given keeps getting a new Pod and sort Pods with a new one every time, I'm not sure how it works like ↓ this kind of cases- e.g., if we move low-priority-Pod to ToScheduleQ but then higher-priority-Pod comes into the scheduling queue, we have to schedule higher-priority-Pod first.
Another simpler proposal just popped up in my mind: can we just make a new , which only protects activeQ? Then we don't need to take during PreEnqueue and event handling (QHints), but we only need it when we actually put Pods into activeQ. Could it cause any race condition within the scheduling queue?
I think it's worth exploring. I can volunteer to write a prototype. One advantage of this approach is that we can avoid map writes in the proposal by (in addition to the subsequent insert into the heap), which are known to be slow. EDIT: does it have to be a map? can it be a slice? it doesn't have to be ordered and duplicates are maybe ok? So, right now, for Pod deletions is preventing PreEnqueue from running if it was rejected by the SchedulingGates plugin. My hope is that we can generalize that using . Another alternative would be to run PreEnqueue plugins much earlier, so we can skip those pods faster. We can inform this using profiling. Overall, I think we should pursue both approaches, but maybe starting with optimizing locking is the best first step.
I meant, actually, before we a shortcut in , we already filtered out Pod deletion events by , and we didn't call PreEnqueue for scheduling-gated Pods. A flow before we introduced a shortcut was: Pod/Delete arrives -iterates each unsched Pods - filtered out scheduling-gated Pods from requeueing process because the scheduling gate plugin doesn't have Pod/Delete in . (-scheduling-gated Pods don't reach PreEnqueue) So, it was generalized with EventsToRegister, but then it wasn't fast enough. EDIT: I revisited the past issue, and given , it might be possible to optimize by bringing back how a lightweight handled events into . Either way, I don't expect we can make a general solution as fast as a current shortcut, since filtering out Pods with is messively fast. But, once we success in the activeQ improvement and make the requeueing throughput decouple from the scheduling throughput, how fast we need in the requeueing throughput would be another story (). We could remove a shortcut if it's OK for the requeueing throughput.
Lock for activeQ is something I thought in the first place, but considering all the three queues work as a unit, only lock one queue seems risky to me, and we have to add a lot of lock/unlock logic additionally, adding complexities, so I think a decoupled architecture seems more simple, Simple doesn't mean it's Easy, we have to pay the cost of synchronization of activeQ and ToScheduleQ, or something else. Worth the exploration anyway.
Isn't that only if QueueingHints is enabled? Then maybe it's worth doing it for when QueueingHints is disabled (without calling the hints themselves, of course).
No, we do so with the feature flag disabled too. "A flow before we introduced a shortcut" I mentioned in the last comment is true for both the feature enabled and disabled. if plugin-a registers Node/update only and a pod is rejected by plugin-a in a sched cycle or gated by plugin-a, all Node/update events move the pod, and then the pod goes thru PreEnqueue right before getting enqueued to activeQ. All the other events are filtered out and the pod doesn't get moved to activeQ, thus doesn't reach PreEnqueue.
But then we would need to merge for it to work for SchedulingGates, wouldn't we?
No. The flow and result wouldn't be changed with The scheduling gate plugin doesn't have any events, and then the scheduling queue (more specifically, isPodWorthRequeuing) regards it as not-interested in any events, no need to requeue Pods rejected by the scheduling gate plugin (i.e., sched-gated Pods) with any events.
Gotcha, thanks for clarifying. I guess it might still be worth exploring moving those skips earlier, but let's see how far we can get with managing locks better first.
I also just noticed that the scheduling cycle can be blocked by the queue's lock when putting rejected Pods back to the queue, which we also have to deal with in order to decouple the scheduling throughput and the requeueing throughput. For this point, maybe we can just do in goroutine.
I'm not sure we can do that. When there are preemptions, it might be important that the recently rejected pod is back in the queue before starting another scheduling cycle. But we also have backoffs in most scenarios, so maybe it isn't so bad? For comparison, in Kueue we definitely have to lock the queue, as it provides stronger FIFO guarantees.
Good point, we must handle nominatedNode stuff before starting a next sched cycle, otherwise a pod Pop()ed next might rob the first pod of the place created via preemption (i.e., nominated node). Created for a possible solution.
I drafted the proposal to add and use it while manipulating the (). I don't think it would be possible to lock the less than in that commit. If someone see the spaces for more improvements, please let me know. I don't see any scheduling throughput improvements compared to pre change state ( test). Moreover, removing the condition turns back the throughput from 120 pods/s to 55 pods/s even using the new locks.
Maybe you can try to benchmark again with my commit on since requeueing back Pods to the queue also takes the queue's lock. Also, can you check if there's any other point(s) in scheduling cycles that take a shared lock of the queue? or more generally, you can check with pprof where, in scheduling cycles, takes time if we remove a shortcut of .
I spot a few more places in scheduling cycles that takes the lock in the scheduling queue: Calling method . It doesn't affect the throughput in benchmark, but for sure can be a bottleneck in some use cases. In my opinion it's not possible not to lock the whole queue there. The question is if we can somehow make it concurrent. in each cycle which is an actual bottleneck. Without locking on it, scheduling throughput jumps to around 250 pods/s for new locks with shortcut () and 125 pods/s with new locks but without shortcut. The problem is, we cannot use the new lock for this method as it the . I'm wondering, if we could lock the separately, so introduce a yet another lock. , which does't affect the throughput, but still can be a bottleneck. I think fix for the could also work for this case.
Thanks, that's awesome: : It takes a shared lock. We can just put in a goroutine as you said. /: Both should be done before next scheduling cycle starts (= we cannot do this concurrently) because nominated Nodes affects scheduling results... ... I'm wondering, if we could lock the nominator separately, so introduce a yet another lock. Regarding nominatedPod ones, so you meant it's taking time with the lock of ? The lock of is already another one from a shared lock of , right?
lock is the same as lock, so any ongoing operations on the queue affect and block the . In only waiting for lock takes significant time.
Ah now I got it, so we embed in and hence and are the same actually. Okay, that's where I misunderstood. Then, I think splitting a lock for and is worth exploring.
Really good research. I agree with suggestion to have a lock for the nominator.
Just be careful not to regress this
I think one solution could be that the nominator only holds information about which pods are nominated to each node, but it doesn't have the rest of the information about the pods (no PodInfo). Then, the function can be implemented by the SchedulingQueue by first gathering the list of pods from the nominator, and producing the pod infos from its cache. So this operation would require a full lock, but this only happens when we need to do preemption. On the contrary, would only need the lock for the nominator. Just to analyze a different option: locking twice when processing events: once to list all the pods that are in the queue (this could be a read lock) PreEnqueue, scheduling hints and so on, without any lock. again (write this time) to move all the pods that need to be moved between queues. The problem is that, between step 1 and 3, the scheduler could requeue additional pods, which could be made schedulable by the event. So this option doesn't work. Unless I missed something? A reminder of our goal: we are trying to optimize for the case where scheduling is successful (pods can be bound), while we have many gated pods.
... The problem is that, between step 1 and 3, the scheduler could requeue additional pods, which could be made schedulable by the event. I think we probably can solve the problem actually. So, the problem in the twice-lock approach, in details, is: events, which have happened until , are stored in () runs QHints for those past events. a. Between (2) and (3), the event handler receives an event and the queue handles it. This event may make the Pod schedulable. queue doesn't find any events that could make the Pod schedulable and puts the Pod back to unschedQ. The point is there's a moment between (2) and (3). Currently, between (2) and (3), no event could be handled because takes lock that the event handling () also requires. But, if we took the twice-lock approach, could happen. One solution towards this problem is that we put the pod anyway in unschedQ first, and then handle . i.e., would be: always put the Pod into unschedQ first. runs QHints for . it find the Pod should be moved to activeQ/backoffQ, then moves it from unschedQ. otherwise it just keeps the Pod in unschedQ. In this way, there's no moment at all that could miss the events. All events that have happened during the scheduling cycle are handled as . And, all events that have happened during running are handled in side (as we put the Pod in unschedQ) There could be events that are handled twice (in and in as ), but that should be OK.
As I understand in this case we would like to store only the pod UID (or name + namespace) in the nominator. But how the pod infos should be obtained from the cache in then? Using the and structures according to the ?
Yes, would be implemented by the SchedulingQueue by grabbing the global lock and pulling information from all the necessary places.
I somewhat follow your proposal, but does it work without queueing hints? If not, I guess we can hold from implementing this two stage locking until we have more confidence about queueing hints.
Yes, I bet the same solution should work in case without QHints too.
(But, I'm not saying the twice-lock approach is better. I'm just trying to provide another possible option that we might want to explore. If the splitting lock approach, which is working on, brings an enough throughput, then I'm ok with that.)
I agree. Once we have some stability, I think we can circle back to queuing hints.

Discussion:
"k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/scheduler/framework" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/interpodaffinity" <ins> "k8s.io/kubernetes/pkg/scheduler/framework/plugins/names" </ins> "k8s.io/kubernetes/pkg/scheduler/internal/heap" "k8s.io/kubernetes/pkg/scheduler/metrics" "k8s.io/kubernetes/pkg/scheduler/util"

activated := false for _, pInfo := range podInfoList { <del> // Since there may be many gated pods and they will not move from the // unschedulable pool, we skip calling the expensive isPodWorthRequeueing. if pInfo.Gated { </del> <ins> // When handling events takes time, a scheduling throughput gets impacted negatively // because of a shared lock within PriorityQueue, which Pop() also requires. // // Scheduling-gated Pods never get schedulable with any events, // except the Pods themselves got updated, which isn't handled by movePodsToActiveOrBackoffQueue. // So, we can skip them early here so that they don't go through isPodWorthRequeuing, // which isn't fast enough to keep a sufficient scheduling throughput // when the number of scheduling-gated Pods in unschedulablePods is large. // https://github.com/kubernetes/kubernetes/issues/124384 // This is a hotfix for this issue, which might be changed // once we have a better general solution for the shared lock issue. // // Note that we cannot skip all pInfo.Gated Pods here // because PreEnqueue plugins apart from the scheduling gate plugin may change the gating status // with these events. if pInfo.Gated && pInfo.UnschedulablePlugins.Has(names.SchedulingGates) { </ins> continue } schedulingHint := p.isPodWorthRequeuing(logger, pInfo, event, oldObj, newObj) if schedulingHint == queueSkip { // QueueingHintFn determined that this Pod isn't worth putting to activeQ or backoffQ by this event.

"k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/scheduler/framework" plfeature "k8s.io/kubernetes/pkg/scheduler/framework/plugins/feature" <ins> "k8s.io/kubernetes/pkg/scheduler/framework/plugins/names" </ins> "k8s.io/kubernetes/pkg/scheduler/framework/plugins/queuesort" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/schedulinggates" "k8s.io/kubernetes/pkg/scheduler/metrics"

expectedQ: unschedulablePods, }, { <del> name:    "QueueHintFunction is not called when Pod is gated", podInfo: setQueuedPodInfoGated(&framework.QueuedPodInfo{PodInfo: mustNewPodInfo(p), UnschedulablePlugins: sets.New("foo")}), </del> <ins> name:    "QueueHintFunction is not called when Pod is gated by SchedulingGates plugin", podInfo: setQueuedPodInfoGated(&framework.QueuedPodInfo{PodInfo: mustNewPodInfo(p), UnschedulablePlugins: sets.New(names.SchedulingGates, "foo")}), </ins> hint: func(logger klog.Logger, pod *v1.Pod, oldObj, newObj interface{}) (framework.QueueingHint, error) { return framework.Queue, fmt.Errorf("QueueingHintFn should not be called as pod is gated") }, expectedQ: unschedulablePods, }, <ins> { name:      "QueueHintFunction is called when Pod is gated by a plugin other than SchedulingGates", podInfo:   setQueuedPodInfoGated(&framework.QueuedPodInfo{PodInfo: mustNewPodInfo(p), UnschedulablePlugins: sets.New("foo")}), hint:      queueHintReturnQueue, expectedQ: activeQ, }, </ins> } for _, test := range tests {

QueueingHintFn: test.hint, }, } <del> test.podInfo.UnschedulablePlugins = sets.New("foo") </del> cl := testingclock.NewFakeClock(now) q := NewTestQueue(ctx, newDefaultQueueSort(), WithQueueingHintMapPerProfile(m), WithClock(cl)) <del> // add to unsched pod pool </del> q.activeQ.Add(q.newQueuedPodInfo(test.podInfo.Pod)) if p, err := q.Pop(logger); err != nil || p.Pod != test.podInfo.Pod { t.Errorf("Expected: %v after Pop, but got: %v", test.podInfo.Pod.Name, p.Pod.Name) } <ins> // add to unsched pod pool </ins> err := q.AddUnschedulableIfNotPresent(logger, test.podInfo, q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)

return nil } <del> // This test helps to verify registering nil events for schedulingGates plugin works as expected. func TestSchedulingGatesPluginEventsToRegister(t *testing.T) { </del> <ins> // This test helps to verify registering nil events for PreEnqueue plugin works as expected. func TestPreEnqueuePluginEventsToRegister(t *testing.T) { </ins> testContext := testutils.InitTestAPIServer(t, "preenqueue-plugin", nil) num := func(pl framework.Plugin) int {

} tests := []struct { <del> name              string withEvents        bool </del> <ins> name       string withEvents bool // count is the expected number of calls to PreEnqueue(). </ins> count             int queueHintEnabled  []bool expectedScheduled []bool

{ name:              "preEnqueue plugin with event registered", withEvents:        true, <del> count:             2, </del> <ins> count:             3, </ins> queueHintEnabled:  []bool{false, true}, expectedScheduled: []bool{true, true}, },

t.Run(tt.name+fmt.Sprintf(" queueHint(%v)", queueHintEnabled), func(t *testing.T) { featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, queueHintEnabled) <del> // new plugin every time to clear counts </del> <ins> // use new plugin every time to clear counts </ins> var plugin framework.PreEnqueuePlugin if tt.withEvents { plugin = &SchedulingGatesPluginWithEvents{SchedulingGates: schedulinggates.SchedulingGates{}}