Code change:
In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events. If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity. But, after is implemented, the calculation can be configured to include those Nodes.) We don't move the unschedulable Pods with Node delete events... That's a problem. Even when the plugin use Node delete event in , scheduling queue doesn't move the unschedulable Pods with any Node delete events.)
/cc Could you please take a look
These are all intended optimizations. These basic checks prevent the scheduler from retrying unschedulable pods over and and over. So far we haven't had the need to make these checks configurable as they check well established scheduling features. +1 on documenting the behavior, but I'm not sure about changing it or making it configurable (that adds complexity that might not be justified). What is it that you are trying to do?
What is it that you are trying to do? We are trying to remove flushing in queue in , right? Without improving like I suggest, some Pods in unschedulableQ cannot move back to activeQ forever in some cases if we remove flushing. (Actually, our company's custom plugin needs to wait flush because non-assigned Pod's events don't let scheduling queue move unschedulable Pods rejected by that plugin.) So, I think we should simply inform all events to queue and let queue move unschedulable Pods with that events. This can happen in some custom plugins like our company's one and also even in in-tree plugins. For example, Pod Topology Spread (after is implemented). When a user uses and , the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result.
cc who implemented NodeInclusionPolicy. Can you provide a concrete example? The policy just refers to the topology spread calculations. It doesn't mean that the node doesn't have to match the nodeaffinity to be able to accept the pod. I'm asking what is your custom plugin trying to do. Depending on whether it makes sense, we need to evaluate if it's worth the complexity.
For example, for Pod Topology Spread (it uses Pod All event in EventsToRegister), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable This is actually a problem here, 5 mins would be too long for background flush. But it doesn't mean that we should simply shorten the interval time. For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread. So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior. What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account? EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally. If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted

Related discussion:
text: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)