Code change:
Currently we unconditionally move pods out of unschedulable status when an event happens. However, we can optimize this by setting some conditions for each event to avoid unnecessarily doing that: 1) PV, PVC, StorageClass or CSINode add/update: move only pods with a PVC. 2) Service add/update: only if ServiceAffinity plugin is enabled in one of the profiles 3) Node add/update: test the pods against the , this include: NodeAffinity, NodeName NodePorts, NodeResources and TaintsTolerations filters. 4) Pod delete: keep it as is (spreading or affinity constraints must be tested, which is expensive to do) This is necessary to avoid wasting scheduling cycles. Related proposal: /sig scheduling /assign
GitHub didn't allow me to assign the following users: adtac. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Currently we unconditionally move pods out of unschedulable status when an event happens. However, we can optimize this by setting some conditions for each event to avoid unnecessarily doing that: 2) Service add/update: only if ServiceAffinity plugin is enabled in one of the profiles 3) Node add/update: test the pods against the , this include: NodeAffinity, NodeName NodePorts, NodeResources and TaintsTolerations filters. 4) Pod delete: keep it as is (spreading or affinity constraints must be tested, which is expensive to do) /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc I'm thinking of if we can amplify the scope: which is to say, not only enable/disable moving (all) pods, but also provides mechanics to let plugins notify the scheduler framework which particular pods should be moved. This would benefits in 's coscheduling implementation to move the preceding (N-1) pods back to activeQ when the Nth pod comes - which satisfies the "minAvaialbe" requirement. Moreover, this would help in getting rid of flushing pods back to activeQ .
The suggestion is not to enable/disable moving all pods in all events, this is only true for the Service events. For the other two, node and volumes, we will test each pod and move the ones that pass the condition. I agree that our proposal of integrating this with filters/permit plugins is more generic, but it adds a lot of complexity and potential for bugs, so we need to be careful with that I think.
One approach could be to start simple, validate and establish a reference point in terms of perf improvements, and then migrate to something more generic. The more generic approach will not be more performant than what is proposed in this issue.
Yes, filters/permit runs in the critical path, and move Pods isn't a lock-free operation. What I want to say is during the design/implementation of this proposal, keep it in mind that plugins may need to actively "suggest" moving particular Pods back to activeQ. Agree.
Right, we already designed the interface for such a potential eventuality:
just fixing double "moving" in title: /retitle Avoid moving pods out of unschedulable status unconditionally
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign /unassign
/cc
what is the current plan? Is it to do the entire proposal in "Efficient Requeuing" doc or just the minimal checks proposed in the issue?
I plan to fix the problems mentioned in this issue, but not entirely following the logic mentioned here. For example, it's inappropriate to move a Pod with PVC once a CSINode happens - the Pod may fail due to a reason irrelevant to its PVC. What I'm proposing is more aligned with the Google doc, but there are 2 major distinctions: I'm not associating the interface to a particular extension point - the EventsToRegister is more a general hook instead of semantically associated with a particular scheduling phase. I'm not going to pass the changed event object to let users decide whether to move or not - that'd increase the latency, also impose an extra burden on plugin developers. Fixing this issue is the minimum goal I can promise in 1.21, but in the future, we may provide other interfaces like enable users to proactively move Pods inside scheduling queue.
That is true, but this is the case right now, not something new this issue is proposing to do. This issue is proposing to address the obvious moves that we can avoid with minimal complexity, which will also likely give us the most bang for the buck. Do you have a more detailed proposal for what you are planning to do?
Yes, the code is almost complete - PR1 () and PR2 (), I will send out another PR later to use ServiceAffinity as an example to show case how it works in details. Basically, the plugin can implement an optional interface called and return a number of ClusterEvents that a Pod failed by that plugin can be made re-schedulable, and also upon a Pod's scheduling failure, we store the "failed plugin" into , so that upon moving Pods in the internal queues, we're able to tell whether the Pod's failed plugins are associated/registered with the current event, and hence perform efficient enqueuing.
You can wait late today or tomorrow when the 3rd PoC PR will be ready, and it'd show you a whole picture.
FYI: the 3rd PR to showcase how it works: PoC: ServiceAffinity enqueue .
Do we have a benchmark we can use to prove that the changes are having a positive impact?
Also, are we planning to keep the existing checks that some events have?
Will do that. Yes, they will be kept.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
I think we should still do 3: or even just NodeAffinity+TaintsTolerations The problem is that just one node is being and potentially we are retrying all pending pods. And this means that we are testing all Nodes for each of those Pods.
I think we can do sth. like this: upon a Node Add/Update, for each unschedulablePodInfo, only verify if the Pod can pass NodeAffinity/NodePorr/etc. filtering logic when the Pod failed by NodeAffinity/NodePort/etc. plugin. In this way, we can save cycles for Pods not failed with NodeAffinity/NodePort.
Latest status of this issue: Interface and implementation in framework side [x] Surface info of failed plugins during PerFilter, Filter and Permit [] [x] Store a cluster event to plugin map in SchedulerQueue [] Plugin implementation and perf evaluation.
/help Some tips/notes for contributors who are interested: please dedicate some time to the item(s) you claimed as code freeze is around the corner, we'd like to finish the above items as early as possible works as a sample you can refer, basically you need to think about which events can make the unschedulabed Pods failed by the specified plugins schedulable, and specify the reasons. Also give a benchmark test result proving it. to design a desired benchmark test, you may need to try the "churn operator", which means you need to rebase locally (unless they get merged) if you meet timeout issue frequently during running a 5000Nodes test, you can work it around by rebasing it's a good practice to run the benchmark test multiple times and eliminate the outliers. Here is a for your information.
I would like to take ‘ tainttoleration’ to learn something about scheduling plugins.
Thanks! I updated the tips to include a to automate running the benchmark tests multiple times.
I claim "nodeports" and shall learn about it firstly.
Can I try to deal with the "nodeaffinity" plugin
I would like take "podtopologyspread" plugin.
Can I take ?
I can take plugin.
Can I take ?
Can i take ?
I opened and will add testcase and test result tommorrow.
I would like to take on
I think I'm left with
updated with proper owners. Thanks! Please assign the reviewer to me when your PR is ready. /remove-help
Wei, there were followups from , are we tracking them somewhere?
the only followup is I will work on that. Let me know if any other followup I missed.
sg, can you open an issue to track that unless you have an open PR.
Sure, create two issues tracking the follow-ups: [ ] sched: cleanup clusterEventReg from [] [x] sched: requeue optimization upon Node Add/Update []
I submitted a PR ( ) for NodePorts, not yet tested. I shall test it this week.
After discussion with , due to 1.21 code freeze (today), we decided to postpone merging most of the opening PRs here. Apologize for that - we'd better not rush to avoid regressions. So priority your time on the on-going PRs, we will still review them, just take your time :)
Use to convert the json files to table.
That's a really useful tool thanks so much for writing that!
oh bummer. I would have liked at least serviceaffinity to go in.
Actually, to get rid of the reaction to service changes... all plugins would have to opt-out, so nvm.
Yes, to achieve that, every plugin has to implement EventsToRegister()... BTW: one way is work it around is to add some temporary logic on framework side - skip movingPods upon Service events if ServiceAffinity is not enabled. And remove this temporary logic when all plugins opt-in EventsToRegister() in the next release.
That wouldn't qualify for codefreeze though.
Nvm then :)
We should simply not register for service updates at all if ServiceAffinity is not configured, which is the default.
Probably not a great idea to totally disable the registration, as an out-of-tree plugin may choose to respond to Service events.
We need to find a way for external plugins to register events because we will remove this registration once the plugin is deprecated.
Yes, also should take CRD into consideration. Probably dynamic informer.
I still think we should disabled Service event handlers when the plugin is not registered. If an external plugin is using it, then they could enable the service affinity filter plugin with an empty affinity labels or patch event handlers until we figure out a way for external plugins to register extra events. btw, do you know of any external plugins that actually rely on service events?
That's fair, and I can work on this, adding some comments caveating this is a temporary workaround. I'm not aware so far.
tbh, I would say plugins depending on services is an antipattern, given the existence of pod affinity. Probably some CRDs might make sense for specialized plugins.
we should add metrics to track the efficiency of the solution. The metric would track the number of pods that were not put back into the queue broken down by plugin and even type for example.
SG. I compiled a list to include on-going and to-do items here .
Are you still working on this?
did we have PRs for number 2 and number 3?
I am asking about the two points from the four mentioned in the description of this issue:
Oops, misunderstanding, ignore me.
number 2: mitigated the problem. In short, includes two parts: one part is to remove Service event from default list so that only Pods failed by a plugin (e.g., ServiceAffinity) which explicitly are interested in Service events can be moved; the other part is to implement EventsToRegister() in ServcieAfffinity plugin. But that doesn't prevent the internal processing cycles like going through the unschedulabe pods and verify whether each is failed by that plugin, so it's a mitigation. PR will fix this problem thoroughly. number 3:
ah, yeah, I knew we worked on that in the previous release, lol. Thanks.

Related discussion:
text: return CSIName } <ins> // EventsToRegister returns the possible events that may make a Pod // failed by this plugin schedulable. func (pl *CSILimits) EventsToRegister() []framework.ClusterEvent { return []framework.ClusterEvent{ {Resource: framework.CSINode, ActionType: framework.Add}, {Resource: framework.Pod, ActionType: framework.Delete}, } } </ins> // Filter invoked at the filter extension point. func (pl *CSILimits) Filter(ctx context.Context, _ *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status { // If the new pod doesn't have any volume attached to it, the predicate will always be true