Repository: kubernetes/kubernetes (language: go)

Issue:
NodeResourcesBalancedAllocation cause too many pods scheduled to the same node
### What happened?

NodeResourcesBalancedAllocation will return different score if pod request is empty.

```
I1210 06:42:54.701779       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/task-worker-9" node="10.20.96.50" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727588291] requestedResource=map[cpu:14580 memory:24593301504] resourceScore=96
I1210 06:42:54.701793       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/task-worker-9" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:15350 memory:24235737088] resourceScore=92
I1210 06:42:54.701807       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/task-worker-9" node="10.20.96.9" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:13480 memory:19474153472] resourceScore=90
```


### What did you expect to happen?

NodeResourcesBalancedAllocation score should return zero if podRequest is zero. 

https://github.com/kubernetes/kubernetes/blob/a499facee693a1a83daadb82d88f7b51d324ffc5/pkg/scheduler/framework/plugins/noderesources/resource_allocation.go#L85-L114

The code below use `IsPrefixedNativeResource`, which ignore `cpu` and `memory`. 
https://github.com/kubernetes/kubernetes/blob/a499facee693a1a83daadb82d88f7b51d324ffc5/pkg/scheduler/util/utils.go#L158-L162

I think we should use below function.

https://github.com/kubernetes/kubernetes/blob/a499facee693a1a83daadb82d88f7b51d324ffc5/pkg/apis/core/v1/helper/helpers.go#L54-L60

### How can we reproduce it (as minimally and precisely as possible)?

create pod whcih request is empty and enable NodeResourcesBalancedAllocation.

scheduler config:
```yaml
- name: NodeResourcesBalancedAllocation
  args:
    resources:
      - name: cpu
      - name: memory
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
Server Version: v1.24.15
```

</details>


### Cloud provider

<details>
vanilla
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
/sig scheduling
/kind bug
/assign
We only bypass extended resource follow the commit message below, the code check native resource with prefix (kubernetes.io/) however.

https://github.com/kubernetes/kubernetes/commit/20f84b12#diff-75172373a06386d9bba8f7a8841fdeab9f8681ce68a1af56331fbc7ff41102e0R86


IMO, we should bypass `cpu` and `mempory` if pod request is zero.
cc @sanposhiho @AxeZhan @macsko 
Maybe we should first confirm whether this change is reasonable. 
> IMO, we should bypass cpu and mempory if pod request is zero.

I don't think so, the logic here seems to have deliberately taken into account the CPU/memory that the node has already been requested for.

What do you mean by "return wrong score"? If nodes have different requested resources, the score should be different on different nodes, no?
What's the proposal, I mean, not from the implementation perspective, but from the behavior perspective? (i.e., what is going to change after the proposal from users' pov?)

For now, it makes sense to me to still return scores (based on already-allocated resources of nodes) for pods even with zero requests because they'll anyway use some CPU/memory once they're running. It's safer to run pods with zero requests in less busy nodes.

We ignore extended resources because, I guess, we assume Pods without the request for extended resources won't consume those resources at runtime.
> If nodes have different requested resources, the score should be different on different nodes, no?

I think the socre should be same. If the scores differ, too many pods with zero resource requests might get scheduled to the same node.

Additionally, the requested resources for the scheduled node would no longer change.
> It's safer to run pods with zero requests in less busy nodes.

the pods with zero requests will not make less busy nodes to busy, because it will not change the node requested.
> too many pods with zero resource requests might get scheduled to the same node.

I think this will not happen, the requested resources of a node will be updated with a default value if the pod has no requests.
https://github.com/kubernetes/kubernetes/blob/a499facee693a1a83daadb82d88f7b51d324ffc5/pkg/scheduler/framework/types.go#L1035
> the pods with zero requests will not make less busy nodes to busy

I mean, they just don't have any allocation, but they do use cpu/memory as long as the resource limit and the node's resource capacity allows. So, if you only look at the requested resources, then the node looks not busy, but actually it's busier with Pod(s) with no request.

> too many pods with zero resource requests might get scheduled to the same node.

~~But, it's a reasonable argument, does any plugin look at the pod number at the score phase? (I'm outside now and cannot check the impl myself)
I remember, at the filter, the resource fit plugin does.~~

EDIT: I didn't look at @AxeZhan 's comment. 
But, I'm wondering if it still would make sense to take the pod number into consideration at Score phase. (If we haven't yet anywhere)
Let's say the amount of requested resources on each node is the same, we probably prefer to place a new pod to a node with the fewest running pods...? (given the node has the upper limit on the num of running pods.)
> the requested resources of a node will be updated with a default value if the pod has no requests.

@AxeZhan You are right, but the NodeResourcesBalancedAllocation plugin use **real** `requestedResource`.

see #105845. https://github.com/kubernetes/kubernetes/blob/a499facee693a1a83daadb82d88f7b51d324ffc5/pkg/scheduler/framework/plugins/noderesources/balanced_allocation.go#L132


> But, I'm wondering if it still would make sense to take the pod number into consideration at Score phase. (If we haven't yet anywhere)

It works in node resource plugin using `NonZeroReuqested`. 

For real `Requested`, this maybe cause too many pods scheduled to a node that running a `big pod` which request all this node resource.
> For real Requested, this maybe cause too many pods scheduled to a node that running a big pod which request all this node resource.

I know, it might be not ideal in some scenarios, but, in the first place, users must put the resource requests if they want an ideal behavior. We cannot determine what is ideal to do with zero request pods.

---

Again, @Bowser1704 do you have your own proposal to this problem? (Or, you just wanted to open a discussion for this problem.)

If you still propose always returning a 0 score for pods with zero request, what's a counter punch for [my previous comments](https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2531490373)?
I think what @Bowser1704 's proposal is to return same score for every node when pod has no cpu&memory request.
I'm a bit concerned about this, as it means we've lost the functionality of resource plugin with this kind pod.

I checked https://github.com/kubernetes/kubernetes/pull/105845 and I'm not sure why we stop using `nonZeroRequested` in  `NodeResourcesBalancedAllocation`.
https://github.com/kubernetes/kubernetes/pull/102925 says it's because too many zero requested containers can result in a huge nonZeroRequest and thus influence the schedule result.

Checking pod num on the nodes feels right for me. 
And besides, with https://github.com/kubernetes/enhancements/issues/2837, maybe we can change the default cpu/memory value from container-level to pod-level?
As the reason described at my previous comments, I still prefer to keep the current behaviour (+ add a pod number based scoring, but probably smaller weight than cpu/mem), compared to just returning zero scores with pods with no resources. 
If there're node-a with no running pod, node-b with a small pod, and node-c with a huge pod, it makes sense to schedule a pod onto node-a, even if the request is zero, because the pod will anyway consume some resources.
/cc @alculquicondor @macsko @dom4ha 
For now my proposol is just make NodeResourcesBalancedAllocation plugin `useRequest` to be configurable.



> I know, it might be not ideal in some scenarios, but, in the first place, users must put the resource requests if they want an ideal behavior. We cannot determine what is ideal to do with zero request pods.


Got that. I think it's just a tradeoff. 

Return same score means it will use the reservoir sampling, so all feasible nodes are fair, but it might schedule pod to `really busy` nodes.

Use pod-number score is ok, but there's still a potential to schedule too many zero-resource-request pods onto a node which has huge pod.


> I checked https://github.com/kubernetes/kubernetes/pull/105845 and I'm not sure why we stop using nonZeroRequested in NodeResourcesBalancedAllocation.
https://github.com/kubernetes/kubernetes/pull/102925 says it's because too many zero requested containers can result in a huge nonZeroRequest and thus influence the schedule result.

@AxeZhan I test NodeResourcesBalancedAllocation with `nonZeroRequested`, the requested (for cpu and memory) maybe larger than allocatable, and plugin will return score `100` for the `overflow` node. see code below:

https://github.com/kubernetes/kubernetes/blob/8a39b6062060ffb4fba52f53c011bd00428db194/pkg/scheduler/framework/plugins/noderesources/balanced_allocation.go#L145-L147.

scheduler logs:

```log
I1211 07:27:28.840885       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-4" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20150 memory:34616639488] resourceScore=100
I1211 07:27:28.841500       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-40" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20250 memory:34826354688] resourceScore=100
I1211 07:27:28.842080       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-41" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20350 memory:35036069888] resourceScore=100
I1211 07:27:28.842676       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-42" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20450 memory:35245785088] resourceScore=100
I1211 07:27:28.843178       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-43" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20550 memory:35455500288] resourceScore=100
I1211 07:27:28.843733       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-44" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20650 memory:35665215488] resourceScore=100
I1211 07:27:28.844389       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-45" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20750 memory:35874930688] resourceScore=100
I1211 07:27:28.845017       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-46" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20850 memory:36084645888] resourceScore=100
I1211 07:27:28.845602       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-47" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:20950 memory:36294361088] resourceScore=100
I1211 07:27:28.846116       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-48" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:21050 memory:36504076288] resourceScore=100
I1211 07:27:28.847625       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-5" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:21150 memory:36713791488] resourceScore=100
I1211 07:27:28.848933       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-6" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:21250 memory:36923506688] resourceScore=100
I1211 07:27:28.849620       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-7" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:21350 memory:37133221888] resourceScore=100
I1211 07:27:28.850208       1 resource_allocation.go:70] "Listing internal info for allocatable resources, requested resources and score" pod="tuyaco-k8s/hongqi-update-task-worker-8" node="10.20.96.8" resourceAllocationScorer="NodeResourcesBalancedAllocation" allocatableResource=map[cpu:15600 memory:28727600579] requestedResource=map[cpu:21450 memory:37342937088] resourceScore=100

```


> Return same score means it will use the reservoir sampling, so all feasible nodes are fair, but it might schedule pod to really busy nodes.

All the feasible nodes are selected truly randomly. (assuming pods with zero request don't get affected by any other scoring decision) But it doesn't mean all pods with zero requests would be spread to all nodes fairly. That depends on the scheduling order. All zero-request pods may go to one very-busy node, unluckily (extreme example).

Another thought: what if we calculate scores for pods with zero requests completely separately so that they would get spread to all nodes equally? So, all nodes would get the burden coming from pods with zero request truly equally.
It might be simpler/safer than adding a pod-number based scoring, which has a broader blast radius.
> All the feasible nodes are selected truly randomly. 

Every feasible node probability is 1/cntOfMaxScore if node score is same. see code below:

https://github.com/kubernetes/kubernetes/blob/8a39b6062060ffb4fba52f53c011bd00428db194/pkg/scheduler/schedule_one.go#L893-L899
Yes, and that means all pods with the same score could select the same node, if you're unlucky.

---

At a high level, there are two options right now:

- Keep the current resource-based scoring even for zero-request pods basically, but try to avoid scheduling too many zero-request pods onto the same node somehow.
  - maybe a pod-number based scoring or maybe bring back a pod default request. (But, looks like both are not perfect?)
- Try to spread zero-request pods equally.
  - It's essentially similar to what @Bowser1704 wants to propose, from the behavior perspective, no? (If not, correct me what is your proposal _from the behavior perspective_, [like I said at the very beginning](https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2530824028))
  - I'm suggesting, if we want to go this way, it's better to still give scores based on how each node has zero-request pods, not just returning zero score, which is too random.
/unassign

I didn't participate in the discussion later. sorry for this :)
> Yes, and that means all pods with the same score could select the same node, if you're unlucky.

IMO, this probability is too small, it's exponential growth.

> maybe a pod-number based scoring or maybe bring back a pod default request. (But, looks like both are not perfect?)

Yes, but we can use nonZeroRequested and fix NodeResourcesBalancedAllocation plugin problem above https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2534185206.

> Try to spread zero-request pods equally.

Yes, it's what I want to propose.

> if we want to go this way, it's better to still give scores based on how each node has zero-request pods

Agreed that.


> IMO, this probability is too small, it's exponential growth.

That depends on the scheduling order. The scheduler doesn't always schedule zero-request pods.
I know all pods go to the same node is the extreme example anyway though.

Also, there's a deletion of existing running pods, so random assigning (= returning zero score) doesn't make sense to make a good result. Anyways, on this second option, I think we're on the same page according to your last comment; we should give a score based on the existing running zero-request pods.

> we can use nonZeroRequested and fix NodeResourcesBalancedAllocation plugin problem above https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2534185206.

How does "return score 100 for the overflow node" make sense? It means that we keep putting pods to the overflow node, no?
> we should give a score based on the existing running zero-request pods.

Agreed that, but it maybe cost a lot to count all existing running zero-request pods.

> How does "return score 100 for the overflow node" make sense? It means that we keep putting pods to the overflow node, no?

Yes, so we need to fix it. Maybe we can return score 0 for the overflow node.
>  it maybe cost a lot to count all existing running zero-request pods.

We do need to pay attention to the calculation cost, yes. But I believe that won't be an issue by having a cache within the nodeinfo or somewhere. 

> Maybe we can return score 0 for the overflow node.

That would bring back the behavior before https://github.com/kubernetes/kubernetes/pull/102925. 
How would you prevent the problem raised there and also discussed in https://github.com/kubernetes/kubernetes/issues/105220?
> How would you prevent the problem raised there and also discussed in https://github.com/kubernetes/kubernetes/issues/105220?

I got, it can't. It's really hard to make schedule balance with nonZeroRequested in NodeResourcesBalancedAllocation, because every node fraction of cpu to memory is different.

You are right, use nonZeroRequested is not perfect.

> But I believe that won't be an issue by having a cache within the nodeinfo or somewhere.

Agreed that.
Yeah, so we'd need an idea to address those problems raised there, if we go with that option.
I'd like to hear the opinion from broader folks before actually proceeding, especially because the resource scoring is one of the sensitive areas as it could impact many users directly... @kubernetes/sig-scheduling-approvers @macsko @dom4ha @AxeZhan.

I summarized the current discussion for you all to catch up easily.

### Problem

Currently, too many zero-request pods can go to the same node because assigning zero-request pods doesn't change the scoring result. 
So, zero-request pods keep being assigned to the same node until pods with requests are assigned somewhere and change some node's allocation.

### Solution 1: Basically keep the current behavior and try to avoid the problem

So, the cause is assigning zero-request pods to a node doesn't change the score and the plugin keeps making the same decision for all the other zero-request pods. If we can solve this part, we can just keep the current direction.

Two options have been discussed.

#### Option 1: Pod-number based scoring

We may be able to mitigate it by having a pod-number based scoring in NodeResourcesBalancedAllocation. So, each node has the cap of the pod number, we regard the pod number as one resource (and each pod consumes 1), and take it into the calculation of score.
Then, assigning zero-request pods to a node does change the scoring because they don't increase request resources on the node, but they increases the running pod number on the node.

But, it doesn't solve the problem perfectly, e.g., the cluster has various sized pods.
As an extreme example, let's say one node has one huge pod (100 CPU req), while another one has many small pods (1 CPU req * 100 pods), then zero-request pods would keep going to the former node.
But, here, there's another way of thinking, we don't have to solve the problem perfectly; if users need a perfect ideal solution, they should just put a resource request to their pods.

Another concern here is that it has a broader blast radius and contains a risk of introducing some unintended calculation results, apart from the scenarios involving zero-request pods.

#### Option 2: Bring back the pod-default request

We previously applied a default resource request to zero-request pods so that we can take zero-request pods into consideration at scoring.
But, we stopped doing it, the reasons are discussed at https://github.com/kubernetes/kubernetes/pull/102925 (stopped using the pod-default request) and https://github.com/kubernetes/kubernetes/issues/105220 (considered re-using the pod-default request, but decided not to).

If we somehow solve those reasons and manage to bring back the pod-default request, the problem would be solved together.
...but, we're not sure how.

### Solution 2: Try to spread zero-request pods equally

So, it's a new idea of treating zero-request pods. We calculate scores for zero-request pods completely separately from others, and try to spread them to all nodes equally.
So, all nodes would get the burden coming from pods with zero request truly equally, regardless of the current allocated resources. Implementation-wise, we'd track how many zero-request pods are running on each node, and calculate the score based on that if a new zero-request pod is coming; i.e., give high score for nodes with fewer zero-request running pods.
Before delving into the technical details, I'd like to take a step back and better understand the user scenario and motivation behind "running a large number of best-efforts pods". In a real-world k8s cluster (prod/staging/test), I'm curious about the purpose of: a) running a large number of best-efforts pods, while also b) expecting them to be distributed evenly.
I'm also not sure, in the first place, in which cases users don't want to allocate certain resources to containers, but just want to make them best-effort. It's _probably_ always the best practice to give containers the resource requests..? (as far as I can imagine) Maybe @Bowser1704 has some real-world examples.
But, anyways, I can only say that the scheduler should try to do a better job for them just because kubernetes supports such pods. We cannot just stop considering a better scheduling for those pods.

If we go with the second solution, Yes, the behavior would be to distribute all zero-request pods evenly. The argument is that we cannot know how large zero-request pods consume the resources; maybe they're very tiny containers that only need a very small resource, or maybe they actually consume as large resources on the node as possible. 
Just distributing them evenly is the option to avoid the worst situation in which one node is too busy with many zero-request pods (which currently could happen). We distribute zero-request pods evenly so that each node has a similar number of zero-request pods, which we can _suppose_ those pods on each node consume a similar amount of resources in total, on average.

If we think about this solution more thoroughly for a real world, we probably should do it not evenly always, but probably like proportionally to the node's size. i.e., if node-a has 10 allocatable CPU and node-b has 20, then the distribution of zero-request pods should be 1:2. 
> Option 1: Pod-number based scoring

I think we need to be careful when modifying the BalancedAllocation plugin that would also affect scenarios without zero-request pods. This plugin is already a bit slow (and works always similarly to NodeResourceFit) and the change might regress the scheduling throughput.


For now, the available options are solution1's option1 or solution2, and I prefer solution2 for a similar reason; solution 1's option1 has a broader blast radius including non-zero request pods, from both the scoring result and performance perspective.
But, given solution2 is a new direction treating zero-request pods, I'd like to make sure with everyone's eyes that [the above reasoning](https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2538021490) really makes sense and we don't miss any critical scenarios/concerns.
> But, anyways, I can only say that the scheduler should try to do a better job for them just because kubernetes supports such pods. We cannot just stop considering a better scheduling for those pods.

True, but let's define what "better" means, or take a step back, revisit the scenarios and define what's the expected behavior (once we agreed, comes with an integration test to ensure we don't regress on the functionality)

In my opinion, PR #102925, along with subsequent "fixes", introduces complexity by relying on what was perceived as a "reasonable" behavior. I may prefer to go back to the behavior pre-#102925 based on:

- a best-efforts pod, physically consumes resources anyways. To reflect that, logically we should always impose a default cpu/memory requests in resource-related score plugins (but how we craft the default value is another topic, we can revisit to adjust it, and try other criteria mentioned [here](https://github.com/kubernetes/kubernetes/issues/122131#issuecomment-1939568963))
- in terms of best-efforts pods' scoring, it's hard for scheduler to make assumption on the expected behavior - it could be a tiny or mighty one. What scheduler can do here is try its best to avoid the **worst** case. What's the worst case? To me, it's OOMing, CPU throttling in runtime. So no matter of which scorer, the logic of checking overflow should be consistent: once resources overflowed, score less (whether scored to 0 can be discussed).

> If we go with the second solution, Yes, the behavior would be to distribute all zero-request pods evenly. The argument is that we cannot know how large zero-request pods consume the resources; maybe they're very tiny containers that only need a very small resource, or maybe they actually consume as large resources on the node as possible.

Specifying non-zero Pod request is the responsibility of app's submitter. If ppl don't set requests, it means they're not serious about the their own workloads. With a lot of best-efforts pods, it's hard for scheduler to make assumption on the expected behavior.
> running a large number of best-efforts pods case

In our scenario, we have many small workloads, and it's acceptable for these pods to be killed or evicted, so submitters don't want to set resource requests to submitting more pods.

The issue I am encountering is that scheduling too many pods on the same node can cause CPU throttling and out-of-memory (OOM) issues.

So maybe we can rollback NodeResourcesBalancedAllocation to `nonZeroRequested` and fix this https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2534185206
> let's define what "better" means, or take a step back, revisit the scenarios and define what's the expected behavior

> What scheduler can do here is try its best to avoid the worst case. 

Yeah, so the discussed solutions aim at a different "better" to avoid the worst case (We're on the same page about the worst case; one node is too busy because of zero-request pods, causing CPU throttling / OOM).
How each zero-request pod consumes resources is unknown for the scheduler. The difference among solutions just comes from how to put an assumption for this unknown, how each pod would consume resources. 

- The default-request idea assumes that all zero-request pods consume resources similarly on average, hence assigns the same default requests to all zero-request pods, and takes them into consideration at scoring. 
- The evenly-spreading idea assumes that each zero-request pod might be different in terms of their resource consumption natures, but if we evenly spread them, each group of zero-request pods spread evenly onto each node, and the total resource consumption by different zero-request pods would be similar eventually on each node.
  - e.g., there are 3 zero-request deployments and each has a very different nature in resource consumption. Even though pods from each deployment are very different from one another, if we spread them evenly, each node would get the similar number of pods from each deployment, and the total burden that comes from them would be similar. 

> (but how we craft the default value is another topic, we can revisit to adjust it, and try other criteria mentioned https://github.com/kubernetes/kubernetes/issues/122131#issuecomment-1939568963)
> ...
> So no matter of which scorer, the logic of checking overflow should be consistent: once resources overflowed, score less (whether scored to 0 can be discussed).

I believe we all basically agree that we can give 0 scores to overflow nodes.
But, the point raised at https://github.com/kubernetes/kubernetes/pull/102925 is how to define "overflow". (e.g., If 5 CPU node has 10 zero-request pods, should the scheduler regard it as overflow node?) Essentially, that's how to define the default resource request. And, we'd end up hitting the question, is it possible to define the default resource request that works for all clusters; each cluster is composed of different node sizes, and, even within the same cluster, there are different sized nodes. I treat it as not another topic; we should have an answer for this question, if we go with this option.

> With a lot of best-efforts pods, it's hard for scheduler to make assumption on the expected behavior.

...and yeah. So, each solution would fit/wouldn't fit depending on how zero-request pods work in clusters (use cases), just because of a different assumption towards the unknown resource consumption.
I know it's really hard to find the solution that works for all zero-request pods perfectly, so the point would be which solution's assumption is more general, would fit clusters more likely.

...In any cases, people should specify the resource requests if they seriously desire a better scheduling result 😓 
> So, it's a new idea of treating zero-request pods. We calculate scores for zero-request pods completely separately from others, and try to spread them to all nodes equally.

Maybe we can do something even simpler: if it's a pod with zero requests, then the score for all nodes is zero. Basically, it's up to other plugins to decide.
That's actually what @Bowser1704 initially proposed.
I see that like giving up; the plugin would just stop considering them at all. Yes, we may end up having to give up if we find the scoring for zero request pods too difficult after all though, I'd still like to seek a way the plugin can give a score for a better scheduling, from the resource perspective.
Ah, I was missing one thing; Pod topology spread has a [built-in default constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#cluster-level-default-constraints). So, for an even spreading direction, yes, it's one option to just return 0 score, expecting this default Pod topology spread would spread zero-request pods (belonging to the same deployment) evenly, to some extent. Looks the simplest hence safest solution.
Oops, sorry I missed the initial proposal.

It makes some sense: If the pod is not "changing" the balance, then it should be up to other plugins.

But if the pod has, for example, a CPU request but not a memory request, we should calculate the score like today.
Adding my 3 cents. I suspect I may miss some important details, but as my first impression, shouldn't we take user spec literally and indeed consider "0 spec" pods as consuming no resources?

The approach 2 to spread 0 spec pods evenly regardless of the number of non-0 spec pods (their resource allocation) sounds very reasonable. I think it should be aligned with users expectations, otherwise they would provide some small values.

I even think we could ignore the pods count node limit (in other words, not consider the number of pods as a resource). If we really want to be a bit "smarter", we could take the limit into consideration, but only when getting closer to it, to avoid reaching the limit with only 0 spec pods.
I like the idea of returning 0 for all nodes when dealing with best-effort pods. However, we should evaluate this carefully. Based on a quick mental calculation, I'm unsure whether the weight of the default topology spread would consistently take precedence and ensure an even distribution of pods. In some cases, like with MostAllocation, we _might_ inadvertently end up stacking best-effort pods on the same node.

I think we've reached a consensus on the behavior: under no circumstances should best-effort pods be packed onto the same node, as this can lead to OOM kills or CPU throttling. Regardless of the approach we take, this time I'd propose starting by crafting integration tests. These tests will define the expected behavior and ensure our codebase adheres to it, preventing regressions in the future.
> I think we've reached a consensus on the behavior: under no circumstances should best-effort pods be packed onto the same node, as this can lead to OOM kills or CPU throttling.

Shouldn't the plugin return non 0 score then to implement the desired behavior? How returning 0 and relying on other plugins behavior is better?
> Shouldn't the plugin return non 0 score then to implement the desired behavior? How returning 0 and relying on other plugins behavior is better?

That's thanks to Pod topology spread plugin:

> Pod topology spread has a [built-in default constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#cluster-level-default-constraints).
https://github.com/kubernetes/kubernetes/issues/129138#issuecomment-2542681466

As Wei said, it's still true that Pod Topology Spread plugin's score might not always be strong enough to achieve even-spreading of them though, such cases are that some other plugin(s) returned a large score(s in total) to other nodes due to some other perspective, and probably make sense as a whole result.

Of course, we can still consider the idea returning scores from NodeResourcesBalancedAllocation plugin, if there's any need that shows we cannot just return 0 score and rely on Pod Topology Spread plugin. Do you (or anyone) have anything such in mind?
> Of course, we can still consider the idea returning scores from NodeResourcesBalancedAllocation plugin, if there's any need that shows we cannot just return 0 score and rely on Pod Topology Spread plugin. Do you (or anyone) have anything such in mind?

Having the explicit implementation could be more predictable and independent from the Topology Spread plugin (for instance less prone to topology spread configuration/implementation changes), but I guess we don't have clear guidelines and criteria saying how to spread them.

On the other hand returning 0 is the cheapest to implement and run. Since a user haven't specified resources, we could say (and document) that the balancing behavior is undefined and let other plugins decide, so indeed this approach seems reasonable.

Regarding real life use cases, the only obvious one I could think of is scheduling of some type of pods that are supposed to run on each machine regardless of their other assignments (to guarantee scheduling). In this case using the topology spread (by hostname) seems to be actually the right method of spreading. Do know about other resonable use cases?
> under no circumstances should best-effort pods be packed onto the same node, as this can lead to OOM kills or CPU throttling.

I disagree that this is absolutely necessary. If that was the intent, then users should either be setting pod requests or have lower pod count limits.

All I'm saying is that it should be up to other plugins, as the user has influence over what they do, given their startup configuration or explicit spreading rules.

But yes, we should be doing some experiments to see what the behavior is using the default configuration. If it helps, maybe it can be done in a couple of releases, by first multiplying the scores for pods with no requests by a fractional number, essentially making them closer to zero.
Hello everyone,

I’ve organized the key points from our discussion regarding best-effort pods and scheduling behavior.

Problem:

Best-effort pods (with zero resource requests) may be scheduled onto the same node repeatedly. (Doesn’t affect Deployments, due to Pod topology spread has a [built-in default constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#cluster-level-default-constraints).)

Proposed Solutions:

1. Score zero for best-effort pods in NodeResourcesBalancedAllocation (revert to pre-#102925 logic). This allows other plugins (e.g., topology-spread) to prioritize spreading.
Supporters: @dom4ha @alculquicondor
2. Introduce a pod-count-based scorer to balance nodes by pod density.
Supporters: @sanposhiho @AxeZhan
3. Retain current behavior, requiring users to set explicit pod requests.
Supporters: @alculquicondor @Huang-Wei


Related PRs:

* #102925: Uses NonZeroRequested, capping the resource fraction at 1.
* #105845: Uses real requested, which introduces the problem.
IMO, the Short-Term fix would be to revert to pre-#105845.
As far as I remember, I think all (?) of us kinda agreed with the solution number 1, just returning 0 to those pods so that the scheduler honors scores from other plugins. That allows other plugins (i.e., probably Pod Topology Spread in simple cases) to influence the result.
But, our first suggestion for users is always the number 3, setting the resource request for any cases.
> our first suggestion for users is always the number 3

yes, but it's for user, we can make some fix to avoid this situation.


> I think all (?) of us kinda agreed with the solution number 1

so do we have two options: revert to pre-#102925 or pre-#105845?


The best fix is to return `Skip` at PreScore when the pod's request is 0, instead of reverting.

Fix patch:
diff --git a/pkg/scheduler/framework/plugins/noderesources/balanced_allocation.go b/pkg/scheduler/framework/plugins/noderesources/balanced_allocation.go
index 9162758da486e..30c31f14e8f06 100644
--- a/pkg/scheduler/framework/plugins/noderesources/balanced_allocation.go
+++ b/pkg/scheduler/framework/plugins/noderesources/balanced_allocation.go
@@ -63,8 +63,15 @@ func (s *balancedAllocationPreScoreState) Clone() framework.StateData {
 
 // PreScore calculates incoming pod's resource requests and writes them to the cycle state used.
 func (ba *BalancedAllocation) PreScore(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod, nodes []*framework.NodeInfo) *framework.Status {
+	podRequests := ba.calculatePodResourceRequestList(pod, ba.resources)
+	if ba.isBestEffortPod(podRequests) {
+		// Skip BalancedAllocation scoring for best-effort pods to
+		// prevent a large number of pods from being scheduled to the same node.
+		// See https://github.com/kubernetes/kubernetes/issues/129138 for details.
+		return framework.NewStatus(framework.Skip)
+	}
 	state := &balancedAllocationPreScoreState{
-		podRequests: ba.calculatePodResourceRequestList(pod, ba.resources),
+		podRequests: podRequests,
 	}
 	cycleState.Write(balancedAllocationPreScoreStateKey, state)
 	return nil
@@ -93,6 +100,9 @@ func (ba *BalancedAllocation) Score(ctx context.Context, state *framework.CycleS
 	s, err := getBalancedAllocationPreScoreState(state)
 	if err != nil {
 		s = &balancedAllocationPreScoreState{podRequests: ba.calculatePodResourceRequestList(pod, ba.resources)}
+		if ba.isBestEffortPod(s.podRequests) {
+			return 0, nil
+		}
 	}
 
 	// ba.score favors nodes with balanced resource usage rate.
@@ -154,7 +164,6 @@ func balancedResourceScorer(requested, allocable []int64) int64 {
 	// Otherwise, set the std to zero is enough.
 	if len(resourceToFractions) == 2 {
 		std = math.Abs((resourceToFractions[0] - resourceToFractions[1]) / 2)
-
 	} else if len(resourceToFractions) > 2 {
 		mean := totalFraction / float64(len(resourceToFractions))
 		var sum float64
diff --git a/pkg/scheduler/framework/plugins/noderesources/resource_allocation.go b/pkg/scheduler/framework/plugins/noderesources/resource_allocation.go
index ad4970f2aaa37..71eb9138bd45f 100644
--- a/pkg/scheduler/framework/plugins/noderesources/resource_allocation.go
+++ b/pkg/scheduler/framework/plugins/noderesources/resource_allocation.go
@@ -146,3 +146,12 @@ func (r *resourceAllocationScorer) calculatePodResourceRequestList(pod *v1.Pod,
 	}
 	return podRequests
 }
+
+func (r *resourceAllocationScorer) isBestEffortPod(podRequests []int64) bool {
+	for _, request := range podRequests {
+		if request != 0 {
+			return false
+		}
+	}
+	return true
+}


Test patch:
diff --git a/pkg/scheduler/framework/plugins/noderesources/balanced_allocation_test.go b/pkg/scheduler/framework/plugins/noderesources/balanced_allocation_test.go
index cc7060fc36546..f5e6bbb074611 100644
--- a/pkg/scheduler/framework/plugins/noderesources/balanced_allocation_test.go
+++ b/pkg/scheduler/framework/plugins/noderesources/balanced_allocation_test.go
@@ -23,7 +23,6 @@ import (
 	"github.com/google/go-cmp/cmp"
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/klog/v2/ktesting"
 	"k8s.io/kubernetes/pkg/scheduler/apis/config"
 	"k8s.io/kubernetes/pkg/scheduler/backend/cache"
@@ -57,14 +56,6 @@ func TestNodeResourcesBalancedAllocation(t *testing.T) {
 		},
 		NodeName: "node1",
 	}
-	labels1 := map[string]string{
-		"foo": "bar",
-		"baz": "blah",
-	}
-	labels2 := map[string]string{
-		"bar": "foo",
-		"baz": "blah",
-	}
 	cpuOnly := v1.PodSpec{
 		NodeName: "node1",
 		Containers: []v1.Container{
@@ -119,29 +110,23 @@ func TestNodeResourcesBalancedAllocation(t *testing.T) {
 	}
 
 	tests := []struct {
-		pod          *v1.Pod
-		pods         []*v1.Pod
-		nodes        []*v1.Node
-		expectedList framework.NodeScoreList
-		name         string
-		args         config.NodeResourcesBalancedAllocationArgs
-		runPreScore  bool
+		pod                    *v1.Pod
+		pods                   []*v1.Pod
+		nodes                  []*v1.Node
+		expectedList           framework.NodeScoreList
+		name                   string
+		args                   config.NodeResourcesBalancedAllocationArgs
+		runPreScore            bool
+		wantPreScoreStatusCode framework.Code
 	}{
 		{
-			// Node1 scores (remaining resources) on 0-MaxNodeScore scale
-			// CPU Fraction: 0 / 4000 = 0%
-			// Memory Fraction: 0 / 10000 = 0%
-			// Node1 Score: (1-0) * MaxNodeScore = MaxNodeScore
-			// Node2 scores (remaining resources) on 0-MaxNodeScore scale
-			// CPU Fraction: 0 / 4000 = 0 %
-			// Memory Fraction: 0 / 10000 = 0%
-			// Node2 Score: (1-0) * MaxNodeScore = MaxNodeScore
-			pod:          st.MakePod().Obj(),
-			nodes:        []*v1.Node{makeNode("node1", 4000, 10000, nil), makeNode("node2", 4000, 10000, nil)},
-			expectedList: []framework.NodeScore{{Name: "node1", Score: framework.MaxNodeScore}, {Name: "node2", Score: framework.MaxNodeScore}},
-			name:         "nothing scheduled, nothing requested",
-			args:         config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
-			runPreScore:  true,
+			// bestEffort pods, skip in PreScore
+			pod:                    st.MakePod().Obj(),
+			nodes:                  []*v1.Node{makeNode("node1", 4000, 10000, nil), makeNode("node2", 4000, 10000, nil)},
+			name:                   "nothing scheduled, nothing requested, skip in PreScore",
+			args:                   config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
+			runPreScore:            true,
+			wantPreScoreStatusCode: framework.Skip,
 		},
 		{
 			// Node1 scores on 0-MaxNodeScore scale
@@ -161,76 +146,6 @@ func TestNodeResourcesBalancedAllocation(t *testing.T) {
 			args:         config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
 			runPreScore:  true,
 		},
-		{
-			// Node1 scores on 0-MaxNodeScore scale
-			// CPU Fraction: 0 / 4000= 0%
-			// Memory Fraction: 0 / 10000 = 0%
-			// Node1 std: 0
-			// Node1 Score: (1-0) * MaxNodeScore = MaxNodeScore
-			// Node2 scores on 0-MaxNodeScore scale
-			// CPU Fraction: 0 / 4000= 0%
-			// Memory Fraction: 0 / 10000 = 0%
-			// Node2 std: 0
-			// Node2 Score: (1-0) * MaxNodeScore = MaxNodeScore
-			pod:          st.MakePod().Obj(),
-			nodes:        []*v1.Node{makeNode("node1", 4000, 10000, nil), makeNode("node2", 4000, 10000, nil)},
-			expectedList: []framework.NodeScore{{Name: "node2", Score: framework.MaxNodeScore}, {Name: "node2", Score: framework.MaxNodeScore}},
-			name:         "no resources requested, pods without container scheduled",
-			pods: []*v1.Pod{
-				st.MakePod().Node("node1").Labels(labels2).Obj(),
-				st.MakePod().Node("node1").Labels(labels1).Obj(),
-				st.MakePod().Node("node2").Labels(labels1).Obj(),
-				st.MakePod().Node("node2").Labels(labels1).Obj(),
-			},
-			args:        config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
-			runPreScore: true,
-		},
-		{
-			// Node1 scores on 0-MaxNodeScore scale
-			// CPU Fraction: 0 / 250 = 0%
-			// Memory Fraction: 0 / 1000 = 0%
-			// Node1 std: (0 - 0) / 2 = 0
-			// Node1 Score: (1 - 0)*MaxNodeScore = 100
-			// Node2 scores on 0-MaxNodeScore scale
-			// CPU Fraction: 0 / 250 = 0%
-			// Memory Fraction: 0 / 1000 = 0%
-			// Node2 std: (0 - 0) / 2 = 0
-			// Node2 Score: (1 - 0)*MaxNodeScore = 100
-			pod:          st.MakePod().Obj(),
-			nodes:        []*v1.Node{makeNode("node1", 250, 1000*1024*1024, nil), makeNode("node2", 250, 1000*1024*1024, nil)},
-			expectedList: []framework.NodeScore{{Name: "node1", Score: 100}, {Name: "node2", Score: 100}},
-			name:         "no resources requested, pods with container scheduled",
-			pods: []*v1.Pod{
-				st.MakePod().Node("node1").Obj(),
-				st.MakePod().Node("node1").Obj(),
-			},
-			args:        config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
-			runPreScore: true,
-		},
-		{
-			// Node1 scores on 0-MaxNodeScore scale
-			// CPU Fraction: 6000 / 10000 = 60%
-			// Memory Fraction: 0 / 20000 = 0%
-			// Node1 std: (0.6 - 0) / 2 = 0.3
-			// Node1 Score: (1 - 0.3)*MaxNodeScore = 70
-			// Node2 scores on 0-MaxNodeScore scale
-			// CPU Fraction: 6000 / 10000 = 60%
-			// Memory Fraction: 5000 / 20000 = 25%
-			// Node2 std: (0.6 - 0.25) / 2 = 0.175
-			// Node2 Score: (1 - 0.175)*MaxNodeScore = 82
-			pod:          st.MakePod().Obj(),
-			nodes:        []*v1.Node{makeNode("node1", 10000, 20000, nil), makeNode("node2", 10000, 20000, nil)},
-			expectedList: []framework.NodeScore{{Name: "node1", Score: 70}, {Name: "node2", Score: 82}},
-			name:         "no resources requested, pods scheduled with resources",
-			pods: []*v1.Pod{
-				{Spec: cpuOnly, ObjectMeta: metav1.ObjectMeta{Labels: labels2}},
-				{Spec: cpuOnly, ObjectMeta: metav1.ObjectMeta{Labels: labels1}},
-				{Spec: cpuOnly2, ObjectMeta: metav1.ObjectMeta{Labels: labels1}},
-				{Spec: cpuAndMemory, ObjectMeta: metav1.ObjectMeta{Labels: labels1}},
-			},
-			args:        config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
-			runPreScore: true,
-		},
 		{
 			// Node1 scores on 0-MaxNodeScore scale
 			// CPU Fraction: 6000 / 10000 = 60%
@@ -298,18 +213,6 @@ func TestNodeResourcesBalancedAllocation(t *testing.T) {
 			args:        config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
 			runPreScore: true,
 		},
-		{
-			pod:          st.MakePod().Obj(),
-			nodes:        []*v1.Node{makeNode("node1", 0, 0, nil), makeNode("node2", 0, 0, nil)},
-			expectedList: []framework.NodeScore{{Name: "node1", Score: 100}, {Name: "node2", Score: 100}},
-			name:         "zero node resources, pods scheduled with resources",
-			pods: []*v1.Pod{
-				{Spec: cpuOnly},
-				{Spec: cpuAndMemory},
-			},
-			args:        config.NodeResourcesBalancedAllocationArgs{Resources: defaultResourceBalancedAllocationSet},
-			runPreScore: true,
-		},
 		// Node1 scores on 0-MaxNodeScore scale
 		// CPU Fraction: 3000 / 3500 = 85.71%
 		// Memory Fraction: 5000 / 40000 = 12.5%
@@ -342,19 +245,25 @@ func TestNodeResourcesBalancedAllocation(t *testing.T) {
 			runPreScore: true,
 		},
 		// Only one node (node1) has the scalar resource, pod doesn't request the scalar resource and the scalar resource should be skipped for consideration.
-		// Node1: std = 0, score = 100
-		// Node2: std = 0, score = 100
+		// Node1 scores on 0-MaxNodeScore scale
+		// CPU Fraction: 3000 / 3500 = 85.71%
+		// Memory Fraction: 5000 / 40000 = 12.5%
+		// Node1 std: (0.8571 - 0.125) / 2 = 0.36605
+		// Node1 Score: (1 - 0.22705)*MaxNodeScore = 63
+		// Node2 scores on 0-MaxNodeScore scale
+		// CPU Fraction: 3000 / 3500 = 85.71%
+		// Memory Fraction: 5000 / 40000 = 12.5%
+		// Node2 std: (0.8571 - 0.125) / 2 = 0.36605
+		// Node2 Score: (1 - 0.22705)*MaxNodeScore = 63
 		{
-			pod:          st.MakePod().Obj(),
+			pod:          &v1.Pod{Spec: cpuAndMemory},
 			nodes:        []*v1.Node{makeNode("node1", 3500, 40000, scalarResource), makeNode("node2", 3500, 40000, nil)},
-			expectedList: []framework.NodeScore{{Name: "node1", Score: 100}, {Name: "node2", Score: 100}},
-			name:         "node without the scalar resource results to a higher score",
-			pods: []*v1.Pod{
-				{Spec: cpuOnly},
-				{Spec: cpuOnly2},
-			},
+			expectedList: []framework.NodeScore{{Name: "node1", Score: 63}, {Name: "node2", Score: 63}},
+			name:         "node without the scalar resource should skip the scalar resource",
+			pods:         []*v1.Pod{},
 			args: config.NodeResourcesBalancedAllocationArgs{Resources: []config.ResourceSpec{
 				{Name: string(v1.ResourceCPU), Weight: 1},
+				{Name: string(v1.ResourceMemory), Weight: 1},
 				{Name: "nvidia.com/gpu", Weight: 1},
 			}},
 			runPreScore: true,
@@ -392,13 +301,17 @@ func TestNodeResourcesBalancedAllocation(t *testing.T) {
 			fh, _ := runtime.NewFramework(ctx, nil, nil, runtime.WithSnapshotSharedLister(snapshot))
 			p, _ := NewBalancedAllocation(ctx, &test.args, fh, feature.Features{})
 			state := framework.NewCycleState()
-			for i := range test.nodes {
-				if test.runPreScore {
-					status := p.(framework.PreScorePlugin).PreScore(ctx, state, test.pod, tf.BuildNodeInfos(test.nodes))
-					if !status.IsSuccess() {
-						t.Errorf("PreScore is expected to return success, but didn't. Got status: %v", status)
-					}
+			if test.runPreScore {
+				status := p.(framework.PreScorePlugin).PreScore(ctx, state, test.pod, tf.BuildNodeInfos(test.nodes))
+				if status.Code() != test.wantPreScoreStatusCode {
+					t.Errorf("unexpected status code, want: %v, got: %v", test.wantPreScoreStatusCode, status.Code())
 				}
+				if status.Code() == framework.Skip {
+					t.Log("skipping score test as PreScore returned skip")
+					return
+				}
+			}
+			for i := range test.nodes {
 				nodeInfo, err := snapshot.Get(test.nodes[i].Name)
 				if err != nil {
 					t.Errorf("failed to get node %q from snapshot: %v", test.nodes[i].Name, err)
diff --git a/pkg/scheduler/schedule_one_test.go b/pkg/scheduler/schedule_one_test.go
index 8ed4d7bf09847..403fc88cc366b 100644
--- a/pkg/scheduler/schedule_one_test.go
+++ b/pkg/scheduler/schedule_one_test.go
@@ -2939,7 +2939,7 @@ func TestZeroRequest(t *testing.T) {
 				{Spec: large1}, {Spec: noResources1},
 				{Spec: large2}, {Spec: small2},
 			},
-			expectedScore: 150,
+			expectedScore: 50,
 		},
 		{
 			pod:   &v1.Pod{Spec: small},
@@ -3105,10 +3105,10 @@ func Test_prioritizeNodes(t *testing.T) {
 						},
 						{
 							Name:  "NodeResourcesBalancedAllocation",
-							Score: 100,
+							Score: 0,
 						},
 					},
-					TotalScore: 110,
+					TotalScore: 10,
 				},
 				{
 					Name: "node2",
@@ -3119,10 +3119,10 @@ func Test_prioritizeNodes(t *testing.T) {
 						},
 						{
 							Name:  "NodeResourcesBalancedAllocation",
-							Score: 100,
+							Score: 0,
 						},
 					},
-					TotalScore: 200,
+					TotalScore: 100,
 				},
 			},
 		},
@@ -3172,10 +3172,10 @@ func Test_prioritizeNodes(t *testing.T) {
 						},
 						{
 							Name:  "NodeResourcesBalancedAllocation",
-							Score: 100,
+							Score: 0,
 						},
 					},
-					TotalScore: 420,
+					TotalScore: 320,
 				},
 				{
 					Name: "node2",
@@ -3190,10 +3190,10 @@ func Test_prioritizeNodes(t *testing.T) {
 						},
 						{
 							Name:  "NodeResourcesBalancedAllocation",
-							Score: 100,
+							Score: 0,
 						},
 					},
-					TotalScore: 330,
+					TotalScore: 230,
 				},
 			},
 		},
@@ -3222,10 +3222,10 @@ func Test_prioritizeNodes(t *testing.T) {
 						},
 						{
 							Name:  "NodeResourcesBalancedAllocation",
-							Score: 100,
+							Score: 0,
 						},
 					},
-					TotalScore: 110,
+					TotalScore: 10,
 				},
 				{
 					Name: "node2",
@@ -3236,10 +3236,10 @@ func Test_prioritizeNodes(t *testing.T) {
 						},
 						{
 							Name:  "NodeResourcesBalancedAllocation",
-							Score: 100,
+							Score: 0,
 						},
 					},
-					TotalScore: 200,
+					TotalScore: 100,
 				},
 			},
 		},