Change:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : redefine the formula, the score should be capped per images instead of pod. Why is this needed: For the pod that going to be scheduled, calculate the score based on the below steps, Scaled score base on the image size and it's spread over the cluster, image[i].nodes: number of nodes that already have image ! Capped score based on the assumption 90% image size on dockerhub falls into the range [23M ~ 1000M] lowerbound: 23M upperbound:1000M ! Final scores returned for scheduling, maxscore: 100 ! The second step is not well guarded, the assumption is based on the image size while the pod may hold multiple images, it is not surprise to see that total image size of the pods is larger than 1000M, while each of might be only around 300M. In this below case, following the above formula, NODE1 and NODE2 willl get the same scores(100), since both of them are capped by the max score (100), the ideal result is score of NODE2 should be larger than NODE1 (all requested images are ready on NODE2) and is preferred, the larger the sample1 here the more scores should be expected for NODE2. !
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc
Thanks a lot for summarizing the problem. I agree that the formula needs improvement. ~I think doing the capping for each image solves the original problem and what said in There are two options: only the total from to for each container. We have to ask ourselves how much signal each formula gives and depending on what. Think about different scenarios: a pod with a lot of very small containers, a pod with a few big containers, etc.
/assign
Here is my thoughts, step1: calculate image level priority, for each image (should be same as exsting func ) step2: cap the each image's score with thresholds, minThreshold and maxThreshold. (should be same as existing func except that it's per image instead of pod) step3: score normalization per image. step4: calculate pod level priority, and return the final score, range (0~100) step1 and step2 primarily update the exiting code to make it is image level priorities instead of pod level. Based on the above steps, consider the below two cases, a pod with four images, case1: All images are ready on node2 node1 (0, 80, 90, 60) - final score is: (80+90+60) / 4 = 57 node2 (90, 80, 90, 60) - final score is: (90+80+90+60) / 4 = 80 √ case2: each node has three images ready but image1 is larger than image2. node1 (0, 80, 70, 60) - final score is: (80+70+60) / 4 = 52 node2 (90, 0, 70, 60) - final score is: (90 +70+60) / 4 = 55 √ thoughts? cc
Thanks can you update the the examples with the score that we would have gotten with the current function?
For reference, see this analysis for a different plugin
Also it's worth to see the difference with just capping per pod, instead of per container.
can you also describe the image sizes you used? the example in the original post assumed three images, while your analysis is assuming four.
agree with and the explanation in looks good but just to be sure it would be nice to see the math worked out with image sizes, and compared to the current formula
Sorry for the confusion, let me switch back to the original three images, update the image size a little bit to make it easier to understand, and also fit to the case of and . Here is the pod, ! and three nodes in the cluster, ! Let's see the difference between two approaches, capping per pod, step1: ! NODE1: scores = 30 2/3 + 1500 2/3 = 1020 NODE2: scores = 30 2/3 + 600 2/3 + 1500 2/3 = 1420 step2: ! NODE1: 1020 1000 (maxThreshold), cappedscore = 1000 NODE2: 1420 1000 (maxThreshold), cappedscore = 1000 step3: ! upperbound = 100 / lowerbound = 23 / maxscore = 100 NOTE1: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: score = 100 (1000 - 23) / (1000 - 23) = 100 potentially, NODE1 will be chose for scheduling. capping per images, step1: calculate image level priority, NODE1: image1: score = 30 2/3 = 20 image2: score = 0 image3: score = 1500 2/3 = 1000 NODE2: image1: score = 30 2/3 = 20 image2: score = 600 2/3 = 400 image3: score = 1500 2/3 = 1000 step2/3: cap each image's score with thresholds and calculate the image score, ! NODE1: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 0 image3: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 100 (400 - 23) / (1000 - 23) = 35 image3: score = 100 * (1000 - 23) / (1000 - 23) = 100 step4: final score NODE1: finalscore = 100 / 3 = 33 NODE2: final_score = 135 / 3 = 45 NODE2 will be chosen for scheduling.
Computing the score per image and normalizing by the number of images make sense to me at first glance, but I would like to see more examples. It would be much easier to digest the analysis if it was summarized in tabular form comparing three scoring functions: (1) current one, (2) one in and (3) multiple big images, multiple small images (total less than the threshold) and a mix. Also, compare cases where the spread is high vs low.
Yes, we don't need to see the individual steps, but a good set of scenarios with final scores. I didn't mean just the old function. We know it's very ineffective. What I meant is capping per pod using the strategy you used in
Did some comparison here, spread has been included based on the layout of each node, it's not easy for me to pull this factor out to focus on this factor only. I will design more testcases when the patch is ready for review which I hope could be more clear. column 1: pod with three images column 2: three nodes' images layout column 3 / 4 / 5: score of each node for different approaches. pod cap and normalizing per image
[10, 20, 30] & small [0,0,0] [800, 1200, 1500] & big [33, 59, 67] [30, 600, 1500] & mix [33, 45, 0]
Normalizing by the number of images gives each image the same "weight". Another variation of the third option is to try and multiply each image score with its size divided by the total size of all images, can you add that as another option?
I don't understand that suggestion. The score of an image is already proportional to its size. I would say the difference between the two suggestions is: Capping per pod cares about total downloaded size Capping per image gives roughly the same importance to each image. I think capping per pod makes more sense. However, there is some information we are losing in both suggestions, as seen in the first case. Even if a lot of small images need to be downloaded, we are giving a 0 score to all nodes. I think a cap from makes more sense, as suggested in the initial PR.
It helps giving more importance to larger images.
If the statistics are accurate with regards to image size distribution, then having many small images all scoring zero isn't common. For the more common cases (sizes between 32-1000), it seems that both approaches will likely give similar results, so I am fine with both approaches.
That would make the score proportional to the square of the size. Just to play safe (ie, not make the score too strong in a way that could affect spreading or resource allocation) I suggest we go with capping per pod. And then we can re-evaluate next release.
Speaking of statistics. They might have changed since this plugin was first developed. Is there a reliable source?
True for uncapped images, but everything we are doing is just heuristics, and the suggestion is just for the sake of exploring options, not advocating a specific one. Playing it safe sounds good to me.
This approach needs the smallest change and the minimum impact.
+1, this is not common case we can see that a pod with lots of small images (all < 23M), but a corner case that might be improved. No perfect solution so far, but either one is better than the original one IMHO.
/cc , Do you have any comments so far?
btw, has been updated to the second approach (capping and normalization per image).
I would go with this approach as well, it preserves the behavior for multiple small images, and improve it for multiple big ones.
seems like a simple enough compromise for now, and running through some of the examples with that formula it does get more of the specificity you're looking for with large images without normalizing all smaller images right to 0 (so we keep some of the detail there too). I would like to see some updated statistics on the thresholds, and while 90% of the images might fall between them I'm wondering if that's a totally normal distribution. suggestion in got me (tangentially) thinking if we could use thresholds that are more relevant to the pod in question, rather than all of Dockerhub. Because as it is now hypothetical pods (and clusters) which have all small images don't get much benefit from ImageLocality. But I recognize that could be intentional since download resources for small images is ultimately negligible.
I also believe that scoring of small images by zero is intentional, but seems like making lots of small images in pod accountable for the scheduling is slightly better.
It looks like we have consensus. Can you update the PR?
PR updated, pls kindly let me know if I miss something.

Discussion:
// The two thresholds are used as bounds for the image score range. They correspond to a reasonable size range for // container images compressed and stored in registries; 90%ile of images on dockerhub drops into this range. const ( <del> mb           int64 = 1024 * 1024 minThreshold int64 = 23 * mb maxThreshold int64 = 1000 * mb </del> <ins> mb                    int64 = 1024 * 1024 minThreshold          int64 = 23 * mb maxContainerThreshold int64 = 1000 * mb </ins> ) // ImageLocality is a score plugin that favors nodes that already have requested pod container's images.

} totalNumNodes := len(nodeInfos) <del> score := calculatePriority(sumImageScores(nodeInfo, pod.Spec.Containers, totalNumNodes)) </del> <ins> score := calculatePriority(sumImageScores(nodeInfo, pod.Spec.Containers, totalNumNodes), len(pod.Spec.Containers)) </ins> return score, nil }

// calculatePriority returns the priority of a node. Given the sumScores of requested images on the node, the node's // priority is obtained by scaling the maximum priority value with a ratio proportional to the sumScores. <del> func calculatePriority(sumScores int64) int64 { </del> <ins> func calculatePriority(sumScores int64, numContainers int) int64 { maxThreshold := maxContainerThreshold * int64(numContainers) </ins> if sumScores < minThreshold { sumScores = minThreshold } else if sumScores > maxThreshold {

Image: "gcr.io/10", }, { <del> Image: "gcr.io/2000", </del> <ins> Image: "gcr.io/4000", }, }, } test300600900 := v1.PodSpec{ Containers: []v1.Container{ { Image: "gcr.io/300", }, { Image: "gcr.io/600", }, { Image: "gcr.io/900", }, }, } test3040 := v1.PodSpec{ Containers: []v1.Container{ { Image: "gcr.io/30", }, { Image: "gcr.io/40", </ins> }, }, }

}, } <ins> node60040900 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/600:latest", }, SizeBytes: int64(600 * mb), }, { Names: []string{ "gcr.io/40:latest", }, SizeBytes: int64(40 * mb), }, { Names: []string{ "gcr.io/900:latest", }, SizeBytes: int64(900 * mb), }, }, } node300600900 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/300:latest", }, SizeBytes: int64(300 * mb), }, { Names: []string{ "gcr.io/600:latest", }, SizeBytes: int64(600 * mb), }, { Names: []string{ "gcr.io/900:latest", }, SizeBytes: int64(900 * mb), }, }, } node400030 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/4000:latest", }, SizeBytes: int64(4000 * mb), }, { Names: []string{ "gcr.io/30:latest", }, SizeBytes: int64(30 * mb), }, }, } node203040 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/20:latest", }, SizeBytes: int64(20 * mb), }, { Names: []string{ "gcr.io/30:latest", }, SizeBytes: int64(30 * mb), }, { Names: []string{ "gcr.io/40:latest", }, SizeBytes: int64(40 * mb), }, }, } </ins> nodeWithNoImages := v1.NodeStatus{} tests := []struct {

// Node2 // Image: gcr.io/250:latest 250MB <del> // Score: 100 * (250M/2 - 23M)/(1000M - 23M) = 100 </del> <ins> // Score: 100 * (250M/2 - 23M)/(1000M * 2 - 23M) = 5 </ins> pod:          &v1.Pod{Spec: test40250}, nodes:        []*v1.Node{makeImageNode("machine1", node403002000), makeImageNode("machine2", node25010)}, <del> expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: 10}}, </del> <ins> expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: 5}}, </ins> name:         "two images spread on two nodes, prefer the larger image one", }, {

// Node1 // Image: gcr.io/40:latest 40MB, gcr.io/300:latest 300MB <del> // Score: 100 * ((40M + 300M)/2 - 23M)/(1000M - 23M) = 15 </del> <ins> // Score: 100 * ((40M + 300M)/2 - 23M)/(1000M * 2 - 23M) = 7 </ins> // Node2 // Image: not present // Score: 0 pod:          &v1.Pod{Spec: test40300}, nodes:        []*v1.Node{makeImageNode("machine1", node403002000), makeImageNode("machine2", node25010)}, <del> expectedList: []framework.NodeScore{{Name: "machine1", Score: 15}, {Name: "machine2", Score: 0}}, </del> <ins> expectedList: []framework.NodeScore{{Name: "machine1", Score: 7}, {Name: "machine2", Score: 0}}, </ins> name:         "two images on one node, prefer this node", }, { <del> // Pod: gcr.io/2000 gcr.io/10 </del> <ins> // Pod: gcr.io/4000 gcr.io/10 </ins> // Node1 <del> // Image: gcr.io/2000:latest 2000MB // Score: 100 (2000M/2 >= 1000M, max-threshold) </del> <ins> // Image: gcr.io/4000:latest 2000MB // Score: 100 (4000 * 1/2 >= 1000M * 2, max-threshold) </ins> // Node2 // Image: gcr.io/10:latest 10MB // Score: 0 (10M/2 < 23M, min-threshold) pod:          &v1.Pod{Spec: testMinMax}, <del> nodes:        []*v1.Node{makeImageNode("machine1", node403002000), makeImageNode("machine2", node25010)}, </del> <ins> nodes:        []*v1.Node{makeImageNode("machine1", node400030), makeImageNode("machine2", node25010)}, </ins> expectedList: []framework.NodeScore{{Name: "machine1", Score: framework.MaxNodeScore}, {Name: "machine2", Score: 0}}, name:         "if exceed limit, use limit", }, { <del> // Pod: gcr.io/2000 gcr.io/10 </del> <ins> // Pod: gcr.io/4000 gcr.io/10 </ins> // Node1 <del> // Image: gcr.io/2000:latest 2000MB // Score: 100 * (2000M/3 - 23M)/(1000M - 23M) = 65 </del> <ins> // Image: gcr.io/4000:latest 4000MB // Score: 100 * (4000M/3 - 23M)/(1000M * 2 - 23M) = 66 </ins> // Node2 // Image: gcr.io/10:latest 10MB <del> // Score: 0 (10M/2 < 23M, min-threshold) </del> <ins> // Score: 0 (10M*1/3 < 23M, min-threshold) </ins> // Node3 // Image: // Score: 0 pod:          &v1.Pod{Spec: testMinMax}, <del> nodes:        []*v1.Node{makeImageNode("machine1", node403002000), makeImageNode("machine2", node25010), makeImageNode("machine3", nodeWithNoImages)}, expectedList: []framework.NodeScore{{Name: "machine1", Score: 65}, {Name: "machine2", Score: 0}, {Name: "machine3", Score: 0}}, </del> <ins> nodes:        []*v1.Node{makeImageNode("machine1", node400030), makeImageNode("machine2", node25010), makeImageNode("machine3", nodeWithNoImages)}, expectedList: []framework.NodeScore{{Name: "machine1", Score: 66}, {Name: "machine2", Score: 0}, {Name: "machine3", Score: 0}}, </ins> name:         "if exceed limit, use limit (with node which has no images present)", }, <ins> { // Pod: gcr.io/300 gcr.io/600 gcr.io/900 // Node1 // Image: gcr.io/600:latest 600MB, gcr.io/900:latest 900MB // Score: 100 * (600M * 2/3 + 900M * 2/3 - 23M) / (1000M * 3 - 23M) = 32 // Node2 // Image: gcr.io/300:latest 300MB, gcr.io/600:latest 600MB, gcr.io/900:latest 900MB // Score: 100 * (300M * 1/3 + 600M * 2/3 + 900M * 2/3 - 23M) / (1000M *3 - 23M) = 36 // Node3 // Image: // Score: 0 pod:          &v1.Pod{Spec: test300600900}, nodes:        []*v1.Node{makeImageNode("machine1", node60040900), makeImageNode("machine2", node300600900), makeImageNode("machine3", nodeWithNoImages)}, expectedList: []framework.NodeScore{{Name: "machine1", Score: 32}, {Name: "machine2", Score: 36}, {Name: "machine3", Score: 0}}, name:         "pod with multiple large images, machine2 is preferred", }, { // Pod: gcr.io/30 gcr.io/40 // Node1 // Image: gcr.io/20:latest 20MB, gcr.io/30:latest 30MB gcr.io/40:latest 40MB // Score: 100 * (30M + 40M * 1/2 - 23M) / (1000M * 2 - 23M) = 1 // Node2 // Image: 100 * (30M - 23M) / (1000M * 2 - 23M) = 0 // Score: 0 pod:          &v1.Pod{Spec: test3040}, nodes:        []*v1.Node{makeImageNode("machine1", node203040), makeImageNode("machine2", node400030)}, expectedList: []framework.NodeScore{{Name: "machine1", Score: 1}, {Name: "machine2", Score: 0}}, name:         "pod with multiple small images", }, </ins> } for _, test := range tests {