Change:
What happened: We upgraded 15 kubernetes clusters from 1.17.5 to 1.18.2/1.18.3 and started to see that daemonsets does not work properly anymore. The problem is that all daemonset pods does not provision. It will return following error message to events: However, all nodes are available and it does not have node selector. Nodes does not have taints either. daemonset I have tried to restart apiserver/schedulers/controller-managers but it does not help. Also I have tried to restart that single node that is stuck (nodes-z2-1-kaasprod-k8s-local) but it does not help either. Only deleting that node and recreating it helps. We are seeing this randomly in all of our clusters. What you expected to happen: I expect that daemonset will provision to all nodes. How to reproduce it (as minimally and precisely as possible): No idea really, install 1.18.x kubernetes and deploy daemonset and after that wait days(?) Anything else we need to know?: When this happens we cannot provision any other daemonsets to that node either. Like you can see logging fluent-bit is also missing. I cannot see any errors in that node kubelet logs and like said, restarting does not help. Like you can see, most of the daemonsets are missing one pod Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: openstack OS (e.g: ): debian buster Kernel (e.g. ): Linux nodes-z2-1-kaasprod-k8s-local 4.19.0-8-cloud-amd64 SMP Debian 4.19.98-1+deb10u1 (2020-04-27) x86_64 GNU/Linux Install tools: kops Network plugin and version (if this is a network-related bug): calico Others:
/sig scheduling
Can you provide the full yaml of the node, daemonset, an example pod, and the containing namespace as retrieved from the server?
node: daemonset: example pod (working): example pod (not scheduling): namespace:
DaemonSet pods schedule with a nodeAffinity selector that only matches a single node, so the "12 out of 13 didn't match" message is expected.
I don't see a reason why the scheduler would be unhappy with the pod/node combo… there's no ports that could conflict in the podspec, the node is not unschedulable or tainted, and has sufficient resources
Okay I restarted all 3 schedulers (changed loglevel to 4 if we can see something interesting there). However, it fixed the issue now all daemonsets are provisioned correctly. Weird, anyways something wrong with the scheduler it seems
cc
We see same similar issue on v1.18.3, one node cannot be scheduled for daemonset pods. restart scheduler helps.
Hard to debug without knowing how to repreduce. Do you have the scheduler logs by any chance for the failed to schedule pod?
I assume only one of them is named , correct? Can you share what you noticed?
set loglevel to 9, but it seems there is nothing more interesting, below logs are looping.
yeah I could not see anything more than same line
the strange thing is that the log message is showing the result for 7 nodes only, like the issue reported in
/cc
this does look like the same issue I reported there, it seems like we either have a filter plugin that doesn't always report its error or some other condition that's failing silently if I had to guess
Note that in my issue, restarting the scheduler also fixed it (as mentioned in this thread too ) Mine was also about a daemonset, so I think this is a duplicate. If that's the case we can close this and continue discussion in
Anyways scheduler needs more verbose logging option, its impossible to debug these issues if there are not logs about what it does
+1, the scheduler could use significant improvements to its current logging abilities. That's an upgrade I've been meaning to tackle for a while and I've finally opened an issue for it here:
/assign I'm looking into this. A few questions to help me narrow the case. I haven't been able to reproduce yet. What was created first: the daemonset or the node? Are you using the default profile?
Do you have extenders?
nodes were created before the daemonset. suppose we used the default profile, which profile do you mean and how to check? no extenders. Another thing that may impact is the disk performance is not very good for etcd, etcd complains about slow operations.
Yes, those flags would make scheduler run with the default profile. I'll continue looking. I still couldn't reproduce.
Still nothing... Anything else you are using that you think could be impacting? taints, ports, other resources?
Made some tries related to this. When the issue is on, pods can still be scheduled to the node (without definition or with "nodeName" selector). If trying to use Affinity/Antiaffinity, pods doesn't get scheduled to node. Working when issue is on: Not working at the same time: Also when checked the latter's even those were quite interesting: First event is when manifest was just applied (nothing done to the non-schedulable node). Second and third were when node was removed with kubectl and then restarted. Fourth came when the node came back up. The node that had an issue was master, so node was not going there (but it shows that node was not found at 3 earlier events). Interesting thing with fourth event is that there's still information from one node missing. Event says there's 0/9 nodes available, but description is given only from 8.
"nodeName" is not a selector. Using nodeName would bypass scheduling. You are saying that the reason why the pod shouldn't have been scheduled in the missing node is because it was a master? We are seeing going to 7. I assume no nodes were removed at this point, correct?
"NodeName" try was to highligh, that node is usable and pod gets there if wanted. So thing is not node's unability to start pods. Test cluster has 9 nodes; 3 masters and 6 workers. Before the non-working node was successfully started, events told information about all available nodes: . But when that node that would match node selector came up, the event told Explanation tells that there's 8 that are not matching, but doesn't tell anything about the ninth (that was acknowledged on the previous event). So the event state: 1st event: 9 nodes available, the error noticed with daemonset 2nd and 3rd event: 8 nodes available. The one that was not receiving pod was restarting 4th event: 9 nodes available (so the one started that was restarted). At the end test pod wasn't started at the matching node because of taints, but that's other story (and should have been the case already at the 1st event).
Note that nothing guards against over-committing a node, but the scheduler. So this doesn't really show much. My question is: was the 9th node tainted from the beginning? I'm trying to look for (1) reproducible steps to reach the state or (2) where the bug could be.
Yes, taint was there all the time at this case, as the non-receiving node was master. But we have seen the same issue on both masters and workers. Still no idea where the issue comes from, just that at least recreation of node and restart of the node seem to be fixing the issue. But those are a bit "hard" ways to fix the things.
Long shot, but if you run into it again... could you check if there are any nominated pods to the node that doesn't show up?
I'm posting questions as I think of possible scenarios: Do you have other master nodes in your cluster? Do you have extenders?
All clusers have 3 masters (so restarting of those is easy) No. One interesting thing noticed today: I had cluster where one master was not receiving pod from DaemonSet. We have ChaosMonkey in use, which terminated one of the worker nodes. That's interesting, this made the pod to go to the master that was not receiving it earlier. So somehow removal of other node than the problematic one seemed to be fixing the issue at that point. Because of that "fix" I have to wait problem to reoccur to be able to answer about the nominated pods.
I'm confused now... Does your daemonset tolerate the taint for master nodes? In other words... is the bug for you just the scheduling event or also the fact that the pods should have been scheduled?
Issue is, that node is not found by scheduler even there's at least one matching affinity (or antiaffinity) settings. That's why I said that the taint error is expected, and should have been there already at the first event (as taint is not part of the affinity criteria)
Understood. I was trying to confirm your setup to make sure I'm not missing something. I don't think the node is "unseen" by the scheduler. Given that we see , we can conclude that the node is indeed in the cache. It's more like the unschedulable reason is lost somewhere, so we don't include it in the event.
True, total count matches always with actual node count. Just more descriptive event text is not given on all nodes, but that can be separate issue as you mentioned.
Are you able to look at your kube-scheduler logs? Anything that seems relevant?
I think tried to look for that without success. I can try when the issue occurs again (as well as that nominated pod thing asked earlier)
If possible, also run 1.18.5, in case we inadvertently fixed the issue.
I am able to reproduce this reliably on my test cluster if you need any more logs
Please share repro steps. Can you somehow identify what is the filter that is failing?
It appears to be just the of the node for the ds pod... weird. Here's the pod yaml: <details<summaryPod yaml:</summary<p</p</detailsThe way I reproduce this is by spinning up new cluster with 3 masters and 3 worker nodes (using Cluster API) and applying Cilium 1.7.6: <details<summaryCilium yaml:</summary<p</p</details
Here's the scheduler log: After restarting scheduler pods, the pending pod immediately schedules.
What pod events do you get? Do you know if there are taints in the node where it doesn't get scheduled? Does it only fail for master nodes or any nodes? Is there enough space in the node? On Thu., Jul. 9, 2020, 7:49 p.m. dilyevsky, <:
Could you try increasing the loglevel and using grep to filter for the node or the pod? On Thu., Jul. 9, 2020, 7:55 p.m. dilyevsky, <:
These are events: The node only has two taints but the pod tolerates all existing taints and yeah it seems to only happen on masters: There is enough space and pod is best effort with no reservation anyway: I'll try increasing scheduler log level now...
Your pod yaml doesn't actually have toleration. So it shouldn't have been scheduled in the master.
Hi! We are hitting the same issue. However, we see the same problem with deployments, where we use anti-affinity to make sure a pod gets scheduled on each node or a pod selector targeting the specific node. Simply creating a pod with a node selector set to match the hostname of the failing node was sufficient to cause the scheduling to fail. It was saying that 5 nodes were not matching the selector, but nothing about the sixth. Restarting the scheduler solved the issue. That looks like something gets cached about that node and prevents the scheduling on the node. As other people said before, we have nothing in the log about the failure. We striped the failing deployment to the bare minimum (we had removed the taint on the master that is failing): We were having the same issue when the master had a taint, and the deployment a toleration for the taint. So it does not seem related to daemonsets, tolerations or affinity / anti-affinity specifically. When the failure starts happening, nothing that targets the specific node can be scheduled. We see the issue in 1.18.2 up to 1.18.5 (did not try with 1.18.0 or .1)
Could you clarify if it starting failing after you created such pod or before? I assume this node didn't have a taint that the pod didn't tolerate. is going to help to reproduce. Could you look at the code for NodeSelector? You might need to add extra log lines while testing. You can also print the cache. Get PID of kube-scheduler: Trigger queue dump: . Note this won't kill the scheduler process. Then in scheduler log, search for strings "Dump of cached NodeInfo", "Dump of scheduling queue" and "cache comparer started". /priority critical-urgent
/unassign
We were already seeing some daemonset and deployment stuck in "Pending" before we tried deploying this test deployment, so it was already failing. and the taints had been removed from the node. Right now we lost the environment where this was happening because we had to reboot the nodes so the issue is not visible anymore. As soon as we reproduce, we will try to come back with more info
Please do so. I have tried to reproduce this in the past without success. I'm more interested in the first instance of failure. It might still be related to taints.
We have reproduced the issue. I ran the command you asked for, here are the info : Unfortunately that does not seem to help
Dumping the cache is for debugging, it won't change anything. Could you please include the dump?
Also, assuming this was the first error, could you include the pod yaml and node?
that's pretty much everything that was dumped, I just removed the other nodes. This was not the first error, but you can see coredns pod in the dump, that was the first one. I am not sure what else you are asking for in the dump. I'll fetch the yamls
Thanks, I didn't realize that you had trimmed the relevant node and pod.
Could you include the scheduled pods for that node though? Just in case there is a bug in resource usage calculations.
That seems odd.
Here are the other pods on that node : name: kube-controller-manager-master-0-bug, namespace: kube-system, uid: -4752-419b-aac7-, phase: Running, nominated node: name: kube-proxy-xwf6h, namespace: kube-system, uid: -9eb8-4584-ba3c-, phase: Running, nominated node: name: kube-apiserver-master-0-bug, namespace: kube-system, uid: -b0bc-4cef-9bad-, phase: Running, nominated node: name: kube-multus-ds-amd64-tpkm8, namespace: kube-system, uid: -599c-41d5-a029-, phase: Running, nominated node: name: openstack-cloud-controller-manager-wrb8n, namespace: kube-system, uid: -84a1-4416-a701-, phase: Running, nominated node: name: kube-scheduler-master-0-bug, namespace: kube-system, uid: -3122-4e99-92f6-, phase: Running, nominated node: name: subport-controller-28j9v, namespace: kube-system, uid: -763a-4ff2-bdae-, phase: Running, nominated node: name: csi-cinder-controllerplugin-0, namespace: kube-system, uid: -a871-454e-98a3-, phase: Running, nominated node: name: calico-node-d899t, namespace: kube-system, uid: -53b1-4356-a5df-, phase: Running, nominated node: ``
All the nodes have allowedPodNumber set to 0 in the requested resources in the dump, but the other nodes are schedulable
The node yaml : and the pod :
Thanks a lot for all the information. can you take it?
We're also trying with now, to get more info
/help feel free to take this and submit a PR if you find the bug. The log lines you are likely to be helpful. Otherwise, I'm opening to contributors.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Is there anything specific to timing when this issue occurs first time? For example, does it happen right after node starts?
No, there quite some pods that get scheduled there and run fine. But once the issue happens no of can be scheduled anymore.
Lowering priority until we have a reproducible case.
We were able to reproduce the bug with a scheduler that had the additional log entries. What we see is that one of the masters completely disappears from the list of nodes that are iterated through. We can see that the process starts with the 6 nodes (from the snapshot) : but after, that we can see that it iterates only over 5 nodes, and then we get : So one of the nodes is removed from the list of potential nodes. Unfortunately we did not have enough loging at the start of the process, but we'll try to get more.
Code references by log line: Did you see any lines for ? Otherwise, could you check for bugs on ?
No, that log did not appear. I would also think it could be that we either have an issue with the parallelization or that node is filtered out earlier. If it was failing with an error here : it would be visible in the logs afaik, so I'll try to add more debug entries around specifically the function and the parallelization.
I just realized that one node is going through the filtering twice!
The logs are : As you can see the node worker-pool1-60846k0y-scheduler goes twice through filtering
Yeah, an error there would manifest as an Scheduling Error in the pod events. I honestly wouldn't think that parallelization has bugs (still worth checking), but this could be a sign that we failed to build the snapshot from the cache (as we saw from the cache dump, the cache is correct), by adding a node twice. Since statuses is a map, then it makes sense that we only "see" 5 nodes at the last log line. This is the code (tip of 1.18)
cc
I will try to add a lot of logs on the cache part of the scheduler, specifically around node addition and update, and around the snapshot. However, from the last line of the logs, you can see that the snapshot is actually correct, and contains all the nodes, so whatever happens seem to happen later on, when working over that snapshot
cache != snapshot Cache is the living thing that gets updated from events. The snapshot is updated (from the cache) before each scheduling cycle to "lock" the state. We optimizations to make this last process as fast as possible. It's possible that the bug is there.
Thanks This is very useful. Your logs indicate that is duplicated in the list already at before any parallel code gets executed. That would indeed mean that it's duplicated before the snapshot is updated.
Actually, that comes from the snapshot, that happens before this log message : . So it rather looks like the content of the snapshot has the duplication, since it comes from
That's right. I meant it's already duplicated before the update of the snapshot finishes.
No, The snapshot is updated at the start of the scheduling cycle. The bug is either during snapshot update or before that. But the cache is correct, according to the dump in EDIT: I read it wrong, I didn't see the word "finishes" :)
The PR optimizing update snapshot was done in 1.18: and
I wonder if the node tree also has duplicate records
could you show a dump of the full list of nodes in the cache?
we don't add/remove items from NodeInfoList, we either create the full list from the tree or not, so if there are duplicates, those are likely coming from the tree, I think.
Just to clarify: 1) the cluster has 6 nodes (including the masters) 2) the node that is supposed to host the pod wasn't examined at all (no log line indicating that), which may mean it is not in NodeInfoList at all 3) NodeInfoList has 6 nodes, but one of them is duplicate
a dump of each the node tree, list and map would be great.
I'll work on getting those. In the meantime, a small update. We can see in the logs : And that is the exact point when the missing node disappears. The last occurence in the logs is at 13:37:24. In the next scheduling, the missing node is gone. so it looks like the bug is in /follows the update of the node_tree. All nodes go through that update, it's just that this worker 608 is the last one to go through it. When dumping the cache (with SIGUSR2) all six nodes are listed there, with the pods running on the nodes, without duplication or missing nodes.
We'll give it a new try with debug around the snapshot functionality :
Interesting, I think the remove/add are triggered by an updateNode call. The zone key is missing on the remove, but exist on the add, so the update was basically adding the zone and region labels? Do you have other scheduler logs related to this node?
We're trying to reproduce the bug with the logging. I'll come back when I have more info
yes, it happened for all nodes, and it is expected. the coincidence being that this node specifically is the last updated one, and it is at that exact time that the other node goes missing
Did you get update logs for the missing node?
lol, was just typing this question.
perhaps the bug is that whole zone is deleted from the tree before all nodes are removed.
Just to clarify, I'm not personally looking at the code, I'm just trying to make sure we have all the information. And I think that, with what we have now, we should be able to spot the bug. Feel free to submit PRs and much better if you can provide a unit test that fails.
yes, it shows that the zone is updated for that missing node. There is a log entry for all nodes To be honest, I still have no clue about the reason of the bug, but if we can get close to finding it out, I'll submit a PR or unit tests.
If so, then I think assuming that this "is the exact point when the missing node disappears." may not be correlated. Lets wait for the new logs. It would be great if you can share all scheduler logs you get in a file.
I'll do when we reproduce with the new logging. From the existing ones, we can actually see that the pod scheduling immediately after that update is the first one to fail. But it does not give enough information to know what happened in between, so stay tuned...
Have you seen a message starting with in the scheduler logs? Would it be possible for you to provide full scheduler logs?
no, that message is not present. I could give a striped down log file (to avoid repetition), but let's first wait until we have the output with more logs around the snapshot
I have found the bug. The issue is with the nodeTree next() function, that does not return a list of all nodes in some cases. It is visible if you add the following here : The main problem is that when you add a node, the indexes are not at 0 for some of the zones. For this to happen you must have at least two zones, one being shorter than the other, and the longer one having an index not set to 0 when calling the next function for the first time. The fix I went with is to reset the index before calling next() the first time. I opened a PR to show my fix. Of course it is against the 1.18 release as this is what I have been working on, but it is mostly for discussing how to fix it (or maybe fix the next() function itself). I can open a proper PR towards master and do the backports if needed afterwards.
I noticed the same problem with iteration. But I failed to link that to a duplicate in the snapshot. Have you managed to create a scenario where that would happen,
yes, you can run it in the unit tests by adding the small code I put
I am now working on adding a test case for the snapshot, to make sure this is properly tested.
big thumbs up to for the help in reproducing the issue and running the tests in his setup
Thanks all for debugging this notorious issue. Resetting the index before creating the list is safe, so I think we should go with that for 1.18 and 1.19 patches, and have a proper fix in the master branch. The purpose of the function changed with the introduction of the NodeInfoList, and so we can certainly simplify it and perhaps change it to , a function that creates a list from the tree and simply start from the beginning every time.
I understand the issue now: The calculation of whether or not a zone is exhausted is wrong because it doesn't consider where in each zone we started this "UpdateSnapshot" process. And yeah, it would only be visible with uneven zones. Great job spotting this I would think we have the same issue in older versions. However, it is hidden by the fact that we do a tree pass every time. Whereas in 1.18 we snapshot the result until there are changes in the tree. Now that the round-robin strategy is implemented in , we might be fine with simply resetting all counters before UpdateSnapshot, as your PR is doing. Just to double check this should be fine even in a cluster were new nodes are /removed all the time, right?
Thanks for spotting the root cause! Given that is only called in building the snapshot nodeInfoList, I think it's also safe to remove the indexes (both zoneIndex and nodeIndex) from nodeTree struct. Instead, come up with a simple function to iterate through its zone/node in a round-robin manner.
BTW: there is a typo in , the case should be:
I am assuming you are talking about the logic in , if so yes, it is doesn't matter much if nodes were or removed, the main thing we need to avoid is iterating over the nodes in the same order every time we schedule a pod, we just need a good approximation of iterating over the nodes across pods.
yes, we just need to iterate over all zones/nodes in the same order every time.
I have updated the PR with a unit test for the function updating the snapshotlist, for that bug specifically. I can also take care of refactoring the next() function to iterate over the zones and nodes without round-robin, hence removing the issue.
Thanks, sounds good, but we should still iterate between zones the same way we do now, that is by design.
I don't really get what you mean here. Is it so that the order of the nodes matter and we must still go round-robin between zones or can we list all nodes of a zone, one zone after the other ? Let's say that you have two zones of two nodes each, in which order do you expect them, or does it even matter at all ?
The order matters, we need to alternate between zones while creating the list. If you have two zones of two nodes each and , then the list should be
ok, thanks, I'll give it a thought. Can we meanwhile proceed with the quick fix ? btw, some tests are failing on it, but I am not sure how that relates to my PR
Those are flakes. Please send a patch to 1.18 as well.
Ok, will do. Thanks
do you mean this test ignore the 'node-5'? I found after fixed the append in , the test result all the nodes can be iterated: The node-5, 6, 7, 8, 3 can be iterated. Forgive me if misunderstand something here.
yes, it was on purpose, based on what was there, but I can see how this can be cryptic, so better make it so that the append is behaving in a clearer way. Thanks for the patch.
How far back do you believe this bug was present? 1.17? 1.16? I've just seen the exact same problem in 1.17 on AWS and restarting the unscheduled node fixed the problem.
could you provide more details? Log lines, cache dumps, etc. So we can determine whether the issue is the same. As I noted in , I believe this bug was present in older versions, but my thinking is that it's transient. would you be able to investigate?
Please also share the distribution of nodes in the zones.
unfortunately I can't at this point. Sorry.
sorry, I already rebuilt the cluster for other reasons, but it may have been a network configuration problem related to multi-az deployments, and in what subnet the faulty node got launched, so I wouldn't worry about it for now in the context of this issue. If I notice it again I'll report back with better details. Thanks!
/retitle Some nodes are not considered in scheduling when there is zone imbalance

Discussion:
if updateAll { // Take a snapshot of the nodes order in the tree snapshot.nodeInfoList = make([]*framework.NodeInfo, 0, cache.nodeTree.numNodes) <ins> cache.nodeTree.resetExhausted() </ins> for i := 0; i < cache.nodeTree.numNodes; i++ { nodeName := cache.nodeTree.next() if n := snapshot.nodeInfoMap[nodeName]; n != nil {

return nil } <ins> func TestSchedulerCache_updateNodeInfoSnapshotList(t *testing.T) { // Create a few nodes to be used in tests. nodes := []*v1.Node{} i := 0 // List of number of nodes per zone, zone 0 -> 2, zone 1 -> 6 for zone, nb := range []int{2, 6} { for j := 0; j < nb; j++ { nodes = append(nodes, &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: fmt.Sprintf("node-%d", i), Labels: map[string]string{ v1.LabelZoneRegion:        fmt.Sprintf("region-%d", zone), v1.LabelZoneFailureDomain: fmt.Sprintf("zone-%d", zone), }, }, }) i++ } } var cache *schedulerCache var snapshot *Snapshot addNode := func(t *testing.T, i int) { if err := cache.AddNode(nodes[i]); err != nil { t.Error(err) } _, ok := snapshot.nodeInfoMap[nodes[i].Name] if !ok { snapshot.nodeInfoMap[nodes[i].Name] = cache.nodes[nodes[i].Name].info } } updateSnapshot := func(t *testing.T) { cache.updateNodeInfoSnapshotList(snapshot, true) if err := compareCacheWithNodeInfoSnapshot(cache, snapshot); err != nil { t.Error(err) } } tests := []struct { name       string operations func(t *testing.T) expected   []string }{ { name:       "Empty cache", operations: func(t *testing.T) {}, expected:   []string{}, }, { name: "Single node", operations: func(t *testing.T) { addNode(t, 0) }, expected: []string{"node-0"}, }, { name: "Two nodes", operations: func(t *testing.T) { addNode(t, 0) updateSnapshot(t) addNode(t, 1) }, expected: []string{"node-0", "node-1"}, }, { name: "bug 91601, two nodes, update the snapshot and add two nodes in different zones", operations: func(t *testing.T) { addNode(t, 2) addNode(t, 3) updateSnapshot(t) addNode(t, 4) addNode(t, 0) }, expected: []string{"node-2", "node-0", "node-3", "node-4"}, }, { name: "bug 91601, 6 nodes, one in a different zone", operations: func(t *testing.T) { addNode(t, 2) addNode(t, 3) addNode(t, 4) addNode(t, 5) updateSnapshot(t) addNode(t, 6) addNode(t, 0) }, expected: []string{"node-2", "node-0", "node-3", "node-4", "node-5", "node-6"}, }, { name: "bug 91601, 7 nodes, two in a different zone", operations: func(t *testing.T) { addNode(t, 2) updateSnapshot(t) addNode(t, 3) addNode(t, 4) updateSnapshot(t) addNode(t, 5) addNode(t, 6) addNode(t, 0) addNode(t, 1) }, expected: []string{"node-2", "node-0", "node-3", "node-1", "node-4", "node-5", "node-6"}, }, { name: "bug 91601, 7 nodes, two in a different zone, different zone order", operations: func(t *testing.T) { addNode(t, 2) addNode(t, 1) updateSnapshot(t) addNode(t, 3) addNode(t, 4) updateSnapshot(t) addNode(t, 5) addNode(t, 6) addNode(t, 0) }, expected: []string{"node-2", "node-1", "node-3", "node-0", "node-4", "node-5", "node-6"}, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { cache = newSchedulerCache(time.Second, time.Second, nil) snapshot = NewEmptySnapshot() test.operations(t) // Always update the snapshot at the end of operations and compare it. cache.updateNodeInfoSnapshotList(snapshot, true) if err := compareCacheWithNodeInfoSnapshot(cache, snapshot); err != nil { t.Error(err) } nodeNames := make([]string, len(snapshot.nodeInfoList)) for i, nodeInfo := range snapshot.nodeInfoList { nodeNames[i] = nodeInfo.Node().Name } if !reflect.DeepEqual(nodeNames, test.expected) { t.Errorf("The nodeInfoList is incorrect. Expected %v , got %v", test.expected, nodeNames) } }) } } </ins> func BenchmarkUpdate1kNodes30kPods(b *testing.B) { // Enable volumesOnNodeForBalancing to do balanced resource allocation defer featuregatetesting.SetFeatureGateDuringTest(nil, utilfeature.DefaultFeatureGate, features.BalanceAttachedNodeVolumes, true)()