Change:
/sig scheduling A way to check the scoring of each node to figure out why a node has a high scheduling priority. proposal 1: add some debug level logs to Kube-scheduler, so we can know node score on each scoring step. proposal 2: a simple way to decrease the priority of one node(in my case a node with more than 200 pods is with the highest scheduling priority for an unknown reason) proposal 3:? All new pods are scheduled to one of five nodes in my cluster. I have checked both CPU/memory requested usage and all five nodes are bare metal and running about 150 pods or 200 pods. Pods on node request 80-93% of the memory. We have a similar issue as Currently, I think is the easiest way.
This is useful. how do you suggest we log this information? note that there are two dimensions for each pod: plugin and node score. Also, we need to limit the log length, we can't simply log the score for each node in clusters with large number of nodes.
I prefer to log only the top 5 or 10 node scores in the log. (I have not read the code yet., will dig into it later.)
We actually merged something like this that I wrote in , but had to revert it because of performance issue in large clusters. That could be a good starting point My general approach was to 1) invert the already available to map (rather than , which it currently does) and then 2) log the top 3 highest scoring plugins for each node However discovered performance issues after this in (we believe due to the many writes this process took) and it was reverted in I think we decided that some performance hit is reasonable, especially given log levels =4, but this was ~50% drop.
thanks for the information. So with current kube-scheduler, I can set log level to 10 to get every node score for each plugin. It helps in some cases.
yes, that's the current case. Though creates a lot of other noise in the logs, so I think we agreed that it is okay for us to add this kind of summary at a lower level like
I think we can use this chance to standardize the logging logic to dump scores dynamically depending on log level: for logging level ranged in , dump the scores for topM nodes/plugins. And make it as performant as possible. for logging level ranged in , dump the scores for topN nodes, and probably show all plugins for logging level = 10, dump the scores for all nodes/plugins.
/assign
/triage accepted
/reopen Some problems that were mentioned by are not fixed yet. /cc
Reopened this issue. <detailsIn response to : /reopen After taking a glance, I can spot several issues: A consensus on the value and . This is not fully discussed yet. Further dig into the performance result. The test runs on v=1 and v=4, but actually they're the same if it's running in the context of integration framework. I'd expect to see the result comparing before/after on v=4 and v=6. Further review on the code, like adding UTs and some logic nits. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since is merged to revert the change,
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Just FYI: this may be related to in the future. In , we cannot add all scores to the event message (because the event msg has the limit on the length), so I am thinking we need to sort node-scores by total score and add the score of top N nodes to the event msg. And, if we do so, we can easily achieve to log the top M Node's score by using this sorted node-scores.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/lifecycle rotten
/remove-lifecycle rotten
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
package scheduler import ( <ins> "container/heap" </ins> "context" "fmt" "math/rand" <ins> "sort" </ins> "sync" "sync/atomic" "time"

// to ensure that a certain minimum of nodes are checked for feasibility. // This in turn helps ensure a minimum level of spreading. minFeasibleNodesPercentageToFind = 5 <ins> // minNodeScoresToDump is the number of nodes' score to dump when the logging // level ranged in [4, 6) minNodeScoresToDump = 3 // minPluginScoresToDump is the number of plugins' score to dump when the // logging level ranged in [4, 6) minPluginScoresToDump = 3 // moreNodeScoresToDump is the number of nodes score to dump when the // logging level ranged in [6, 10), and all plugins' score will be dumped moreNodeScoresToDump = 10 </ins> ) // ErrNoNodesAvailable is used to describe the error that no nodes available to schedule pods.

nextStartNodeIndex       int } <ins> type pluginScore struct { Name  string Score int64 } type pluginScoreList []pluginScore // NodeToPluginScores declares a map from node name to its PluginScoreList. type nodeToPluginScores map[string]pluginScoreList </ins> // snapshot snapshots scheduler cache and node infos for all fit and priority // functions. func (g *genericScheduler) snapshot() error {

pod *v1.Pod, nodes []*v1.Node, ) (framework.NodeScoreList, error) { // If no priority configs are provided, then all nodes will have a score of one. // This is required to generate the priority list in the required format if len(extenders) == 0 && !fwk.HasScorePlugins() {

return nil, scoreStatus.AsError() } <del> if klog.V(10).Enabled() { for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) } } } </del> // Summarize all scores. result := make(framework.NodeScoreList, 0, len(nodes))

} } <ins> if klog.V(4).Enabled() { logNodeAndPluginScores(result, scoresMap, pod) } </ins> if len(extenders) != 0 && nodes != nil { var mu sync.Mutex var wg sync.WaitGroup

percentageOfNodesToScore: percentageOfNodesToScore, } } <ins> // To expose more information about the scheduling process but not has a big drop on performance, // we use a dynamic strategy to dump depending on the logging level: // * for logging level ranged in [4, 6), dump the scores for topM nodes/plugins. And make it as performant as possible. // * for logging level ranged in [6, 10), dump the scores for topN nodes, and probably show all plugins // * for logging level >= 10, dump the scores for all nodes/plugins. func logNodeAndPluginScores(nodeScores framework.NodeScoreList, scoresMap framework.PluginToNodeScores, pod *v1.Pod) { if klog.V(10).Enabled() { for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) } } return } // Log the top M node scores for the pod topMNodes := minNodeScoresToDump if klog.V(6).Enabled() { topMNodes = moreNodeScoresToDump } // Use the min-heap to maintain a top M node score list nsh := make(nodeScoreHeap, 0, topMNodes) for i, nodeScore := range nodeScores { if len(nsh) < int(topMNodes) || nodeScore.Score > nsh[0].Score { if len(nsh) == int(topMNodes) { heap.Pop(&nsh) } heap.Push(&nsh, &nodeScores[i]) } } // Build a map of Nodes->PluginScores on that node nodeScoresMap := make(nodeToPluginScores, topMNodes) for _, nodeScore := range nsh { nodeScoresMap[nodeScore.Name] = make(pluginScoreList, 0) } // Convert the scoresMap (which contains Plugins->NodeScores) to the Nodes->PluginScores map for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { // Get the top M nodes' plugin scores if _, ok := nodeScoresMap[nodeScore.Name]; ok { nodeScoresMap[nodeScore.Name] = append(nodeScoresMap[nodeScore.Name], pluginScore{Name: plugin, Score: nodeScore.Score}) } } } // Log the top N plugin scores for the top M nodes topNPlugins := minPluginScoresToDump if klog.V(6).Enabled() { // Log all plugins' socre if -v6 topNPlugins = -1 } for name, pluginScores := range nodeScoresMap { sort.Slice(pluginScores, func(i int, j int) bool { return pluginScores[i].Score > pluginScores[j].Score }) // Get the top N plugin socres only if logging level ranged in [4, 6) if topNPlugins > 0 && len(pluginScores) > int(topNPlugins) { nodeScoresMap[name] = pluginScores[:topNPlugins] } } scoresMessage := fmt.Sprintf("Top %d plugins for pod on node", topMNodes) for len(nsh) > 0 { nodeScore := heap.Pop(&nsh).(*framework.NodeScore) pluginScores := nodeScoresMap[nodeScore.Name] klog.InfoS(scoresMessage, "pod", klog.KObj(pod), "node", nodeScore.Name, "score", nodeScore.Score, "plugins", pluginScores) } } // nodeScoreHeap is a min-heap ordered by the score of nodes. The // scheduler uses this to find the top-N scoring nodes when the // logging verbose level is ranged in [4, 10) type nodeScoreHeap []*framework.NodeScore // var _ heap.Interface = &nodeScoreHeap{} func (nsh nodeScoreHeap) Len() int { return len(nsh) } func (nsh nodeScoreHeap) Less(i, j int) bool { return nsh[i].Score < nsh[j].Score } func (nsh nodeScoreHeap) Swap(i, j int) { nsh[i], nsh[j] = nsh[j], nsh[i] } func (nsh *nodeScoreHeap) Push(v interface{}) { *nsh = append(*nsh, v.(*framework.NodeScore)) } func (nsh *nodeScoreHeap) Pop() interface{} { ns := (*nsh)[nsh.Len()-1] *nsh = (*nsh)[:nsh.Len()-1] return ns } </ins>