Change:
Looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily. In particular, instead of computing the same information for a given node multiple times, we should cache it as part of NodeInfo: The information that should be cached and reuse later include: [x] node taints: [x] node conditions: disk/memory/inode pressure
Another quick optimization should be optimizing "selectHost" method: In fact it doesn't need to sort the whole list (which is O(nlgn). All we need to do is to choose the elements that have highest score, and sort only those by node name (to leave the logic that we currently have with round-robic logic of choosing from those).
And we should also limit the amount of places where we call "Parallelize" (because those are relatively expensive). The whole MR-like priorities is a step in this direction too.
OK - I've just sent out a PR fixing the first two things.
can we close this, or are there still work items here?
Some of those are not yet done.
- it's not a high priority for us (for now), so if you want it to be done soon, you probably need to find someone to do it.
- assigned.
Why "Parallelize" is considered as relatively expensive ? (comparing to MR style?)
You do MR when there are certain points that need some synchronizations. Or fully parallelized computation will always be faster.
To clarify, I was referring to the Parallelize function: Using this for many very small functions is pretty expensive - you can take a look into cpu-profiles. It happens that 20% or sth of cpu usage is spent on the operations on the synchronization channel.
Is all of the cacheing part completed? I assume "yes". What is remaining here.... Just an audit of parallelization ? Also, I assume "yes". If so I can audit where the parelellization is getting us hung up and then we can close this one out.
It's hard to tell if caching is done - when adding new priority functions/moving existing ones to MR-framework one pretty often discovers that more things need to be cached. There's my PR , that moves basic spreading priority function to MR-framework, which for some reason makes it worse. I didn't have time to look into that though. This needs to be done, which may require modifying MR-framework slightly (allowing different type as a Map phase result).
Im not sure what the MR Framework stuff is referring to. Can you provide me a pointer ? Should we close this issue since the original patches for caching were done? Sounds like the work we're referring to now is actually more like a second iteration. if not, what work remains to be done before we can close this ticket? maybe we can make a checklist.
"MR-style" parallelization, that was mentioned before. Yeah, I guess we can close this - this is more like an ongoing process thing, when everyone who adds anything to the scheduler needs to keep it in mind.

Discussion:
} func PodToleratesNodeTaints(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <del> node := nodeInfo.Node() if node == nil { return false, nil, fmt.Errorf("node not found") } taints, err := v1.GetTaintsFromNodeAnnotations(node.Annotations) </del> <ins> taints, err := nodeInfo.Taints() </ins> if err != nil { return false, nil, err }

// CheckNodeMemoryPressurePredicate checks if a pod can be scheduled on a node // reporting memory pressure condition. func CheckNodeMemoryPressurePredicate(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <del> node := nodeInfo.Node() if node == nil { return false, nil, fmt.Errorf("node not found") } </del> var podBestEffort bool if predicateMeta, ok := meta.(*predicateMetadata); ok { podBestEffort = predicateMeta.podBestEffort

// We couldn't parse metadata - fallback to computing it. podBestEffort = isPodBestEffort(pod) } // pod is not BestEffort pod if !podBestEffort { return true, nil, nil } <del> // is node under pressure? for _, cond := range node.Status.Conditions { if cond.Type == v1.NodeMemoryPressure && cond.Status == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderMemoryPressure}, nil } </del> <ins> // is node under presure? if nodeInfo.MemoryPressureCondition() == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderMemoryPressure}, nil </ins> } return true, nil, nil } // CheckNodeDiskPressurePredicate checks if a pod can be scheduled on a node // reporting disk pressure condition. func CheckNodeDiskPressurePredicate(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <del> node := nodeInfo.Node() if node == nil { return false, nil, fmt.Errorf("node not found") </del> <ins> // is node under presure? if nodeInfo.DiskPressureCondition() == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderDiskPressure}, nil </ins> } <del> // is node under pressure? for _, cond := range node.Status.Conditions { if cond.Type == v1.NodeDiskPressure && cond.Status == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderDiskPressure}, nil } } </del> return true, nil, nil }

// explicitly as int, to avoid conversions and improve performance. allowedPodNumber int <ins> // Cached tains of the node for faster lookup. taints    []v1.Taint taintsErr error // Cached conditions of node for faster lookup. memoryPressureCondition v1.ConditionStatus diskPressureCondition   v1.ConditionStatus </ins> // Whenever NodeInfo changes, generation is bumped. // This is used to avoid cloning it if the object didn't change. generation int64

return n.allowedPodNumber } <ins> func (n *NodeInfo) Taints() ([]v1.Taint, error) { if n == nil { return nil, nil } return n.taints, n.taintsErr } func (n *NodeInfo) MemoryPressureCondition() v1.ConditionStatus { if n == nil { return v1.ConditionUnknown } return n.memoryPressureCondition } func (n *NodeInfo) DiskPressureCondition() v1.ConditionStatus { if n == nil { return v1.ConditionUnknown } return n.diskPressureCondition } </ins> // RequestedResource returns aggregated resource request of pods on this node. func (n *NodeInfo) RequestedResource() Resource { if n == nil {

func (n *NodeInfo) Clone() *NodeInfo { clone := &NodeInfo{ <del> node:                n.node, requestedResource:   &(*n.requestedResource), nonzeroRequest:      &(*n.nonzeroRequest), allocatableResource: &(*n.allocatableResource), allowedPodNumber:    n.allowedPodNumber, generation:          n.generation, </del> <ins> node:                    n.node, requestedResource:       &(*n.requestedResource), nonzeroRequest:          &(*n.nonzeroRequest), allocatableResource:     &(*n.allocatableResource), allowedPodNumber:        n.allowedPodNumber, taintsErr:               n.taintsErr, memoryPressureCondition: n.memoryPressureCondition, diskPressureCondition:   n.diskPressureCondition, generation:              n.generation, </ins> } if len(n.pods) > 0 { clone.pods = append([]*v1.Pod(nil), n.pods...)

if len(n.podsWithAffinity) > 0 { clone.podsWithAffinity = append([]*v1.Pod(nil), n.podsWithAffinity...) } <ins> if len(n.taints) > 0 { clone.taints = append([]v1.Taint(nil), n.taints...) } </ins> return clone }

} } } <ins> n.taints, n.taintsErr = v1.GetTaintsFromNodeAnnotations(node.Annotations) for i := range node.Status.Conditions { cond := &node.Status.Conditions[i] switch cond.Type { case v1.NodeMemoryPressure: n.memoryPressureCondition = cond.Status case v1.NodeDiskPressure: n.diskPressureCondition = cond.Status default: // We ignore other conditions. } } </ins> n.generation++ return nil }

n.node = nil n.allocatableResource = &Resource{} n.allowedPodNumber = 0 <ins> n.taints, n.taintsErr = nil, nil n.memoryPressureCondition = v1.ConditionUnknown n.diskPressureCondition = v1.ConditionUnknown </ins> n.generation++ return nil }