Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --As of now in , we have 3 metrics, for end2endLatency, algorithmLatency and BindingLatency. The problem with algorithmLatency is it is a combination of predicate and priority latencies. So breaking down this metric into predicate and priority latencies would be really helpful for performance analysis. For example, as part of the analysis after these metrics were , we were able to find latencies when we have taints & tolerations alone. In fact, ability to drill down on every predicate and priority would be ideal but want to know other's opinions on this.
/cc
+1 please send a PR.
We also have some demonstrations of how this helps with things like highlighting taint and toleration performance and so on. We should update readme info on how to use these along side the scheduler perf stuff as we've done internally .
You need to be careful about what latency means if we are computing multiple different predicates in parallel (if we decide to split it by predicate). Splitting predicate vs priority is definitely a good idea.
Please label this as part of our 1.7 scheduling forensics milestone .
Let's remove the idea of latency entirely, it is confusing - this is just an attempt at measuring predicate and priority timescales, and there is no response latency that we care about in those small (20ms) timescales , (with the exception of maybe the latency of an etcd hit that might happen in a client if the cache is missing information I guess)
Not really - there were cases of predicates which computation could take over a second.
Yes, and even longer on large clusters with things light affinity.... However here I'm just disputing use of the term latency since latency is usually used to refer to the time between the end of a request and the beggining of a response. What we endeavor to do here is really measure round trip execution time, for predicates, in isolation. If we want to measure some kind of latency then I believe we would need to define some notion of a request/response, which has diminished value outside of something like RPC. If we truly were measuring latency (and it might be interesting to do so with tims tracing work) then we might thus look at the time between writing a new unscheduled pod to the system, and the time it takes to start running the scheduler fit functions or something... Tl;dr there are a lot of varying usages and definitions for latency, but almost all of them, with the exception of measurements of virtual function calls, refer to some unavoidable minimum cost that occurs during distributed communication, and in this case, we are simply measuring "execution time of the predicate filter" which is called from inside the same process. (Alternatively I guess it's ok to call it "fitfunctionpredicate_latency", I guess, since then we are using the fit function as the context for a response measurement).
OK, IIUC you object usage of the word 'latency', right? We use it in this context for quite some time, and as I agree that it doesn't fit formal definition, there's no point in changing it now, as it would confuse people even more.
yeah, i really think they should all be refactored to not be named as such, that is my broader point . but if we want to keep those metrics id suggest we at least update the definition in the fields to clarify that latency is defined WRT rather then to a RPC or some other typical latency calc scenario. aplogies for the bike shed, ill move this to a PR :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale

Discussion:
"//pkg/scheduler/algorithm/predicates:go_default_library", "//pkg/scheduler/algorithm/priorities/util:go_default_library", "//pkg/scheduler/api:go_default_library", <ins> "//pkg/scheduler/metrics:go_default_library", </ins> "//pkg/scheduler/schedulercache:go_default_library", "//pkg/scheduler/util:go_default_library", "//pkg/scheduler/volumebinder:go_default_library",

"k8s.io/kubernetes/pkg/scheduler/algorithm" "k8s.io/kubernetes/pkg/scheduler/algorithm/predicates" schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" <ins> "k8s.io/kubernetes/pkg/scheduler/metrics" </ins> "k8s.io/kubernetes/pkg/scheduler/schedulercache" "k8s.io/kubernetes/pkg/scheduler/util"

} trace.Step("Computing predicates") <ins> startPredicateEvalTime := time.Now() </ins> filteredNodes, failedPredicateMap, err := findNodesThatFit(pod, g.cachedNodeInfoMap, nodes, g.predicates, g.extenders, g.predicateMetaProducer, g.equivalenceCache, g.schedulingQueue) if err != nil { return "", err

FailedPredicates: failedPredicateMap, } } <ins> metrics.SchedulingAlgorithmPredicateEvaluationDuration.Observe(metrics.SinceInMicroseconds(startPredicateEvalTime)) </ins> trace.Step("Prioritizing") <ins> startPriorityEvalTime := time.Now() </ins> // When only one node after predicate, just use it. if len(filteredNodes) == 1 { <ins> metrics.SchedulingAlgorithmPriorityEvaluationDuration.Observe(metrics.SinceInMicroseconds(startPriorityEvalTime)) </ins> return filteredNodes[0].Name, nil }

if err != nil { return "", err } <ins> metrics.SchedulingAlgorithmPriorityEvaluationDuration.Observe(metrics.SinceInMicroseconds(startPriorityEvalTime)) </ins> trace.Step("Selecting host") return g.selectHost(priorityList)

const schedulerSubsystem = "scheduler" <del> var BindingSaturationReportInterval = 1 * time.Second </del> <ins> // All the histogram based metrics have 1ms as size for the smallest bucket. </ins> var ( E2eSchedulingLatency = prometheus.NewHistogram( prometheus.HistogramOpts{

Buckets:   prometheus.ExponentialBuckets(1000, 2, 15), }, ) <ins> SchedulingAlgorithmPredicateEvaluationDuration = prometheus.NewHistogram( prometheus.HistogramOpts{ Subsystem: schedulerSubsystem, Name:      "scheduling_algorithm_predicate_evaluation", Help:      "Scheduling algorithm predicate evaluation duration", Buckets:   prometheus.ExponentialBuckets(1000, 2, 15), }, ) SchedulingAlgorithmPriorityEvaluationDuration = prometheus.NewHistogram( prometheus.HistogramOpts{ Subsystem: schedulerSubsystem, Name:      "scheduling_algorithm_priority_evaluation", Help:      "Scheduling algorithm priority evaluation duration", Buckets:   prometheus.ExponentialBuckets(1000, 2, 15), }, ) </ins> BindingLatency = prometheus.NewHistogram( prometheus.HistogramOpts{ Subsystem: schedulerSubsystem,

prometheus.MustRegister(E2eSchedulingLatency) prometheus.MustRegister(SchedulingAlgorithmLatency) prometheus.MustRegister(BindingLatency) <ins> prometheus.MustRegister(SchedulingAlgorithmPredicateEvaluationDuration) prometheus.MustRegister(SchedulingAlgorithmPriorityEvaluationDuration) </ins> }) }