Change:
I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:

Discussion:
}, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 30 scheduler_pending_pods{queue="backoff"} 0

}, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 15 scheduler_pending_pods{queue="backoff"} 25

}, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 30 scheduler_pending_pods{queue="backoff"} 20

}, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 50 scheduler_pending_pods{queue="backoff"} 0

Subsystem:      SchedulerSubsystem, Name:           "schedule_attempts_total", Help:           "Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"result", "profile"}) e2eSchedulingLatency = metrics.NewHistogramVec( &metrics.HistogramOpts{ <ins> Subsystem:         SchedulerSubsystem, Name:              "e2e_scheduling_duration_seconds", DeprecatedVersion: "1.23.0", Help:              "E2e scheduling latency in seconds (scheduling algorithm + binding). This metric is replaced by scheduling_attempt_duration_seconds.", Buckets:           metrics.ExponentialBuckets(0.001, 2, 15), StabilityLevel:    metrics.ALPHA, }, []string{"result", "profile"}) schedulingLatency = metrics.NewHistogramVec( &metrics.HistogramOpts{ </ins> Subsystem:      SchedulerSubsystem, <del> Name:           "e2e_scheduling_duration_seconds", Help:           "E2e scheduling latency in seconds (scheduling algorithm + binding)", </del> <ins> Name:           "scheduling_attempt_duration_seconds", Help:           "Scheduling attempt latency in seconds (scheduling algorithm + binding)", </ins> Buckets:        metrics.ExponentialBuckets(0.001, 2, 15), <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"result", "profile"}) SchedulingAlgorithmLatency = metrics.NewHistogram( &metrics.HistogramOpts{

Help:      "Number of selected preemption victims", // we think #victims>50 is pretty rare, therefore [50, +Inf) is considered a single bucket. Buckets:        metrics.LinearBuckets(5, 5, 10), <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }) PreemptionAttempts = metrics.NewCounter( &metrics.CounterOpts{ Subsystem:      SchedulerSubsystem, Name:           "preemption_attempts_total", Help:           "Total preemption attempts in the cluster till now", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }) pendingPods = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem:      SchedulerSubsystem, Name:           "pending_pods", Help:           "Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ.", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"queue"}) SchedulerGoroutines = metrics.NewGaugeVec( &metrics.GaugeOpts{

metricsList = []metrics.Registerable{ scheduleAttempts, e2eSchedulingLatency, <ins> schedulingLatency, </ins> SchedulingAlgorithmLatency, PreemptionVictims, PreemptionAttempts,

func observeScheduleAttemptAndLatency(result, profile string, duration float64) { e2eSchedulingLatency.WithLabelValues(result, profile).Observe(duration) <ins> schedulingLatency.WithLabelValues(result, profile).Observe(duration) </ins> scheduleAttempts.WithLabelValues(result, profile).Inc() }

<ins> - name: pending_pods subsystem: scheduler help: Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. type: Gauge stabilityLevel: STABLE labels: - queue - name: preemption_attempts_total subsystem: scheduler help: Total preemption attempts in the cluster till now type: Counter stabilityLevel: STABLE - name: preemption_victims subsystem: scheduler help: Number of selected preemption victims type: Histogram stabilityLevel: STABLE buckets: - 5 - 10 - 15 - 20 - 25 - 30 - 35 - 40 - 45 - 50 - name: schedule_attempts_total subsystem: scheduler help: Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem. type: Counter stabilityLevel: STABLE labels: - profile - result - name: scheduling_attempt_duration_seconds subsystem: scheduler help: Scheduling attempt latency in seconds (scheduling algorithm + binding) type: Histogram stabilityLevel: STABLE labels: - profile - result buckets: - 0.001 - 0.002 - 0.004 - 0.008 - 0.016 - 0.032 - 0.064 - 0.128 - 0.256 - 0.512 - 1.024 - 2.048 - 4.096 - 8.192 - 16.384 </ins> - name: apiserver_request_duration_seconds help: Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.