Change:
In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.
I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works. We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .

Discussion:
return hist.GetSampleSum() / float64(hist.GetSampleCount()) } <del> // Clear clears all fields of the wrapped histogram func (hist *Histogram) Clear() { if hist.SampleCount != nil { *hist.SampleCount = 0 } if hist.SampleSum != nil { *hist.SampleSum = 0 } for _, b := range hist.Bucket { if b.CumulativeCount != nil { *b.CumulativeCount = 0 } } } </del> // Validate makes sure the wrapped histogram has all necessary fields set and with valid values. func (hist *Histogram) Validate() error { if hist.SampleCount == nil || hist.GetSampleCount() == 0 {

} } <del> func TestHistogramClear(t *testing.T) { samples := []float64{0.5, 0.5, 0.5, 0.5, 1.5, 1.5, 1.5, 1.5, 3, 3, 3, 3, 6, 6, 6, 6} bounds := []float64{1, 2, 4, 8} h := samples2Histogram(samples, bounds) if *h.SampleCount == 0 { t.Errorf("Expected histogram .SampleCount to be non-zero") } if *h.SampleSum == 0 { t.Errorf("Expected histogram .SampleSum to be non-zero") } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount == 0 { t.Errorf("Expected histogram bucket to have non-zero comulative count") } } } h.Clear() if *h.SampleCount != 0 { t.Errorf("Expected histogram .SampleCount to be zero, have %v instead", *h.SampleCount) } if *h.SampleSum != 0 { t.Errorf("Expected histogram .SampleSum to be zero, have %v instead", *h.SampleSum) } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount != 0 { t.Errorf("Expected histogram bucket to have zero comulative count, have %v instead", *b.CumulativeCount) } } if b.UpperBound != nil { *b.UpperBound = 0 } } } </del> func TestHistogramValidate(t *testing.T) { tests := []struct { name string

"k8s.io/client-go/restmapper" "k8s.io/component-base/featuregate" featuregatetesting "k8s.io/component-base/featuregate/testing" <ins> "k8s.io/component-base/metrics/legacyregistry" </ins> "k8s.io/klog/v2" "k8s.io/kubernetes/test/integration/framework" testutils "k8s.io/kubernetes/test/utils"

defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, feature, flag)() } dataItems.DataItems = append(dataItems.DataItems, runWorkload(b, tc, w)...) <ins> // Reset metrics to prevent metrics generated in current workload gets // carried over to the next workload. legacyregistry.Reset() </ins> }) } })

mu.Unlock() } <ins> if !concreteOp.SkipWaitToCompletion { // SkipWaitToCompletion=false indicates this step has waited for the Pods to be scheduled. // So we reset the metrics in global registry; otherwise metrics gathered in this step // will be carried over to next step. legacyregistry.Reset() } </ins> case *churnOp: var namespace string if concreteOp.Namespace != nil {

q99 := hist.Quantile(0.95) avg := hist.Average() <del> // clear the metrics so that next test always starts with empty prometheus // metrics (since the metrics are shared among all tests run inside the same binary) hist.Clear() </del> msFactor := float64(time.Second) / float64(time.Millisecond) // Copy labels and add "Metric" label for this metric.