Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create unbound pvc count metrics for pvc controller after creating pvc only Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create unbound pv count metrics for pvc controller after creating pv only Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics should create prometheus metrics for volume provisioning and attach/detach Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create total pv count metrics for with plugin and volume mode labels after creating pv Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics should create prometheus metrics for volume provisioning errors [Slow] Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create none metrics for pvc controller before creating any PV or PVC Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create bound pv/pvc count metrics for pvc controller after creating both pv and pvc Test 2021-05-11 19:20:26 +0000 UTC /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think that this will be fixed by cc
Now we just need to convince someone to merge that PR...
The PR mentioned above was merged 10 days ago, but it didn't fix the failing tests. I checked the recent failures, and I didn't see anything failing exactly with the error reported (). However, the failures look very similar, there are many timeouts when getting the metrics, like: Most of Volume metrics PVController tests are failing with this error: has some random passes, but usually fails with: also has random passes, but usually fails: fails fetching c-m metrics: also fails to find one metric: These all look related :) There are also some cleanup failures in the same job, but I don't think they're related, they started more recently.
If I read correctly, this means that the test doesn't get the metrics that it expects. I'm not entirely sure, but I think if it ran into an actual error, it would fail differently. Can you check whether there's any mention of "Error fetching controller-manager metrics"? This shows that metrics grabbing worked. It found some metrics data, it's just not the expected data. Not related to metrics? Another variant of "didn't get expected metrics".
Nope, there is nothing like that in . Ah, yes, this one doesn't look directly related, and it's different SIG. We should track it in a separate issue probably, I'll check it. I opened another issue for tracking unmounting failures in the same job:
Yes. There was that error message 29 days ago (06/14/2021) when I created this issue. Here is the But now, the error message is not exactly the same but this new one fails at the same step (Getting plugin name).
I think there's enough evidence to claim (until proven otherwise, of course :sweat_smile: ) that metrics grabbing now works. Why the metrics data is not what the tests expect I don't know - someone familiar with these tests and how they apply to Windows would have to check that.
Thanks for the suggestion I'll look for someone in the community who can help us to test this and update the tests if It is necessary.
has a meeting in ~30 minutes and I politely asked to review failures :)
FTR - those tests aren't related to Windows (the windows tests might have their own issues).
BTW - shouldn't we add this to 1.22 milestone?
/milestone v1.22
/assign
might be related
timed out with: I saw that the failure is in the call to , in the test there's a beforeEach hook where metrics are fetched with a function wrapped inside a successfully, However the test attempts to get metrics again later using the same function but because it's wrapped with the output is an empty string next time. Do you know why the function that fetches metrics is wrapped with ? cc
The sync.Once function are used for waiting for the corresponding pods once, I think it is a mistake, and the logic to obtain metrics should be out of the sync.Once function
Thanks for chiming in indeed once I removed the wrapping all of the tests passed in
/reopen I'll check the results in CI and also cherrypick to 1.22, then we should be able to close this issue.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All of the storage metric related failures are green now, the remaining storage tests that are failing are tracked in
thanks let's close this then! /close
Closing this issue. <detailsIn response to : thanks let's close this then! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
} var err error <del> var output string </del> g.waitForSchedulerReadyOnce.Do(func() { <del> var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeScheduler, metav1.NamespaceSystem, kubeSchedulerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for scheduler pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return </del> <ins> if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, g.kubeScheduler, 0); readyErr != nil { err = fmt.Errorf("error waiting for kube-scheduler pod to be ready: %w", readyErr) </ins> } }) if err != nil { return SchedulerMetrics{}, err } <ins> var lastMetricsFetchErr error var output string if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeScheduler, metav1.NamespaceSystem, kubeSchedulerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err := fmt.Errorf("error waiting for kube-scheduler pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return SchedulerMetrics{}, err } </ins> return parseSchedulerMetrics(output) }

} var err error <del> var output string podName := g.kubeControllerManager g.waitForControllerManagerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, podName, 0); readyErr != nil { err = fmt.Errorf("error waiting for controller manager pod to be ready: %w", readyErr) return } var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeControllerManager, metav1.NamespaceSystem, kubeControllerManagerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for controller manager pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return </del> <ins> g.waitForControllerManagerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, g.kubeControllerManager, 0); readyErr != nil { err = fmt.Errorf("error waiting for kube-controller-manager pod to be ready: %w", readyErr) </ins> } }) if err != nil { return ControllerManagerMetrics{}, err } <ins> var output string var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeControllerManager, metav1.NamespaceSystem, kubeControllerManagerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err := fmt.Errorf("error waiting for kube-controller-manager to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return ControllerManagerMetrics{}, err } </ins> return parseControllerManagerMetrics(output) } // GrabFromSnapshotController returns metrics from controller manager func (g *Grabber) GrabFromSnapshotController(podName string, port int) (SnapshotControllerMetrics, error) { if !g.grabFromSnapshotController { <del> return SnapshotControllerMetrics{}, fmt.Errorf("snapshot controller: %w", MetricsGrabbingDisabledError) </del> <ins> return SnapshotControllerMetrics{}, fmt.Errorf("volume-snapshot-controller: %w", MetricsGrabbingDisabledError) </ins> } // Use overrides if provided via test config flags. <del> // Otherwise, use the default snapshot controller pod name and port. </del> <ins> // Otherwise, use the default volume-snapshot-controller pod name and port. </ins> if podName == "" { podName = g.snapshotController }

} var err error g.waitForSnapshotControllerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, podName, 0); readyErr != nil { <del> err = fmt.Errorf("error waiting for snapshot controller pod to be ready: %w", readyErr) return } var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { _, lastMetricsFetchErr = g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, port) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for snapshot controller pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return </del> <ins> err = fmt.Errorf("error waiting for volume-snapshot-controller pod to be ready: %w", readyErr) </ins> } }) if err != nil { return SnapshotControllerMetrics{}, err } <del> output, err := g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, port) if err != nil { </del> <ins> var output string var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, port) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for volume-snapshot-controller pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) </ins> return SnapshotControllerMetrics{}, err } return parseSnapshotControllerMetrics(output) }