Change:
! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig instrumentation /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The flake happened in before.
This is fixed. !
/reopen This is still failing. Sample run:
Reopened this issue. <detailsIn response to : /reopen Sample run: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
The error message in makes me think the apiserver is not available during that test run: . Can someone link to the frequency graphs to see if the problem is widespread?
got it:
I don't think I was right above. I think the "the server is currently unable to handle the request" is an error coming from the proxy. The proxy was around the time this issue was filed (), so it seems likely to be related to it. can you take a look?
This is a known issue, and is working on it . Now that is being blocked by the debugging handler problem, could we merge to mitigate this problem first? cc
I would prefer to reach a decision about enabling the debug handlers, but if that's not possible, then merging PR as mitigation might make sense.
Think I hit this also.
It appears to still be failing based on
Why rebase? Any new test run will be based on merge with current k/k master and thus contain
<img width="1111" alt="Screen Shot 2021-05-28 at 13 04 52" src="https://user-" looks like it has become more stable now.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
grabFromScheduler                 bool grabFromClusterAutoscaler         bool kubeScheduler                     string <ins> waitForSchedulerReadyOnce         sync.Once </ins> kubeControllerManager             string waitForControllerManagerReadyOnce sync.Once }

if g.kubeScheduler == "" { return SchedulerMetrics{}, fmt.Errorf("kube-scheduler pod is not registered. Skipping Scheduler's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeSchedulerPort) </del> <ins> var err error var output string g.waitForSchedulerReadyOnce.Do(func() { var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeSchedulerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for scheduler pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return } }) </ins> if err != nil { return SchedulerMetrics{}, err }

} var err error <ins> var output string </ins> podName := g.kubeControllerManager g.waitForControllerManagerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, podName, 0); readyErr != nil {

var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { <del> _, lastMetricsFetchErr = g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) </del> <ins> output, lastMetricsFetchErr = g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) </ins> return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for controller manager pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr)

if err != nil { return ControllerManagerMetrics{}, err } <del> output, err := g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) if err != nil { return ControllerManagerMetrics{}, err } </del> return parseControllerManagerMetrics(output) }