Change:
What happened: I setup custom metrics based autoscaling with multiple metrics. One of those metrics was not available. HPA reports all metrics as "unknown" (even the ones that are still available) and refuses to operate. What you expected to happen: Pods would be automatically scaled based on metrics that were available, ignoring missing metrics (treating them as zero). A partial failure in the metrics system should not prevent auto-scaling from proceeding with the data that it has. For example, CPU % metrics come from the k8s metrics server whereas the other metrics come from the prometheus adapter. If the prometheus adapter goes away we can still use the CPU metric as a lower bound on the number of replicas. If you consider, e.g. , instead of aborting the whole metric gathering process there, it could have stored a placeholder or zero metric value when it fails to fetch the metric. How to reproduce it (as minimally and precisely as possible): Setup a custom metrics server and tell HPA to scale based on several metrics, one of which is not actually available. Environment: kops 1.8 on AWS Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
/sig autoscaling
cc
/kind bug /priority important-longterm
this return returns nil value of statuses variable. statuses variable contains the value of all metrics.
as long as we failed to get one metric of all metrics, the func computeReplicasForMetrics returns, the scale lose efficacy, it is not highly available. when failed to get metric, should we record this event and do not return:)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still having this problem.
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this issue was resolved or it works as designed? thanks,
I was testing this last week and it is not resolved. I don't think it is reasonable to say this is working as designed.
Seems there is a PR to fix it, but it hasn't been reviewed or merged.
yes, I looked into the master branch code, not fixed yet, just curious about it, it would not be fixed or as dobesv said it will be merged/fixed latter with that PR.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue was discussed at SIG Autoscaling yesterday: I'm going to raise a new PR later this week given the code's changed so much since the original PR was raised by that it doesn't make sense to just rebase it.
Thanks, that would be excellent!

Discussion:
autoscalingv1 "k8s.io/api/autoscaling/v1" autoscalingv2 "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/errors" apimeta "k8s.io/apimachinery/pkg/api/meta"

} // computeReplicasForMetrics computes the desired number of replicas for the metric specifications listed in the HPA, <del> // returning the maximum  of the computed replica counts, a description of the associated metric, and the statuses of </del> <ins> // returning the maximum of the computed replica counts, a description of the associated metric, and the statuses of </ins> // all metrics computed. func (a *HorizontalController) computeReplicasForMetrics(hpa *autoscalingv2.HorizontalPodAutoscaler, scale *autoscalingv1.Scale, metricSpecs []autoscalingv2.MetricSpec) (replicas int32, metric string, statuses []autoscalingv2.MetricStatus, timestamp time.Time, err error) {

statuses = make([]autoscalingv2.MetricStatus, len(metricSpecs)) <ins> if scale.Status.Selector == "" { errMsg := "selector is required" a.eventRecorder.Event(hpa, v1.EventTypeWarning, "SelectorRequired", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", "the HPA target's scale is missing a selector") return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } selector, err := labels.Parse(scale.Status.Selector) if err != nil { errMsg := fmt.Sprintf("couldn't convert selector into a corresponding internal selector object: %v", err) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidSelector", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", errMsg) return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } invalidMetricsCount := 0 var invalidMetricError error </ins> for i, metricSpec := range metricSpecs { <del> if scale.Status.Selector == "" { errMsg := "selector is required" a.eventRecorder.Event(hpa, v1.EventTypeWarning, "SelectorRequired", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", "the HPA target's scale is missing a selector") return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } </del> <ins> replicaCountProposal, metricNameProposal, timestampProposal, err := a.computeReplicasForMetric(hpa, metricSpec, currentReplicas, selector, &statuses[i]) </ins> <del> selector, err := labels.Parse(scale.Status.Selector) </del> if err != nil { <del> errMsg := fmt.Sprintf("couldn't convert selector into a corresponding internal selector object: %v", err) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidSelector", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", errMsg) return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } var replicaCountProposal int32 var timestampProposal time.Time var metricNameProposal string switch metricSpec.Type { case autoscalingv2.ObjectMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(metricSpec.Object.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetObjectMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetObjectMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", nil, time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForObjectMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i], metricSelector) if err != nil { return 0, "", nil, time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.PodsMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(metricSpec.Pods.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetPodsMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetPodsMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", nil, time.Time{}, fmt.Errorf("failed to get pods metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForPodsMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i], metricSelector) if err != nil { return 0, "", nil, time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.ResourceMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForResourceMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i]) if err != nil { return 0, "", nil, time.Time{}, err } case autoscalingv2.ExternalMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForExternalMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i]) if err != nil { return 0, "", nil, time.Time{}, err } default: errMsg := fmt.Sprintf("unknown metric source type %q", string(metricSpec.Type)) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidMetricSourceType", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidMetricSourceType", "the HPA was unable to compute the replica count: %s", errMsg) return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) </del> <ins> invalidMetricsCount++ invalidMetricError = err </ins> } <del> if replicas == 0 || replicaCountProposal > replicas { </del> <ins> if err == nil && (replicas == 0 || replicaCountProposal > replicas) { </ins> timestamp = timestampProposal replicas = replicaCountProposal metric = metricNameProposal } } <del> setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionTrue, "ValidMetricFound", "the HPA was able to successfully calculate a replica count from %s", metric) </del> <ins> // If all metrics are invalid or some are invalid and we would scale down, // return an error and set the condition of the hpa based on the first invalid metric. // Otherwise set the condition as scaling active as we're going to scale if invalidMetricsCount >= len(metricSpecs) || (invalidMetricsCount > 0 && replicas < currentReplicas) { return 0, "", statuses, time.Time{}, fmt.Errorf("Invalid metrics (%v invalid out of %v), last error was: %v", invalidMetricsCount, len(metricSpecs), invalidMetricError) } else { setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionTrue, "ValidMetricFound", "the HPA was able to successfully calculate a replica count from %v", metric) } </ins> return replicas, metric, statuses, timestamp, nil } <ins> // computeReplicasForMetric computes the desired number of replicas for for a specific hpa and single metric specification. func (a *HorizontalController) computeReplicasForMetric(hpa *autoscalingv2.HorizontalPodAutoscaler, spec autoscalingv2.MetricSpec, currentReplicas int32, selector labels.Selector, status *autoscalingv2.MetricStatus) (replicaCountProposal int32, metricNameProposal string, timestampProposal time.Time, err error) { switch spec.Type { case autoscalingv2.ObjectMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(spec.Object.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetObjectMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetObjectMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForObjectMetric(currentReplicas, spec, hpa, selector, status, metricSelector) if err != nil { return 0, "", time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.PodsMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(spec.Pods.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetPodsMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetPodsMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", time.Time{}, fmt.Errorf("failed to get pods metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForPodsMetric(currentReplicas, spec, hpa, selector, status, metricSelector) if err != nil { return 0, "", time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.ResourceMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForResourceMetric(currentReplicas, spec, hpa, selector, status) if err != nil { return 0, "", time.Time{}, err } case autoscalingv2.ExternalMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForExternalMetric(currentReplicas, spec, hpa, selector, status) if err != nil { return 0, "", time.Time{}, err } default: errMsg := fmt.Sprintf("unknown metric source type %q", string(spec.Type)) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidMetricSourceType", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidMetricSourceType", "the HPA was unable to compute the replica count: %v", fmt.Errorf(errMsg)) return 0, "", time.Time{}, fmt.Errorf(errMsg) } return replicaCountProposal, metricNameProposal, timestampProposal, nil } </ins> func (a *HorizontalController) reconcileKey(key string) (deleted bool, err error) { namespace, name, err := cache.SplitMetaNamespaceKey(key) if err != nil {

autoscalingv1 "k8s.io/api/autoscaling/v1" autoscalingv2 "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/meta/testrestmapper" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

} } <ins> func TestScaleUpOneMetricEmpty(t *testing.T) { tc := testCase{ minReplicas:             2, maxReplicas:             6, initialReplicas:         3, expectedDesiredReplicas: 4, CPUTarget:               30, verifyCPUCurrent:        true, metricsTarget: []autoscalingv2.MetricSpec{ { Type: autoscalingv2.ExternalMetricSourceType, External: &autoscalingv2.ExternalMetricSource{ Metric: autoscalingv2.MetricIdentifier{ Name:     "qps", Selector: &metav1.LabelSelector{}, }, Target: autoscalingv2.MetricTarget{ Value: resource.NewMilliQuantity(100, resource.DecimalSI), }, }, }, }, reportedLevels:      []uint64{300, 400, 500}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, } _, _, _, testEMClient, _ := tc.prepareTestClient(t) testEMClient.PrependReactor("list", "*", func(action core.Action) (handled bool, ret runtime.Object, err error) { return true, &emapi.ExternalMetricValueList{}, fmt.Errorf("something went wrong") }) tc.testEMClient = testEMClient tc.runTest(t) } func TestNoScaleDownOneMetricInvalid(t *testing.T) { tc := testCase{ minReplicas:             2, maxReplicas:             6, initialReplicas:         5, expectedDesiredReplicas: 5, CPUTarget:               50, metricsTarget: []autoscalingv2.MetricSpec{ { Type: "CheddarCheese", }, }, reportedLevels:      []uint64{100, 300, 500, 250, 250}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:       true, expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "InvalidMetricSourceType"}, }, } tc.runTest(t) } func TestNoScaleDownOneMetricEmpty(t *testing.T) { tc := testCase{ minReplicas:             2, maxReplicas:             6, initialReplicas:         5, expectedDesiredReplicas: 5, CPUTarget:               50, metricsTarget: []autoscalingv2.MetricSpec{ { Type: autoscalingv2.ExternalMetricSourceType, External: &autoscalingv2.ExternalMetricSource{ Metric: autoscalingv2.MetricIdentifier{ Name:     "qps", Selector: &metav1.LabelSelector{}, }, Target: autoscalingv2.MetricTarget{ Value: resource.NewMilliQuantity(1000, resource.DecimalSI), }, }, }, }, reportedLevels:      []uint64{100, 300, 500, 250, 250}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:       true, expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetExternalMetric"}, }, } _, _, _, testEMClient, _ := tc.prepareTestClient(t) testEMClient.PrependReactor("list", "*", func(action core.Action) (handled bool, ret runtime.Object, err error) { return true, &emapi.ExternalMetricValueList{}, fmt.Errorf("something went wrong") }) tc.testEMClient = testEMClient tc.runTest(t) } </ins> // TODO: add more tests