Change:
I utilize two customized metrics, A and B, in my HPA system. A is a gauge-based metric called SLA Metric, while B is a count-based metric that tracks failed requests with HTTP status code 502 or 503 from Istio. These metrics are scraped by Prometheus. To use custom metrics in HPA, we're employing Kube Metrics Adapter . When the application load increases, the value of the SLA Metric also increases, and then the pods scale up until they reach the maximum replica count as expected. However, the problem arises when the load dissipates, and the pods never scale down. Despite the SLA Metric's value being below the target in Prometheus, the HPA description still displays the metric value with a stale value that can be above or below the target. One possible reason for this is that Metric B, which relies on Istio requests, shows up as unknown since there have been no failed requests with the 502 or 503 status codes. Thus, the Prometheus query fails. We have noticed this behavior after upgrading Kube from version 1.21 to 1.24 and changing the HPA version from autoscaling/v2beta2 to autoscaling/v2. kubectl describe hpa my-hpa To troubleshoot this further we checked the metrics value using Output: Although the metric value appears as zero, we can observe that the HPA description displays a stagnant value. Workaround : HPA behaves as expected when the second metric B is completely removed or modified to return 0 when the query fails. HPA should scale down properly based on one of the metrics, even when the other metric value is not available. Set up the Kube metrics adapter . Create a custom-metric-based HPA that uses two metrics among which the value of one is undefined. Increase the load i.e., the value of the other metric so that the HPA kicks in and scales up the pods to the max replica count. Reduce the load i.e. the value of the metric, it will be stuck at a random value. Does anyone similar issues in hpa or is this behavior of hpa for multiple metrics changed recently, especially in scaling down events? Can anyone from the community look into the issue and give some clarity? <detailsv1.24.7 </details<detailsEKS </details<detailsAlpine Linux </details<details</details<details</details<details</details
/sig autoscaling
About the problem that the HPA description displays a stagnant value, we have seen a 1.24 cluster exposing a similar behavior and here are the notes I took when diagnosing the issue. As far as I can see from reading the code the same behavior is present in more recent k8s versions as well (1.28+). The metric information comes from the hpa.Status.CurrentMetrics field of the hpa object. HPA controller loop updates hpa.Status.CurrentMetrics in setStatus [1] which is at the very end [2] of the hpa reconcile loop reconcileAutoscaler [3]. When a custom metric is in error and can't be retrieved, computeReplicasForMetric [4] will return an error back to computeReplicasForMetrics which in turn returns an error IF the hpa replica count is bellow specReplica based on the remaining available metrics [5]. Due to that error, reconcileAutoscaler returns before it calls setStatus [6] and that's why the currentMetrics field isn't updated. Note that reconcileAutoscaler does not uses hpa.Status.CurrentMetrics to make scaling decisions, the reconcilier pull the metrics from the metric server at each evaluation [7] [8] [9] so scale-ups will keep happening based on the available metrics. On the other hand, HPA not scaling down on incomplete metrics make sense as it cannot know the conditions are met for a scaledown unless it retrieves all the metrics. [1] [2] [3] [4] [5] [6] [7] [8] [9]
/assign
/sig instrumentation
Thanks for the detailed explanation with code snippets. Now I am able to get a high level overview of what had happened after reading your comments. Just update here once the issue is fixed.
/assign /triage accepted
Thanks for the detailed explanation, awaiting for this issue to be resolved.

Discussion:
// computeReplicasForMetrics may return both non-zero metricDesiredReplicas and an error. // That means some metrics still work and HPA should perform scaling based on them. if err != nil && metricDesiredReplicas == -1 { <del> a.setCurrentReplicasInStatus(hpa, currentReplicas) </del> <ins> a.setCurrentReplicasAndMetricsInStatus(hpa, currentReplicas, metricStatuses) </ins> if err := a.updateStatusIfNeeded(ctx, hpaStatusOriginal, hpa); err != nil { utilruntime.HandleError(err) }

if err != nil { a.eventRecorder.Eventf(hpa, v1.EventTypeWarning, "FailedRescale", "New size: %d; reason: %s; error: %v", desiredReplicas, rescaleReason, err.Error()) setCondition(hpa, autoscalingv2.AbleToScale, v1.ConditionFalse, "FailedUpdateScale", "the HPA controller was unable to update the target scale: %v", err) <del> a.setCurrentReplicasInStatus(hpa, currentReplicas) </del> <ins> a.setCurrentReplicasAndMetricsInStatus(hpa, currentReplicas, metricStatuses) </ins> if err := a.updateStatusIfNeeded(ctx, hpaStatusOriginal, hpa); err != nil { utilruntime.HandleError(err) }

return nil, schema.GroupResource{}, firstErr } <del> // setCurrentReplicasInStatus sets the current replica count in the status of the HPA. func (a *HorizontalController) setCurrentReplicasInStatus(hpa *autoscalingv2.HorizontalPodAutoscaler, currentReplicas int32) { a.setStatus(hpa, currentReplicas, hpa.Status.DesiredReplicas, hpa.Status.CurrentMetrics, false) </del> <ins> // setCurrentReplicasAndMetricsInStatus sets the current replica count and metrics in the status of the HPA. func (a *HorizontalController) setCurrentReplicasAndMetricsInStatus(hpa *autoscalingv2.HorizontalPodAutoscaler, currentReplicas int32, metricStatuses []autoscalingv2.MetricStatus) { a.setStatus(hpa, currentReplicas, hpa.Status.DesiredReplicas, metricStatuses, false) </ins> } // setStatus recreates the status of the given HPA, updating the current and