Code change:
When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement. Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)

Related discussion:
text: // - replaces old recommendation with the newest recommendation, // - returns {max,min} of recommendations that are not older than constraints.Scale{Up,Down}.DelaySeconds func (a *HorizontalController) stabilizeRecommendationWithBehaviors(args NormalizationArg) (int32, string, string) { <del> recommendation := args.DesiredReplicas </del> <ins> now := time.Now() </ins> foundOldSample := false oldSampleIndex := 0 <del> var scaleDelaySeconds int32 var reason, message string var betterRecommendation func(int32, int32) int32 </del> <ins> upRecommendation := args.DesiredReplicas upDelaySeconds := *args.ScaleUpBehavior.StabilizationWindowSeconds upCutoff := now.Add(-time.Second * time.Duration(upDelaySeconds)) </ins> <del> if args.DesiredReplicas >= args.CurrentReplicas { scaleDelaySeconds = *args.ScaleUpBehavior.StabilizationWindowSeconds betterRecommendation = min reason = "ScaleUpStabilized" message = "recent recommendations were lower than current one, applying the lowest recent recommendation" } else { scaleDelaySeconds = *args.ScaleDownBehavior.StabilizationWindowSeconds betterRecommendation = max reason = "ScaleDownStabilized" message = "recent recommendations were higher than current one, applying the highest recent recommendation" } maxDelaySeconds := max(*args.ScaleUpBehavior.StabilizationWindowSeconds, *args.ScaleDownBehavior.StabilizationWindowSeconds) obsoleteCutoff := time.Now().Add(-time.Second * time.Duration(maxDelaySeconds)) </del> <ins> downRecommendation := args.DesiredReplicas downDelaySeconds := *args.ScaleDownBehavior.StabilizationWindowSeconds downCutoff := now.Add(-time.Second * time.Duration(downDelaySeconds)) </ins> <del> cutoff := time.Now().Add(-time.Second * time.Duration(scaleDelaySeconds)) </del> <ins> // Calculate the upper and lower stabilization limits. </ins> for i, rec := range a.recommendations[args.Key] { <del> if rec.timestamp.After(cutoff) { recommendation = betterRecommendation(rec.recommendation, recommendation) </del> <ins> if rec.timestamp.After(upCutoff) { upRecommendation = min(rec.recommendation, upRecommendation) } if rec.timestamp.After(downCutoff) { downRecommendation = max(rec.recommendation, downRecommendation) </ins> } <del> if rec.timestamp.Before(obsoleteCutoff) { </del> <ins> if rec.timestamp.Before(upCutoff) && rec.timestamp.Before(downCutoff) { </ins> foundOldSample = true oldSampleIndex = i } } <ins> // Bring the recommendation to within the upper and lower limits (stabilize). recommendation := args.CurrentReplicas if recommendation < upRecommendation { recommendation = upRecommendation } if recommendation > downRecommendation { recommendation = downRecommendation } // Record the unstabilized recommendation. </ins> if foundOldSample { a.recommendations[args.Key][oldSampleIndex] = timestampedRecommendation{args.DesiredReplicas, time.Now()} } else { a.recommendations[args.Key] = append(a.recommendations[args.Key], timestampedRecommendation{args.DesiredReplicas, time.Now()}) } <ins> // Determine a human-friendly message. var reason, message string if args.DesiredReplicas >= args.CurrentReplicas { reason = "ScaleUpStabilized" message = "recent recommendations were lower than current one, applying the lowest recent recommendation" } else { reason = "ScaleDownStabilized" message = "recent recommendations were higher than current one, applying the highest recent recommendation" } </ins> return recommendation, reason, message }