Change:
We need an E2E test for HPA behavior. HPA v2beta2 is graduating to stable! KEP: Tracking issue: Working doc: Slack channel: #sig-autoscaling-api . on #sig-autoscaling-api "Hey, I want to do this." (just so two people don't start the same work) a new E2E test that tests . There are currently two kinds of HPA tests in the Kubernetes E2E suite. One tests and the other tests . The feature is independent of the kind of metrics HPA is using. After HPA calculates scale for all metrics, it select the maximum and then applies user configured limits which are , and . Behavior specifies a scale up and down limit in rate (relative and absolute) as well as a delay. Defaults are: New behavior tests could be part of the CPU or the custom metrics suites, it doesn't really matter. Whichever is easier to implement and has the least flakes. Cases we should test: scale down works. Set , verify scale up, then verify scale down happens within 1 minute. up delay works. Set , continuously send high usage and verify scale up happens only after 5 minutes. autoscaling works. Set and . This will the replacement for the special 0 replicas case when we deprecate it, we should make sure it reliably works. Send a pull request (one squashed commit please) and notify #sig-autoscaling-api so we can take a look too. It will probably need API review. Feel free to reach out for help on #sig-autoscaling-api if you have any questions!
/sig autoscaling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig autoscaling /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign danishprakash
GitHub didn't allow me to assign the following users: danishprakash. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign danishprakash Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hey any update on this?
Yeah, I'll be sharing an update over the weekend.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen are you planning to finish this task? Otherwise I'll take over.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign pbetkier
Reopened this issue. <detailsIn response to : /reopen /assign pbetkier Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten

Discussion:
f := framework.NewDefaultFramework("horizontal-pod-autoscaling") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged <ins> fullWindowOfNewUsage := 30 * time.Second windowWithOldUsagePasses := 30 * time.Second newPodMetricsDelay := 15 * time.Second metricsAvailableDelay := fullWindowOfNewUsage + windowWithOldUsagePasses + newPodMetricsDelay hpaReconciliationInterval := 15 * time.Second actuationDelay := 10 * time.Second maxHPAReactionTime := metricsAvailableDelay + hpaReconciliationInterval + actuationDelay maxConsumeCPUDelay := 30 * time.Second waitForReplicasPollInterval := 20 * time.Second maxResourceConsumerDelay := maxConsumeCPUDelay + waitForReplicasPollInterval waitBuffer := 1 * time.Minute </ins> ginkgo.Describe("with short downscale stabilization window", func() { ginkgo.It("should scale down soon after the stabilization period", func() { ginkgo.By("setting up resource consumer and HPA")

usageForSingleReplica := 110 initPods := 1 initCPUUsageTotal := initPods * usageForSingleReplica <ins> upScaleStabilization := 0 * time.Minute </ins> downScaleStabilization := 1 * time.Minute rc := e2eautoscaling.NewDynamicResourceConsumer(

defer rc.CleanUp() hpa := e2eautoscaling.CreateCPUHorizontalPodAutoscalerWithBehavior( <del> rc, int32(targetCPUUtilizationPercent), 1, 5, int32(downScaleStabilization.Seconds()), </del> <ins> rc, int32(targetCPUUtilizationPercent), 1, 5, e2eautoscaling.HPABehaviorWithStabilizationWindows(upScaleStabilization, downScaleStabilization), </ins> ) defer e2eautoscaling.DeleteHPAWithBehavior(rc, hpa.Name) <del> fullWindowOfNewUsage := 30 * time.Second windowWithOldUsagePasses := 30 * time.Second newPodMetricsDelay := 15 * time.Second metricsAvailableDelay := fullWindowOfNewUsage + windowWithOldUsagePasses + newPodMetricsDelay hpaReconciliationInterval := 15 * time.Second actuationDelay := 10 * time.Second maxHPAReactionTime := metricsAvailableDelay + hpaReconciliationInterval + actuationDelay maxConsumeCPUDelay := 30 * time.Second waitForReplicasPollInterval := 20 * time.Second maxResourceConsumerDelay := maxConsumeCPUDelay + waitForReplicasPollInterval waitBuffer := 1 * time.Minute </del> // making sure HPA is ready, doing its job and already has a recommendation recorded // for stabilization logic before lowering the consumption ginkgo.By("triggering scale up to record a recommendation")

framework.ExpectEqual(timeWaited < deadline, true, "waited %s, wanted less than %s", timeWaited, deadline) }) }) <ins> ginkgo.Describe("with long upscale stabilization window", func() { ginkgo.It("should scale up only after the stabilization period", func() { ginkgo.By("setting up resource consumer and HPA") podCPURequest := 500 targetCPUUtilizationPercent := 25 usageForSingleReplica := 110 initPods := 2 initCPUUsageTotal := initPods * usageForSingleReplica upScaleStabilization := 3 * time.Minute downScaleStabilization := 0 * time.Minute rc := e2eautoscaling.NewDynamicResourceConsumer( "consumer", f.Namespace.Name, e2eautoscaling.KindDeployment, initPods, initCPUUsageTotal, 0, 0, int64(podCPURequest), 200, f.ClientSet, f.ScalesGetter, e2eautoscaling.Disable, e2eautoscaling.Idle, ) defer rc.CleanUp() hpa := e2eautoscaling.CreateCPUHorizontalPodAutoscalerWithBehavior( rc, int32(targetCPUUtilizationPercent), 1, 10, e2eautoscaling.HPABehaviorWithStabilizationWindows(upScaleStabilization, downScaleStabilization), ) defer e2eautoscaling.DeleteHPAWithBehavior(rc, hpa.Name) // making sure HPA is ready, doing its job and already has a recommendation recorded // for stabilization logic before increasing the consumption ginkgo.By("triggering scale down to record a recommendation") rc.ConsumeCPU(1 * usageForSingleReplica) rc.WaitForReplicas(1, maxHPAReactionTime+maxResourceConsumerDelay+waitBuffer) ginkgo.By("triggering scale up by increasing consumption") rc.ConsumeCPU(3 * usageForSingleReplica) waitStart := time.Now() rc.WaitForReplicas(3, upScaleStabilization+maxHPAReactionTime+maxResourceConsumerDelay+waitBuffer) timeWaited := time.Now().Sub(waitStart) ginkgo.By("verifying time waited for a scale up") framework.Logf("time waited for scale up: %s", timeWaited) framework.ExpectEqual(timeWaited > upScaleStabilization, true, "waited %s, wanted more than %s", timeWaited, upScaleStabilization) deadline := upScaleStabilization + maxHPAReactionTime + maxResourceConsumerDelay framework.ExpectEqual(timeWaited < deadline, true, "waited %s, wanted less than %s", timeWaited, deadline) }) }) ginkgo.Describe("with upscale autoscaling disabled", func() { ginkgo.It("shouldn't scale up", func() { ginkgo.By("setting up resource consumer and HPA") podCPURequest := 500 targetCPUUtilizationPercent := 25 usageForSingleReplica := 110 initPods := 1 initCPUUsageTotal := initPods * usageForSingleReplica rc := e2eautoscaling.NewDynamicResourceConsumer( "consumer", f.Namespace.Name, e2eautoscaling.KindDeployment, initPods, initCPUUsageTotal, 0, 0, int64(podCPURequest), 200, f.ClientSet, f.ScalesGetter, e2eautoscaling.Disable, e2eautoscaling.Idle, ) defer rc.CleanUp() hpa := e2eautoscaling.CreateCPUHorizontalPodAutoscalerWithBehavior( rc, int32(targetCPUUtilizationPercent), 1, 10, e2eautoscaling.HPABehaviorWithScaleUpDisabled(), ) defer e2eautoscaling.DeleteHPAWithBehavior(rc, hpa.Name) waitDeadline := maxHPAReactionTime + maxResourceConsumerDelay + waitBuffer ginkgo.By("trying to trigger scale up") rc.ConsumeCPU(8 * usageForSingleReplica) waitStart := time.Now() rc.EnsureDesiredReplicasInRange(initPods, initPods, waitDeadline, hpa.Name) timeWaited := time.Now().Sub(waitStart) ginkgo.By("verifying time waited for a scale up") framework.Logf("time waited for scale up: %s", timeWaited) framework.ExpectEqual(timeWaited > waitDeadline, true, "waited %s, wanted to wait more than %s", timeWaited, waitDeadline) ginkgo.By("verifying number of replicas") replicas := rc.GetReplicas() framework.ExpectEqual(replicas == initPods, true, "had %s replicas, still have %s replicas after time deadline", initPods, replicas) }) }) ginkgo.Describe("with downscale autoscaling disabled", func() { ginkgo.It("shouldn't scale down", func() { ginkgo.By("setting up resource consumer and HPA") podCPURequest := 500 targetCPUUtilizationPercent := 25 usageForSingleReplica := 110 initPods := 3 initCPUUsageTotal := initPods * usageForSingleReplica rc := e2eautoscaling.NewDynamicResourceConsumer( "consumer", f.Namespace.Name, e2eautoscaling.KindDeployment, initPods, initCPUUsageTotal, 0, 0, int64(podCPURequest), 200, f.ClientSet, f.ScalesGetter, e2eautoscaling.Disable, e2eautoscaling.Idle, ) defer rc.CleanUp() hpa := e2eautoscaling.CreateCPUHorizontalPodAutoscalerWithBehavior( rc, int32(targetCPUUtilizationPercent), 1, 10, e2eautoscaling.HPABehaviorWithScaleDownDisabled(), ) defer e2eautoscaling.DeleteHPAWithBehavior(rc, hpa.Name) defaultDownscaleStabilisation := 5 * time.Minute waitDeadline := maxHPAReactionTime + maxResourceConsumerDelay + defaultDownscaleStabilisation ginkgo.By("trying to trigger scale down") rc.ConsumeCPU(1 * usageForSingleReplica) waitStart := time.Now() rc.EnsureDesiredReplicasInRange(initPods, initPods, waitDeadline, hpa.Name) timeWaited := time.Now().Sub(waitStart) ginkgo.By("verifying time waited for a scale down") framework.Logf("time waited for scale down: %s", timeWaited) framework.ExpectEqual(timeWaited > waitDeadline, true, "waited %s, wanted to wait more than %s", timeWaited, waitDeadline) ginkgo.By("verifying number of replicas") replicas := rc.GetReplicas() framework.ExpectEqual(replicas == initPods, true, "had %s replicas, still have %s replicas after time deadline", initPods, replicas) }) }) </ins> })

rc.clientSet.AutoscalingV2().HorizontalPodAutoscalers(rc.nsName).Delete(context.TODO(), autoscalerName, metav1.DeleteOptions{}) } <del> func CreateCPUHorizontalPodAutoscalerWithBehavior(rc *ResourceConsumer, cpu, minReplicas, maxRepl, downscaleStabilizationSeconds int32) *autoscalingv2.HorizontalPodAutoscaler { </del> <ins> func CreateCPUHorizontalPodAutoscalerWithBehavior(rc *ResourceConsumer, cpu int32, minReplicas int32, maxRepl int32, behavior *autoscalingv2.HorizontalPodAutoscalerBehavior) *autoscalingv2.HorizontalPodAutoscaler { </ins> hpa := &autoscalingv2.HorizontalPodAutoscaler{ ObjectMeta: metav1.ObjectMeta{ Name:      rc.name,

}, }, }, <del> Behavior: &autoscalingv2.HorizontalPodAutoscalerBehavior{ ScaleDown: &autoscalingv2.HPAScalingRules{ StabilizationWindowSeconds: &downscaleStabilizationSeconds, }, }, </del> <ins> Behavior: behavior, </ins> }, } hpa, errHPA := rc.clientSet.AutoscalingV2().HorizontalPodAutoscalers(rc.nsName).Create(context.TODO(), hpa, metav1.CreateOptions{})

return hpa } <ins> func HPABehaviorWithScaleUpAndDownRules(scaleUpRule, scaleDownRule *autoscalingv2.HPAScalingRules) *autoscalingv2.HorizontalPodAutoscalerBehavior { return &autoscalingv2.HorizontalPodAutoscalerBehavior{ ScaleUp:   scaleUpRule, ScaleDown: scaleDownRule, } } func HPAStabilizationWindowScalingRule(stabilizationDuration int32) *autoscalingv2.HPAScalingRules { return &autoscalingv2.HPAScalingRules{ StabilizationWindowSeconds: &stabilizationDuration, } } func HPAPolicyDisabledScalingRule() *autoscalingv2.HPAScalingRules { disabledPolicy := autoscalingv2.DisabledPolicySelect return &autoscalingv2.HPAScalingRules{ SelectPolicy: &disabledPolicy, } } func HPABehaviorWithStabilizationWindows(upscaleStabilization, downscaleStabilization time.Duration) *autoscalingv2.HorizontalPodAutoscalerBehavior { return HPABehaviorWithScaleUpAndDownRules( /*scaleUpRule=*/ HPAStabilizationWindowScalingRule(int32(upscaleStabilization.Seconds())), /*scaleDownRule=*/ HPAStabilizationWindowScalingRule(int32(downscaleStabilization.Seconds())), ) } func HPABehaviorWithScaleUpDisabled() *autoscalingv2.HorizontalPodAutoscalerBehavior { return HPABehaviorWithScaleUpAndDownRules( /*scaleUpRule=*/ HPAPolicyDisabledScalingRule(), /*scaleDownRule=*/ nil, ) } func HPABehaviorWithScaleDownDisabled() *autoscalingv2.HorizontalPodAutoscalerBehavior { return HPABehaviorWithScaleUpAndDownRules( /*scaleUpRule=*/ nil, /*scaleDownRule=*/ HPAPolicyDisabledScalingRule(), ) } </ins> func DeleteHPAWithBehavior(rc *ResourceConsumer, autoscalerName string) { rc.clientSet.AutoscalingV2().HorizontalPodAutoscalers(rc.nsName).Delete(context.TODO(), autoscalerName, metav1.DeleteOptions{}) }