Change:
We need an E2E test for HPA behavior. HPA v2beta2 is graduating to stable! KEP: Tracking issue: Working doc: Slack channel: #sig-autoscaling-api . on #sig-autoscaling-api "Hey, I want to do this." (just so two people don't start the same work) a new E2E test that tests . There are currently two kinds of HPA tests in the Kubernetes E2E suite. One tests and the other tests . The feature is independent of the kind of metrics HPA is using. After HPA calculates scale for all metrics, it select the maximum and then applies user configured limits which are , and . Behavior specifies a scale up and down limit in rate (relative and absolute) as well as a delay. Defaults are: New behavior tests could be part of the CPU or the custom metrics suites, it doesn't really matter. Whichever is easier to implement and has the least flakes. Cases we should test: scale down works. Set , verify scale up, then verify scale down happens within 1 minute. up delay works. Set , continuously send high usage and verify scale up happens only after 5 minutes. autoscaling works. Set and . This will the replacement for the special 0 replicas case when we deprecate it, we should make sure it reliably works. Send a pull request (one squashed commit please) and notify #sig-autoscaling-api so we can take a look too. It will probably need API review. Feel free to reach out for help on #sig-autoscaling-api if you have any questions!
/sig autoscaling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig autoscaling /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign danishprakash
GitHub didn't allow me to assign the following users: danishprakash. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign danishprakash Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hey any update on this?
Yeah, I'll be sharing an update over the weekend.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen are you planning to finish this task? Otherwise I'll take over.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign pbetkier
Reopened this issue. <detailsIn response to : /reopen /assign pbetkier Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten

Discussion:
gcm "google.golang.org/api/monitoring/v3" "google.golang.org/api/option" appsv1 "k8s.io/api/apps/v1" <del> as "k8s.io/api/autoscaling/v2" </del> <ins> autoscalingv2 "k8s.io/api/autoscaling/v2" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" <del> utilfeature "k8s.io/apiserver/pkg/util/feature" </del> clientset "k8s.io/client-go/kubernetes" <del> "k8s.io/component-base/featuregate" "k8s.io/kubernetes/pkg/features" </del> "k8s.io/kubernetes/test/e2e/framework" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/instrumentation/monitoring"

externalMetricValue           = int64(85) ) <del> var _ = SIGDescribe("[HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver)", func() { </del> <ins> type externalMetricTarget struct { value     int64 isAverage bool } var _ = SIGDescribe("[HPA] [Feature:CustomMetricsAutoscaling] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver)", func() { </ins> ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessProviderIs("gce", "gke") })

f := framework.NewDefaultFramework("horizontal-pod-autoscaling") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged <del> ginkgo.It("should scale down with Custom Metric of type Pod from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, deployment:      monitoring.SimpleStackdriverExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), metricValue), hpa:             simplePodsHPA(f.Namespace.ObjectMeta.Name, metricTarget)} tc.Run() }) </del> <ins> ginkgo.Describe("with Custom Metric of type Pod from Stackdriver", func() { ginkgo.It("should scale down", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue metricSpecs := []autoscalingv2.MetricSpec{ podMetricSpecWithAverageValueTarget(monitoring.CustomMetricName, metricTarget), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, deployment:      monitoring.SimpleStackdriverExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), metricValue), hpa:             hpa("custom-metrics-pods-hpa", f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, 1, 3, metricSpecs), } tc.Run() }) </ins> <del> ginkgo.It("should scale down with Custom Metric of type Object from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue), hpa:        objectHPA(f.Namespace.ObjectMeta.Name, metricTarget)} tc.Run() }) </del> <ins> ginkgo.It("should scale up with two metrics", func() { initialReplicas := 1 // metric 1 would cause a scale down, if not for metric 2 metric1Value := int64(100) metric1Target := 2 * metric1Value // metric2 should cause a scale up metric2Value := int64(200) metric2Target := int64(0.5 * float64(metric2Value)) metricSpecs := []autoscalingv2.MetricSpec{ podMetricSpecWithAverageValueTarget("metric1", metric1Target), podMetricSpecWithAverageValueTarget("metric2", metric2Target), } containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric1", MetricName:  "metric1", MetricValue: metric1Value, }, { Name:        "stackdriver-exporter-metric2", MetricName:  "metric2", MetricValue: metric2Value, }, } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  3, deployment:      monitoring.StackdriverExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             hpa("custom-metrics-pods-hpa", f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, 1, 3, metricSpecs), } tc.Run() }) </ins> <del> ginkgo.It("should scale down with External Metric with target value from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 2 // metric should cause scale down metricValue := externalMetricValue metricTarget := 3 * metricValue metricTargets := map[string]externalMetricTarget{ "target": { value:     metricTarget, isAverage: false, }, } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, "target", metricValue), hpa:        externalHPA(f.Namespace.ObjectMeta.Name, metricTargets)} tc.Run() </del> <ins> ginkgo.It("should scale down with Prometheus", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue metricSpecs := []autoscalingv2.MetricSpec{ podMetricSpecWithAverageValueTarget(monitoring.CustomMetricName, metricTarget), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, deployment:      monitoring.PrometheusExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), metricValue), hpa:             hpa("custom-metrics-pods-hpa", f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, 1, 3, metricSpecs), } tc.Run() }) </ins> }) <del> ginkgo.It("should scale down with External Metric with target average value from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 2 // metric should cause scale down metricValue := externalMetricValue metricAverageTarget := 3 * metricValue metricTargets := map[string]externalMetricTarget{ "target_average": { value:     metricAverageTarget, isAverage: true, }, } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, "target_average", externalMetricValue), hpa:        externalHPA(f.Namespace.ObjectMeta.Name, metricTargets)} tc.Run() }) </del> <ins> ginkgo.Describe("with Custom Metric of type Object from Stackdriver", func() { ginkgo.It("should scale down", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue metricSpecs := []autoscalingv2.MetricSpec{ objectMetricSpecWithAverageValueTarget(metricTarget), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue), hpa:        hpa("custom-metrics-objects-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs), } tc.Run() }) </ins> <del> ginkgo.It("should scale down with Custom Metric of type Pod from Stackdriver with Prometheus [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, deployment:      monitoring.PrometheusExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), metricValue), hpa:             simplePodsHPA(f.Namespace.ObjectMeta.Name, metricTarget)} tc.Run() </del> <ins> ginkgo.It("should scale down to 0", func() { initialReplicas := 2 // metric should cause scale down metricValue := int64(0) metricTarget := int64(200) metricSpecs := []autoscalingv2.MetricSpec{ objectMetricSpecWithAverageValueTarget(metricTarget), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  0, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue), hpa:        hpa("custom-metrics-objects-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 0, 3, metricSpecs), } tc.Run() }) </ins> }) <del> ginkgo.It("should scale up with two metrics of type Pod from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 1 // metric 1 would cause a scale down, if not for metric 2 metric1Value := int64(100) metric1Target := 2 * metric1Value // metric2 should cause a scale up metric2Value := int64(200) metric2Target := int64(0.5 * float64(metric2Value)) containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric1", MetricName:  "metric1", MetricValue: metric1Value, }, { Name:        "stackdriver-exporter-metric2", MetricName:  "metric2", MetricValue: metric2Value, }, } metricTargets := map[string]int64{"metric1": metric1Target, "metric2": metric2Target} tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  3, deployment:      monitoring.StackdriverExporterDeployment(stackdriverExporterDeployment, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             podsHPA(f.Namespace.ObjectMeta.Name, stackdriverExporterDeployment, metricTargets)} tc.Run() }) </del> <ins> ginkgo.Describe("with External Metric from Stackdriver", func() { ginkgo.It("should scale down with target value", func() { initialReplicas := 2 // metric should cause scale down metricValue := externalMetricValue metricTarget := 3 * metricValue metricSpecs := []autoscalingv2.MetricSpec{ externalMetricSpecWithTarget("target", externalMetricTarget{ value:     metricTarget, isAverage: false, }), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, "target", metricValue), hpa:        hpa("custom-metrics-external-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs), } tc.Run() }) </ins> <del> ginkgo.It("should scale up with two External metrics from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { initialReplicas := 1 // metric 1 would cause a scale down, if not for metric 2 metric1Value := externalMetricValue metric1Target := 2 * metric1Value // metric2 should cause a scale up metric2Value := externalMetricValue metric2Target := int64(math.Ceil(0.5 * float64(metric2Value))) metricTargets := map[string]externalMetricTarget{ "external_metric_1": { value:     metric1Target, isAverage: false, }, "external_metric_2": { value:     metric2Target, isAverage: false, }, } containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric1", MetricName:  "external_metric_1", MetricValue: metric1Value, }, { Name:        "stackdriver-exporter-metric2", MetricName:  "external_metric_2", MetricValue: metric2Value, }, } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  3, deployment:      monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             externalHPA(f.Namespace.ObjectMeta.Name, metricTargets)} tc.Run() </del> <ins> ginkgo.It("should scale down with target average value", func() { initialReplicas := 2 // metric should cause scale down metricValue := externalMetricValue metricAverageTarget := 3 * metricValue metricSpecs := []autoscalingv2.MetricSpec{ externalMetricSpecWithTarget("target_average", externalMetricTarget{ value:     metricAverageTarget, isAverage: true, }), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  1, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, "target_average", externalMetricValue), hpa:        hpa("custom-metrics-external-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs), } tc.Run() }) ginkgo.It("should scale up with two metrics", func() { initialReplicas := 1 // metric 1 would cause a scale down, if not for metric 2 metric1Value := externalMetricValue metric1Target := 2 * metric1Value // metric2 should cause a scale up metric2Value := externalMetricValue metric2Target := int64(math.Ceil(0.5 * float64(metric2Value))) metricSpecs := []autoscalingv2.MetricSpec{ externalMetricSpecWithTarget("external_metric_1", externalMetricTarget{ value:     metric1Target, isAverage: false, }), externalMetricSpecWithTarget("external_metric_2", externalMetricTarget{ value:     metric2Target, isAverage: false, }), } containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric1", MetricName:  "external_metric_1", MetricValue: metric1Value, }, { Name:        "stackdriver-exporter-metric2", MetricName:  "external_metric_2", MetricValue: metric2Value, }, } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  3, deployment:      monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             hpa("custom-metrics-external-hpa", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs), } tc.Run() }) </ins> }) <del> ginkgo.It("should scale down to 0 with Custom Metric of type Object from Stackdriver [Feature:CustomMetricsAutoscaling]", func() { defer withFeatureGate(features.HPAScaleToZero, true)() initialReplicas := 2 // metric should cause scale down metricValue := int64(100) metricTarget := 2 * metricValue tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  0, // Metric exported by deployment is ignored deployment: monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0 /* ignored */), pod:        monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue), hpa:        objectHPA(f.Namespace.ObjectMeta.Name, metricTarget)} tc.Run() </del> <ins> ginkgo.Describe("with multiple metrics of different types", func() { ginkgo.It("should scale up when one metric is missing (Pod and External metrics)", func() { initialReplicas := 1 // First metric a pod metric which is missing. // Second metric is external metric which is present, it should cause scale up. metricSpecs := []autoscalingv2.MetricSpec{ podMetricSpecWithAverageValueTarget(monitoring.CustomMetricName, 2*externalMetricValue), externalMetricSpecWithTarget("external_metric", externalMetricTarget{ value:     int64(math.Ceil(0.5 * float64(externalMetricValue))), isAverage: false, }), } containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric", MetricName:  "external_metric", MetricValue: externalMetricValue, }, // Pod Resource metric is missing from here. } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  3, deployment:      monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)} tc.Run() }) ginkgo.It("should scale up when one metric is missing (Resource and Object metrics)", func() { initialReplicas := 1 metricValue := int64(100) // First metric a resource metric which is missing (no consumption). // Second metric is object metric which is present, it should cause scale up. metricSpecs := []autoscalingv2.MetricSpec{ resourceMetricSpecWithAverageUtilizationTarget(50), objectMetricSpecWithAverageValueTarget(int64(math.Ceil(0.5 * float64(metricValue)))), } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  3, deployment:      monitoring.SimpleStackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), 0), pod:             monitoring.StackdriverExporterPod(stackdriverExporterPod, f.Namespace.Name, stackdriverExporterPod, monitoring.CustomMetricName, metricValue), hpa:             hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)} tc.Run() }) ginkgo.It("should not scale down when one metric is missing (Container Resource and External Metrics)", func() { initialReplicas := 2 // First metric a container resource metric which is missing. // Second metric is external metric which is present, it should cause scale down if the first metric wasn't missing. metricSpecs := []autoscalingv2.MetricSpec{ containerResourceMetricSpecWithAverageUtilizationTarget("container-resource-metric", 50), externalMetricSpecWithTarget("external_metric", externalMetricTarget{ value:     2 * externalMetricValue, isAverage: false, }), } containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric", MetricName:  "external_metric", MetricValue: externalMetricValue, }, // Container Resource metric is missing from here. } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  initialReplicas, verifyStability: true, deployment:      monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)} tc.Run() }) ginkgo.It("should not scale down when one metric is missing (Pod and Object Metrics)", func() { initialReplicas := 2 metricValue := int64(100) // First metric an object metric which is missing. // Second metric is pod metric which is present, it should cause scale down if the first metric wasn't missing. metricSpecs := []autoscalingv2.MetricSpec{ objectMetricSpecWithAverageValueTarget(int64(math.Ceil(0.5 * float64(metricValue)))), podMetricSpecWithAverageValueTarget("pod_metric", 2*metricValue), } containers := []monitoring.CustomMetricContainerSpec{ { Name:        "stackdriver-exporter-metric", MetricName:  "pod_metric", MetricValue: metricValue, }, } tc := CustomMetricTestCase{ framework:       f, kubeClient:      f.ClientSet, initialReplicas: initialReplicas, scaledReplicas:  initialReplicas, verifyStability: true, deployment:      monitoring.StackdriverExporterDeployment(dummyDeploymentName, f.Namespace.ObjectMeta.Name, int32(initialReplicas), containers), hpa:             hpa("multiple-metrics", f.Namespace.ObjectMeta.Name, dummyDeploymentName, 1, 3, metricSpecs)} tc.Run() }) </ins> }) })

// CustomMetricTestCase is a struct for test cases. type CustomMetricTestCase struct { framework       *framework.Framework <del> hpa             *as.HorizontalPodAutoscaler </del> <ins> hpa             *autoscalingv2.HorizontalPodAutoscaler </ins> kubeClient      clientset.Interface deployment      *appsv1.Deployment pod             *v1.Pod initialReplicas int scaledReplicas  int <ins> verifyStability bool </ins> } // Run starts test case.

defer tc.kubeClient.AutoscalingV2().HorizontalPodAutoscalers(tc.framework.Namespace.ObjectMeta.Name).Delete(context.TODO(), tc.hpa.ObjectMeta.Name, metav1.DeleteOptions{}) waitForReplicas(tc.deployment.ObjectMeta.Name, tc.framework.Namespace.ObjectMeta.Name, tc.kubeClient, 15*time.Minute, tc.scaledReplicas) <ins> if tc.verifyStability { ensureDesiredReplicasInRange(tc.deployment.ObjectMeta.Name, tc.framework.Namespace.ObjectMeta.Name, tc.kubeClient, tc.scaledReplicas, tc.scaledReplicas, 10*time.Minute) } </ins> } func createDeploymentToScale(f *framework.Framework, cs clientset.Interface, deployment *appsv1.Deployment, pod *v1.Pod) error {

} } <del> func simplePodsHPA(namespace string, metricTarget int64) *as.HorizontalPodAutoscaler { return podsHPA(namespace, stackdriverExporterDeployment, map[string]int64{monitoring.CustomMetricName: metricTarget}) </del> <ins> func podMetricSpecWithAverageValueTarget(metric string, targetValue int64) autoscalingv2.MetricSpec { return autoscalingv2.MetricSpec{ Type: autoscalingv2.PodsMetricSourceType, Pods: &autoscalingv2.PodsMetricSource{ Metric: autoscalingv2.MetricIdentifier{ Name: metric, }, Target: autoscalingv2.MetricTarget{ Type:         autoscalingv2.AverageValueMetricType, AverageValue: resource.NewQuantity(targetValue, resource.DecimalSI), }, }, } </ins> } <del> func podsHPA(namespace string, deploymentName string, metricTargets map[string]int64) *as.HorizontalPodAutoscaler { var minReplicas int32 = 1 metrics := []as.MetricSpec{} for metric, target := range metricTargets { metrics = append(metrics, as.MetricSpec{ Type: as.PodsMetricSourceType, Pods: &as.PodsMetricSource{ Metric: as.MetricIdentifier{ Name: metric, }, Target: as.MetricTarget{ Type:         as.AverageValueMetricType, AverageValue: resource.NewQuantity(target, resource.DecimalSI), }, </del> <ins> func objectMetricSpecWithAverageValueTarget(targetValue int64) autoscalingv2.MetricSpec { return autoscalingv2.MetricSpec{ Type: autoscalingv2.ObjectMetricSourceType, Object: &autoscalingv2.ObjectMetricSource{ Metric: autoscalingv2.MetricIdentifier{ Name: monitoring.CustomMetricName, </ins> }, <del> }) } return &as.HorizontalPodAutoscaler{ ObjectMeta: metav1.ObjectMeta{ Name:      "custom-metrics-pods-hpa", Namespace: namespace, }, Spec: as.HorizontalPodAutoscalerSpec{ Metrics:     metrics, MaxReplicas: 3, MinReplicas: &minReplicas, ScaleTargetRef: as.CrossVersionObjectReference{ APIVersion: "apps/v1", Kind:       "Deployment", Name:       deploymentName, </del> <ins> DescribedObject: autoscalingv2.CrossVersionObjectReference{ Kind: "Pod", Name: stackdriverExporterPod, }, Target: autoscalingv2.MetricTarget{ Type:  autoscalingv2.ValueMetricType, Value: resource.NewQuantity(targetValue, resource.DecimalSI), </ins> }, }, } } <del> func objectHPA(namespace string, metricTarget int64) *as.HorizontalPodAutoscaler { var minReplicas int32 = 1 return &as.HorizontalPodAutoscaler{ ObjectMeta: metav1.ObjectMeta{ Name:      "custom-metrics-objects-hpa", Namespace: namespace, }, Spec: as.HorizontalPodAutoscalerSpec{ Metrics: []as.MetricSpec{ { Type: as.ObjectMetricSourceType, Object: &as.ObjectMetricSource{ Metric: as.MetricIdentifier{ Name: monitoring.CustomMetricName, }, DescribedObject: as.CrossVersionObjectReference{ Kind: "Pod", Name: stackdriverExporterPod, }, Target: as.MetricTarget{ Type:  as.ValueMetricType, Value: resource.NewQuantity(metricTarget, resource.DecimalSI), }, }, }, }, MaxReplicas: 3, MinReplicas: &minReplicas, ScaleTargetRef: as.CrossVersionObjectReference{ APIVersion: "apps/v1", Kind:       "Deployment", Name:       dummyDeploymentName, </del> <ins> func resourceMetricSpecWithAverageUtilizationTarget(targetValue int32) autoscalingv2.MetricSpec { return autoscalingv2.MetricSpec{ Type: autoscalingv2.ResourceMetricSourceType, Resource: &autoscalingv2.ResourceMetricSource{ Name: v1.ResourceCPU, Target: autoscalingv2.MetricTarget{ Type:               autoscalingv2.UtilizationMetricType, AverageUtilization: &targetValue, </ins> }, }, } } <del> type externalMetricTarget struct { value     int64 isAverage bool </del> <ins> func containerResourceMetricSpecWithAverageUtilizationTarget(containerName string, targetValue int32) autoscalingv2.MetricSpec { return autoscalingv2.MetricSpec{ Type: autoscalingv2.ContainerResourceMetricSourceType, ContainerResource: &autoscalingv2.ContainerResourceMetricSource{ Name:      v1.ResourceCPU, Container: containerName, Target: autoscalingv2.MetricTarget{ Type:               autoscalingv2.UtilizationMetricType, AverageUtilization: &targetValue, }, }, } </ins> } <del> func externalHPA(namespace string, metricTargets map[string]externalMetricTarget) *as.HorizontalPodAutoscaler { var minReplicas int32 = 1 metricSpecs := []as.MetricSpec{} </del> <ins> func externalMetricSpecWithTarget(metric string, target externalMetricTarget) autoscalingv2.MetricSpec { </ins> selector := &metav1.LabelSelector{ MatchLabels: map[string]string{"resource.type": "gke_container"}, MatchExpressions: []metav1.LabelSelectorRequirement{

}, }, } <del> for metric, target := range metricTargets { var metricSpec as.MetricSpec metricSpec = as.MetricSpec{ Type: as.ExternalMetricSourceType, External: &as.ExternalMetricSource{ Metric: as.MetricIdentifier{ Name:     "custom.googleapis.com|" + metric, Selector: selector, }, </del> <ins> metricSpec := autoscalingv2.MetricSpec{ Type: autoscalingv2.ExternalMetricSourceType, External: &autoscalingv2.ExternalMetricSource{ Metric: autoscalingv2.MetricIdentifier{ Name:     "custom.googleapis.com|" + metric, Selector: selector, </ins> }, <del> } if target.isAverage { metricSpec.External.Target.Type = as.AverageValueMetricType metricSpec.External.Target.AverageValue = resource.NewQuantity(target.value, resource.DecimalSI) } else { metricSpec.External.Target.Type = as.ValueMetricType metricSpec.External.Target.Value = resource.NewQuantity(target.value, resource.DecimalSI) } metricSpecs = append(metricSpecs, metricSpec) </del> <ins> }, } if target.isAverage { metricSpec.External.Target.Type = autoscalingv2.AverageValueMetricType metricSpec.External.Target.AverageValue = resource.NewQuantity(target.value, resource.DecimalSI) } else { metricSpec.External.Target.Type = autoscalingv2.ValueMetricType metricSpec.External.Target.Value = resource.NewQuantity(target.value, resource.DecimalSI) </ins> } <del> hpa := &as.HorizontalPodAutoscaler{ </del> <ins> return metricSpec } func hpa(name, namespace, deploymentName string, minReplicas, maxReplicas int32, metricSpecs []autoscalingv2.MetricSpec) *autoscalingv2.HorizontalPodAutoscaler { return &autoscalingv2.HorizontalPodAutoscaler{ </ins> ObjectMeta: metav1.ObjectMeta{ <del> Name:      "custom-metrics-external-hpa", </del> <ins> Name:      name, </ins> Namespace: namespace, }, <del> Spec: as.HorizontalPodAutoscalerSpec{ </del> <ins> Spec: autoscalingv2.HorizontalPodAutoscalerSpec{ </ins> Metrics:     metricSpecs, <del> MaxReplicas: 3, </del> MinReplicas: &minReplicas, <del> ScaleTargetRef: as.CrossVersionObjectReference{ </del> <ins> MaxReplicas: maxReplicas, ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{ </ins> APIVersion: "apps/v1", Kind:       "Deployment", <del> Name:       dummyDeploymentName, </del> <ins> Name:       deploymentName, </ins> }, }, } <del> return hpa </del> } func waitForReplicas(deploymentName, namespace string, cs clientset.Interface, timeout time.Duration, desiredReplicas int) {

} } <del> // Equivalent of featuregatetesting.SetFeatureGateDuringTest // which can't be used here because we're not in a Testing context. func withFeatureGate(feature featuregate.Feature, desired bool) func() { current := utilfeature.DefaultFeatureGate.Enabled(feature) utilfeature.DefaultMutableFeatureGate.Set(fmt.Sprintf("%s=%v", string(feature), desired)) return func() { utilfeature.DefaultMutableFeatureGate.Set(fmt.Sprintf("%s=%v", string(feature), current)) </del> <ins> func ensureDesiredReplicasInRange(deploymentName, namespace string, cs clientset.Interface, minDesiredReplicas, maxDesiredReplicas int, timeout time.Duration) { interval := 60 * time.Second err := wait.PollImmediate(interval, timeout, func() (bool, error) { deployment, err := cs.AppsV1().Deployments(namespace).Get(context.TODO(), deploymentName, metav1.GetOptions{}) if err != nil { framework.Failf("Failed to get replication controller %s: %v", deployment, err) } replicas := int(deployment.Status.ReadyReplicas) framework.Logf("expecting there to be in [%d, %d] replicas (are: %d)", minDesiredReplicas, maxDesiredReplicas, replicas) if replicas < minDesiredReplicas { return false, fmt.Errorf("number of replicas below target") } else if replicas > maxDesiredReplicas { return false, fmt.Errorf("number of replicas above target") } else { return false, nil // Expected number of replicas found. Continue polling until timeout. } }) // The call above always returns an error, but if it is timeout, it's OK (condition satisfied all the time). if err == wait.ErrWaitTimeout { framework.Logf("Number of replicas was stable over %v", timeout) return </ins> } <ins> framework.ExpectNoErrorWithOffset(1, err) </ins> }