Change:
What happened: The bug described in is still present in 1.14.8 When a change is made to the running deployment that requires a rolling update to be executed, both the current replicaset and the new replicaset is scaled out to max number of pods regardless of actual load. What you expected to happen: The bug to be fixed How to reproduce it (as minimally and precisely as possible): Follow this guide The cluster does NOT have to be an EKS cluster, the bug is not related to the underlying platform or infrastructure. Anything else we need to know?: This is bug is really critical, and should be addressed and hotfixed as soon as possible. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Container Linux by CoreOS 2247.5.0 (Rhyolite) Kernel (e.g. ): 4.19.78-coreos Install tools: kube-aws v.0.14.2 ()
This bug does only apply to the and API version, not the API.
same issue i think it was really critical iuuse so hope hot fixed asap
/sig autoscaling
I've been looking into this and have made some interesting observations about the behaviour within our systems at least: Firstly, during the update, we see even though is greater in size than the current replica count, and this continues to increase over time. This suggests the bug may be somewhere in here: is set to and so I believe is being increased (to greater than ) by the method. Secondly, we see the scale doubling (approximately) during the rolling update. The observation I've made it that it seems to be getting updated to the total count of the old and new replicasets pods combined, as if it is using the number of pods that match the label matcher for the deployment as the desired count when it finally updates the scale. I have yet to find a piece of code that verifies this though, merely an observation. Hopefully this helps narrow down the bug, it may also be a wild goose chase I can do more testing if anyone has any ideas, just let me know
Just for information, I am seeing similar behaviour using a custom metric, not CPU - I am using . In our dev environment the config is of 1, of 3. Even when there are no requests coming in, hpa is scaling to 3 or 4 pods then cooling back down to 3 pods after 5 minutes.
CPU Utilization of Pod-1 is . CPU Utilization of Pod-2 is . So CPU Utilization of Deployment is . So calculated number of replicas is . But is , so as a result, new number of replicas is . New Pod-3 is created due to RollingUpdate with and . CPU Utilization of Pod-1 is . CPU Utilization of Pod-2 is . CPU Utilization of Pod-3 is because metrics-server can't provide metrics of freshly maded Pod-3 and hpa-controller treats pod's resource utilization as when deployment tend to scale in. So CPU Utilization of Deployment is . So new number of replicas is in spite of .
I'd like to fix this issue in .
as workaround, can we just set to solve it?
, just to make sure this is not forgotten Specifying also results in an uncontrolled roll out. Please verify that the fix in apply to the and scenario as well. Or is your about just a typo? Your PR actually specifies , which is what I reported in this bug.
, if I'm not mistaken this bug is related to , not . So specifying won't help. Edit: There is actually no such thing as a , which wouldn't make any logical sense either.
Sorry, I meant but copy pasted the incorrect text. I think is simply a typo. So, as workaround, can we just set to solve it?
Unfortunately is really not a good workaround. Say you have a deployment with 200 pods, would slow down the rolling update dramatically, as only one new pod could be created at a time, instead of all 200 at once. If the startup time for your pod is 10 seconds, replacing all 200 pods would take 10 seconds with , but 2000 seconds (33.3 minutes) with And according to the documentation, setting to anything except for 0 will reduce the number of pods (basically reducing capacity) during a rolling update. We cannot tolerate any reduced capacity during a rolling update, as the service powered by the pods might get saturated with incoming customer requests. Example: when this is set to 30%, the old ReplicaSet can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods. My understanding is that the setting is to ensure that your cluster (worker nodes) are not saturated during a rolling update. In our case we do have enough free capacity in the cluster to always be able to immediately replace all pods for any running service.
I'm sorry. is typo. is correct.
Unfortunately no. When I try scenario described in with , HPA scale deployment regardless of current load. become depends on timing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
What is the status of this issue? I still believe it's a critical bug that needs to be addressed.
+1
We may be running into the same or a problem that may be connected to this. We run a deployment with The HPA, which targets the deployment, is set to scale on a custom metric which is scraped every 30 seconds (as the value is a calculated as a it takes 60 seconds for the first value to appear). When the rolling update runs, even with completely idle deployment at 1 replicas, the HPA starts to add replicas to the deployment until it either hits the max replicas (which is 8 for this deployment) or until the first Pod generates a metric. The in method works as intended. It uses 100% of the target value for pods missing metrics, so, with 2 replicas running, one at 100% and the other one at 1% it still returns 2 replicas as the desired replicas count. I believe the problem is related to how the current number of replicas is taken from the deployment spec during HPA reconciliation (). During a rolling update, we saw the deployment spec and the deployment status being out of sync for long periods of time (over 3 minutes). This does not always happen, but it happens most of the times we do a rolling update. The HPA status is also confusing, as it indicates that the metric is above target when in fact is well below target: Should the HPA reconciliation use the current number of replicas from the status rather that from the spec?
thanks for the report. I am able to reproduce this on 1.14.10 with your test yaml. I will find the root cause and come back to you shortly.
/assign josephburnett
This issue is reproduced in my environment (v1.15.9) too, and it's resolved by replacing the metric type from to . I suspect the formula calculating the average is wrong.
I've cherry picked on top of 1.14.8 and it indeed fixes the bug. I've approved the PR and requested to fix it in as well.
, thx a lot! So in the next release of 1.14, 1.15, 1.16 and 1.17 this bug will be fixed?
Unless I'm mistaken this won't be cherry-picked back to 1.14 as it's fallen out of support as of the GA release of 1.17 as per . At best it'll be getting back to 1.15.
I'll work on backporting this to 1.15 - 1.18.
Cherry pick process (for reference): 1:15: 16: 17: 18:
This bug is affecting us badly , Is this fix merged in 1.16.x release ? if not is there a temporary solution to fix this issue
This fix is available from 1.16.9.

Discussion:
return currentReplicas, utilization, rawUtilization, timestamp, nil } <ins> newReplicas := int32(math.Ceil(newUsageRatio * float64(len(metrics)))) if (newUsageRatio < 1.0 && newReplicas > currentReplicas) || (newUsageRatio > 1.0 && newReplicas < currentReplicas) { // return the current replicas if the change of metrics length would cause a change in scale direction return currentReplicas, utilization, rawUtilization, timestamp, nil } </ins> // return the result, where the number of replicas considered is // however many replicas factored into our calculation <del> return int32(math.Ceil(newUsageRatio * float64(len(metrics)))), utilization, rawUtilization, timestamp, nil </del> <ins> return newReplicas, utilization, rawUtilization, timestamp, nil </ins> } // GetRawResourceReplicas calculates the desired replica count based on a target resource utilization (as a raw milli-value)

tc.runTest(t) } <ins> func TestReplicaCalcDuringRollingUpdateWithMaxSurge(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  2, expectedReplicas: 2, podPhase:         []v1.PodPhase{v1.PodRunning, v1.PodRunning, v1.PodRunning}, resource: &resourceInfo{ name:     v1.ResourceCPU, requests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, levels:   []int64{100, 100}, targetUtilization:   50, expectedUtilization: 10, expectedValue:       numContainersPerPod * 100, }, } tc.runTest(t) } </ins> // TestComputedToleranceAlgImplementation is a regression test which // back-calculates a minimal percentage for downscaling based on a small percentage // increase in pod utilization which is calibrated against the tolerance value.