Change:
Test: https://k8s- Example: https://k8s- The HPA tests for gce-serial are flaking rather regularly; they fail about 2/3 of the time for the last week, and are often the cause of a failed test run. Can we de-flake these? Over multiple test runs the problem seems to be the number of replicas jumping up to 4 unexpectedly: Is the limit not getting set correctly here? Is this an actual bug? /sig autoscaling /priority important-soon /kind failing-test /kind flake
cc:
Recent failures fall into three categories: not starting (, , , , , , ). They fail with error message: increasing unexpectedly (, , ). They fail with error message: waiting for scale down (). This one failed with error message: I suspect 2. and 3. may be caused by test using correct amount of CPU when averaged over minute-long window but spiky (for example using 100% cpu for some time then 0% cpu for rest of the minute). I saw similar behavior when validating recent changes manually. This would result in HPA: behaving as the test expects with before recent changes (because it ignored any spikes during scale*forbidden windows, longer sample windows masking the spike), HPA (correctly) increasing size if spikes align after those changes (failures in group 2.), HPA (correctly) not decreasing size if spikes align after those changes (failure in group 3.), I'll take a look at code of those tests and try to responsible for most of the flakes but I don't have any ideas for a cause yet.
/remove-kind failing-test
I was busy this week but I think I'll be able to prepare a fix for this tomorrow.
/milestone v1.13 /kind failing-test
this tests is failing consistently in
On Friday I verified that CPU usage generated by resource consumer stays really close to the target value but oscillates slightly. To fix this I: Increase CPU usage in the test (resource consumer is implemented in a way that makes me think that deviation fro target are of a fixed size so higher target would make deviation a smaller percent of the target). Lower generated load (the test right now is targeting border between recommendation of 3 instances and 4 instances, I will change the load to something between 2 and 3 instances). I'm checking if this helps the test.
It seems to have helped. I've sent a PR with those changes for review.
Re-opening until we have at least 3 green runs: https://k8s- /reopen
Reopening this issue. <detailsIn response to : Re-opening because we just had another flake last night: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
But it doesn't work on test grid, the test now fails with: I'll look into this.
Yes, and it's now causing another test to fail as well: https://k8s- Escalating to critical-urgent. This test has been failing for 3 weeks now, and is potentially blocking visibility into other failures. /priority critical-urgent /remove-priority important-soon
This test had a couple of successful runs yesterday but is now flaking/failing in master-blocking: https://k8s- It seems to be passing in master-upgrade. Any chance we could prioritize work on it? It's preventing us from seeing whether fixes for other test failures are working.
There was only 1 flake since merged on Thursday (and ~15 green runs). was an unexpected scale up happening during stabilization. I'll check if lowering CPU utilization a bit lower deflakes the test more but I think it's ok to close this issue now.
Lowering CPU utilization didn't help.
Test jobs where these tests have flaked or failed: https://k8s- (today) https://k8s- (last 3 days ago) https://k8s- (yesterday) https://k8s- (yesterday) https://k8s- (today) https://k8s- (yesterday) ... basically, everywhere the tests run, they're flaky as recently as today. More on Triage: You can see here that across all jobs the HPA tests are flaky, with a "worst flake" perioud of 10/23-10/25, but by no means cleared up.
SIG-Autoscaling's boards are also showing flakiness: https://k8s-, although that looks better than the sig-release test boards. Any idea why?
Thanks for the links I've been looking only at one of those dashboards. I've sent that should reduce flakiness of our tests.
Thanks!
The PR merged. Now let's wait and see if it solves the problem.
The test named in the tile of this issue<sup1</supgot better. Last failures at the dashboards you listed: , (before merged), error before taht was on 10-25), , , . So it looks like the test didn't flake in the last ~2 days. Another two tests<sup2</supare flaking , I'll send a PR that should fix it. I think those flakes are result of behavior of HPA intentionally changing<sup3</supso the PR will just relax expectations of the test. <b1</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability <b2</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target value from Stackdriver [Feature:CustomMetricsAutoscaling] and [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target average value from Stackdriver [Feature:CustomMetricsAutoscaling] <b3</bHPA no longer delays scale ups based on metrics other than CPU. Flakes are when HPA scales deployment to 2 or 3 so test doesn't observe it having size 1 and times out waiting for that size.
I misread titles of custom metric tests as tests for scaling up somehow. Those flaking tests are tests for scaling down and scale downs sometimes don't happen. I'm looking into why.
While this test has gotten better, it's still flaking some of the time: https://k8s-
I looked at "Latest failures" in the two top categories according to (all the other categories have almost no recent flakes). All of the failures log as if they were not affected by : instead of . Tests results also say they are running version 1.11... or 1.12... (or more rarely 1.9... or 1.10... but I'm ignoring those for now - they are not related to any changes I made). So it looks like I need to cherry pick the PR into 1.11 and 1.12 branches.
... when I look at git branches it looks like my PR is there. There was with logs that looking like affected it. I'll wait to see if there are more failures in test runs including that change. In case there are I'll add some more logging (I have no ideas for cause of the one failure I saw so far).
/assign
GitHub didn't allow me to assign the following users: jbartosik. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any further update on this? I see it turning into more of a failure than a flake and I see another test failing as well in latest runs
We are due to cut Beta for 1.13 today and would like to understand if this should be a blocker or not.
Also are there cherrypick PRs to backport your change to 1.11 and 1.12 branches?
Potentially related:
6 (consecutive) most recent failures of test do not have judging by their logs. That PR would change them from failures (unexpected scale from 3 to 4 instances during stabilisation) to success (expected 3 to 4 instances during stabilisation). One of the 2 recent failures of looks wrong. One failure was scaling up immediately to 5 instances (wrong). The other was scale up to 4 instances (ok, I'll send a PR relaxing that condition tomorrow). I didn't send PRs cherrypicking test changes to 1.12 (because when I checked it looked like the change was there) not to 1.11 (my changes aren't there so old tests should keep working). merged and it ads more logging, which will help me debug this. Failures in the look like the PR relaxing stability requirement should have helped there too.
the , which has the 6 (consecutive) most recent failures , upgrades the master alone to 1.13(master) and and keep the nodes in old version and then run the old test suite. The latest failing runs show that the tests are run from 1.11.5 branch Doesn't this mean your changes need to be backported to 1.11 branch? That is the reason why you dont see you change in those tests logs
FWIW, the same test also flakes in job which upgrade both master and nodes to 1.13 and runs 1.11 tests.
this failure is currently blocking Beta for 1..13. Can you please investigate these and let us know the path forward.
Fixes done by are all test fixes, not feature fixes. So the problem should be fixed once cherry-pick gets to 1.11. That being said - why are we running 1.11 e2e tests against 1.13 cluster? That assumes full compatibility of both API and implementation. We can never remove a feature without breaking this suite (the tests for the feature will still exist on older branches after all). And the tests are often tweaked to the specifics of implementation. Whenever we make any change to autoscaling algorithm tests need to be updated accordingly. Making tests on 1.11 work well with 1.13 risks they will stop working well with 1.11. Shouldn't the upgrade job create 1.11 cluster, upgrade it to 1.13 and run 1.13 e2e tests on that cluster?
I see these 2 pass in the latest run of https://k8s- Over to to close when he sees fit
We have other flakes on that job now, but HPA is no longer flaking. Thanks so much for the dilligence in getting these flakes cleared up, /close
Closing this issue. <detailsIn response to : We have other flakes on that job now, but HPA is no longer flaking. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
// HPAScaleTest struct is used by the scale(...) function. type HPAScaleTest struct { <del> initPods                    int32 totalInitialCPUUsage        int32 </del> <ins> initPods                    int totalInitialCPUUsage        int </ins> perPodCPURequest            int64 targetCPUUtilizationPercent int32 minPods                     int32

// TODO The use of 3 states is arbitrary, we could eventually make this test handle "n" states once this test stabilizes. func (scaleTest *HPAScaleTest) run(name string, kind schema.GroupVersionKind, rc *common.ResourceConsumer, f *framework.Framework) { const timeToWait = 15 * time.Minute <del> rc = common.NewDynamicResourceConsumer(name, f.Namespace.Name, kind, int(scaleTest.initPods), int(scaleTest.totalInitialCPUUsage), 0, 0, scaleTest.perPodCPURequest, 200, f.ClientSet, f.InternalClientset, f.ScalesGetter) </del> <ins> rc = common.NewDynamicResourceConsumer(name, f.Namespace.Name, kind, scaleTest.initPods, scaleTest.totalInitialCPUUsage, 0, 0, scaleTest.perPodCPURequest, 200, f.ClientSet, f.InternalClientset, f.ScalesGetter) </ins> defer rc.CleanUp() hpa := common.CreateCPUHorizontalPodAutoscaler(rc, scaleTest.targetCPUUtilizationPercent, scaleTest.minPods, scaleTest.maxPods) defer common.DeleteHorizontalPodAutoscaler(rc, hpa.Name)

} scaleTest := &HPAScaleTest{ initPods:                    1, <del> totalInitialCPUUsage:        250, perPodCPURequest:            500, </del> <ins> totalInitialCPUUsage:        500, perPodCPURequest:            1000, </ins> targetCPUUtilizationPercent: 20, minPods:                     1, maxPods:                     5, firstScale:                  3, firstScaleStasis:            stasis, <del> cpuBurst:                    700, </del> <ins> cpuBurst:                    1400, </ins> secondScale:                 5, } scaleTest.run(name, kind, rc, f)

} scaleTest := &HPAScaleTest{ initPods:                    5, <del> totalInitialCPUUsage:        375, perPodCPURequest:            500, </del> <ins> totalInitialCPUUsage:        650, perPodCPURequest:            1000, </ins> targetCPUUtilizationPercent: 30, minPods:                     1, maxPods:                     5,