Change:
[ ] [sig-network] LoadBalancers ESIPP [Slow] should handle updates to ExternalTrafficPolicy field [ ] [sig-network] LoadBalancers ESIPP [Slow] should only target nodes with endpoints [ ] [sig-network] LoadBalancers ESIPP [Slow] should work for type=LoadBalancer [ ] [sig-network] LoadBalancers ESIPP [Slow] should work from pods: a new flake [ ] [sig-network] LoadBalancers should be able to change the type and ports of a UDP service [Slow] [ ] [sig-network] LoadBalancers should be able to change the type and ports of a TCP service [Slow] [ ] [sig-network] LoadBalancers should be able to create an internal type load balancer [Slow] [ ] [sig-network] LoadBalancers should be able to preserve UDP traffic when server pod cycles for a LoadBalancer service on different nodes [ ] [sig-network] LoadBalancers should be able to preserve UDP traffic when server pod cycles for a LoadBalancer service on the same nodes [ ] sig-network] LoadBalancers should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [LinuxOnly] [ ] [sig-network] LoadBalancers should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [LinuxOnly] [ ] [sig-network] LoadBalancers should handle load balancer cleanup finalizer for service [Slow] [ ] [sig-network] LoadBalancers should only allow access from service loadbalancer source ranges [Slow] fixed once before [ ] [sig-network] Services GCE [Slow] should be able to create and tear down a standard-tier load balancer [Slow] still open [ ] [sig-storage] CSI Mock volume storage capacity storage capacity unlimited [ ] [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout 11-14 (gce-master-scale-correctness failed for dumplog problem before that) is a new flake. but I cannot find any other flake in /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Maybe we can try to debug during a job execution ?
There is something odd Some components are using wrong versions The controller manager has lots of these errors related to this
For the three flakes here, [sig-network] LoadBalancers ESIPP [Slow] should work from pods: a new flake only once, and we may just observe and do not need to take actions. [sig-network] Services GCE [Slow] should be able to create and tear down a standard-tier load balancer [Slow] [Flaking test] [sig-network] Services [Slow] should be able to create and tear down a standard-tier load balancer We can use to track [sig-network] LoadBalancers should only allow access from service loadbalancer source ranges [Slow] Fixed in . Still falke in . BTW, overall, is still flaky.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen just see
Reopened this issue. <detailsIn response to : /reopen just see Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is open as it almost failed every run with different flake. will be closed if the overall flake rate is acceptable I will keep this open to track all flakes.
this needs reviewers
/reopen the code has to been revendored and the new cloud-provider-gcp image reused to fix the test
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So this will be pending for and
double checking everything this failure logs context deadline for more than one hour to get an ip on the lb node initialized in during all the test it keeps failing missing the providerid I0228 18:46:27. 11 ] Recording status change CIDRNotAvailable event message for node gce-scale-cluster-master E0228 18:46:27. 11 ] Error updating CIDR for "gce-scale-cluster-master": failed to allocate cidr: Node gce-scale-cluster-master has no CIDRs the controller does not start to provision the loadbalancer until 40 minutes later Automatically polling progress: controllers or informers queues are blocked?
It seems the last run had been successful (03.02)
yeah, increased the number of workers to improve the performance , and bumped the existing controller manager image in CI that picked some fixes , let's see if this is enough ... it is a kind of "death by thousands cuts" problems that impact performance, as soon we start solving some of them we should start to see progress
let's watch this job for another week before closing.
Still flaking /reopen
Reopened this issue. <detailsIn response to : Still flaking /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This has failed in the last two runs. The next run is tomorrow, let's see what happens.
Last failing 03-09 is for . The 03-11 run passed. After is merged, we have 5 runs 3 passed 1 flake for . 1 flake for a lot of
Link recently merged PRs: 02-27: 02-29: 03-01: 03-02: 03-06:
3 failing test case in last run 03-13 [sig-network] LoadBalancers should be able to create an internal type load balancer [Slow] [sig-network] LoadBalancers should handle load balancer cleanup finalizer for service [Slow] [sig-network] Services GCE [Slow] should be able to create and tear down a standard-tier load balancer [Slow] I find those in as well.
Those loadbalancer tests does not work in AWS this fix is not being used, we need to revendor that code into cloud-provider-gcp and then publish a new image and then use this image in the CI
is a recurring failure pattern. Can something be done about this? Someone we (Release Team) from our side can track down to get this resolved? Any help on this would be greatly appreciated! ! Almost all failures you see above are time outs waiting for LBs.
Kubernetes e2e suite: [It] [sig-network] LoadBalancers should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [LinuxOnly] expandless 1h0m10s Kubernetes e2e suite: [It] [sig-network] LoadBalancers ESIPP [Slow] should handle updates to ExternalTrafficPolicy field expandless 1h0m4s Kubernetes e2e suite: [It] [sig-network] LoadBalancers ESIPP [Slow] should work for type=LoadBalancer expand_less 1h0m4s Last failure message like. One of the failure shows . Rate limiter? Can we tuning the for that?
The recent 2 flakes are like
looking into this today
Checking All loadbalancers tests timeout Take a simple one Starts at 17:37:45 and fails timing out waiting for the loadbalancer after 1 hour at 18:37 From the cloud controller manager logs IP is assigned to the loadbalancer at The service controller does not start to configure the loadbalancer until , The service controller has 5 workers, The number of services the controller can process in parallel are defined by overridden in the job with the same number If we analyze the time to process each service (there are metrics but there are no logs :/) (removing the multiprotocol-lb-test as is a loadbalancer that fails on purpose on GCE) For the job we can see there are only 5 services being processed at a time, and once one go to "ensured" another one is processed, so it is clear that the workers are blocked by the slow processing , see as example Because it makes a transition within the test , it has to wait for other test to continue. The job didn't show much changes in the last years, but there were several changes on the tests and a plausible theory is that before were running only a small number of them in parallel Current tests are very expensive and heavy, as they perform multiple checks in per test and mutate the services, but is what we have and the goal is to move them out of tree. By now, the best solution seems to ensure that there are only N load balancers tests running in parallel, with N the number of workers defined in the cloud-controller-manager that is 5 today. cc:
- thanks a ton for looking into it - this is extremely helpful! So to stop the bleeding, my suggestion would be actually to do the simplest possible thing, being bumping the number of workers. Limiting the number of load-balancer tests would require some changes, bumping the workers to 10 or 20 or whatever value is extremely simple change and according to your analysis above should also do the job. Thoughts?
I think that it will be interesting to test the performance of the service controller with a large number of workers,when users face this problem that will be the solution we recommend ... telling users to "please don't deploy more than 5 loadblanacers in parallel" sounds a bit odd ...
Exactly - that was also part of my reasoning - approved the PR, let's see how much it will help us.
/reopen to watch for a week?
Reopened this issue. <detailsIn response to : /reopen to watch for a week? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
last 3 times are success in a row: /close
Closing this issue. <detailsIn response to : last 3 times are success in a row: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Finally ... this was a tough one ..

Discussion:
var totalMillicores int BeforeEach(func() { <del> framework.SkipUnlessProviderIs("gce") </del> <ins> framework.SkipUnlessProviderIs("gce", "gke") </ins> ns = f.Namespace.Name nodes := framework.GetReadySchedulableNodesOrDie(f.Client) nodeCount := len(nodes.Items)