Change:
All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.
The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project. I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.

Discussion:
CreateTCPLoadBalancer(name, region string, externalIP net.IP, ports []int, hosts []string, affinityType api.AffinityType) (string, error) // UpdateTCPLoadBalancer updates hosts under the specified load balancer. UpdateTCPLoadBalancer(name, region string, hosts []string) error <del> // DeleteTCPLoadBalancer deletes a specified load balancer. DeleteTCPLoadBalancer(name, region string) error </del> <ins> // EnsureTCPLoadBalancerDeleted deletes the specified load balancer if it // exists, returning nil if the load balancer specified either didn't exist or // was successfully deleted. // This construction is useful because many cloud providers' load balancers // have multiple underlying components, meaning a Get could say that the LB // doesn't exist even if some part of it is still laying around. EnsureTCPLoadBalancerDeleted(name, region string) error </ins> } // Instances is an abstract, pluggable interface for sets of instances.

return f.Err } <del> // DeleteTCPLoadBalancer is a test-spy implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. </del> <ins> // EnsureTCPLoadBalancerDeleted is a test-spy implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. </ins> // It adds an entry "delete" into the internal method call record. <del> func (f *FakeCloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (f *FakeCloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> f.addCall("delete") return f.Err }

return nil } <del> // DeleteTCPLoadBalancer is an implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. func (gce *GCECloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> // EnsureTCPLoadBalancerDeleted is an implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. func (gce *GCECloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> op, err := gce.service.ForwardingRules.Delete(gce.projectID, region, name).Do() if err != nil && isHTTPErrorCode(err, http.StatusNotFound) { glog.Infof("Forwarding rule %s already deleted. Continuing to delete target pool.", name)

err = gce.waitForRegionOp(op, region) if err != nil { glog.Warningf("Failed waiting for Forwarding Rule %s to be deleted: got error %s.", name, err.Error()) <ins> return err </ins> } } op, err = gce.service.TargetPools.Delete(gce.projectID, region, name).Do()

"fmt" "io" "net" <ins> "net/http" </ins> "regexp" "time"

return nil } <del> func (lb *LoadBalancer) DeleteTCPLoadBalancer(name, region string) error { glog.V(4).Infof("DeleteTCPLoadBalancer(%v, %v)", name, region) vip, err := getVipByName(lb.network, name) if err != nil { return err } pool, err := pools.Get(lb.network, vip.PoolID).Extract() if err != nil { return err </del> <ins> func (lb *LoadBalancer) EnsureTCPLoadBalancerDeleted(name, region string) error { glog.V(4).Infof("EnsureTCPLoadBalancerDeleted(%v, %v)", name, region) // TODO(#8352): Because we look up the pool using the VIP object, if the VIP // is already gone we can't attempt to delete the pool. We should instead // continue even if the VIP doesn't exist and attempt to delete the pool by // name. vip, vipErr := getVipByName(lb.network, name) if vipErr == ErrNotFound { return nil } else if vipErr != nil { return vipErr } // It's ok if the pool doesn't exist, as we may still need to delete the vip // (although I don't believe the system should ever be in that state). pool, poolErr := pools.Get(lb.network, vip.PoolID).Extract() if poolErr != nil { detailedErr, ok := poolErr.(*gophercloud.UnexpectedResponseCodeError) if !ok || detailedErr.Actual != http.StatusNotFound { return poolErr } </ins> } <ins> poolExists := (poolErr == nil) </ins> <del> // Have to delete VIP before pool can be deleted err = vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil { </del> <ins> // We have to delete the VIP before the pool can be deleted, so we can't // continue on if this fails. // TODO(#8352): Only do this if the VIP exists once we can delete pools by // name rather than by ID. err := vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil && err != ErrNotFound { </ins> return err } // Ignore errors for everything following here <del> for _, monId := range pool.MonitorIDs { pools.DisassociateMonitor(lb.network, pool.ID, monId) </del> <ins> if poolExists { for _, monId := range pool.MonitorIDs { // TODO(#8352): Delete the monitor, don't just disassociate it. pools.DisassociateMonitor(lb.network, pool.ID, monId) } pools.Delete(lb.network, pool.ID) </ins> } <del> pools.Delete(lb.network, pool.ID) </del> return nil }

cachedService.service = service s.cache.set(namespacedName.String(), cachedService) case cache.Deleted: <del> err := s.ensureLBDeleted(service) </del> <ins> err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region) </ins> if err != nil { return err, retryable }

// we can recreate it cleanly. if cachedService.Spec.CreateExternalLoadBalancer { glog.Infof("Deleting existing load balancer for service %s that needs an updated load balancer.", namespacedName) <del> if err := s.ensureLBDeleted(cachedService); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(cachedService), s.zone.Region); err != nil { </ins> return err, retryable } }

} else if exists { glog.Infof("Deleting old LB for previously uncached service %s whose endpoint %s doesn't match the service's desired IPs %v", namespacedName, endpoint, service.Spec.PublicIPs) <del> if err := s.ensureLBDeleted(service); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region); err != nil { </ins> return err, retryable } }

return nil } <del> // Ensures that the load balancer associated with the given service is deleted, // doing the deletion if necessary. Should always be retried upon failure. func (s *ServiceController) ensureLBDeleted(service *api.Service) error { // This is only needed because not all delete load balancer implementations // are currently idempotent to the LB not existing. if _, exists, err := s.balancer.GetTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } else if !exists { return nil } if err := s.balancer.DeleteTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } return nil } </del> // ListKeys implements the interface required by DeltaFIFO to list the keys we // already know about. func (s *serviceCache) ListKeys() []string {