Code change:
return nil } <del> func (lb *LoadBalancer) DeleteTCPLoadBalancer(name, region string) error { glog.V(4).Infof("DeleteTCPLoadBalancer(%v, %v)", name, region) vip, err := getVipByName(lb.network, name) if err != nil { return err } pool, err := pools.Get(lb.network, vip.PoolID).Extract() if err != nil { return err </del> <ins> func (lb *LoadBalancer) EnsureTCPLoadBalancerDeleted(name, region string) error { glog.V(4).Infof("EnsureTCPLoadBalancerDeleted(%v, %v)", name, region) // TODO(#8352): Because we look up the pool using the VIP object, if the VIP // is already gone we can't attempt to delete the pool. We should instead // continue even if the VIP doesn't exist and attempt to delete the pool by // name. vip, vipErr := getVipByName(lb.network, name) if vipErr == ErrNotFound { return nil } else if vipErr != nil { return vipErr } // It's ok if the pool doesn't exist, as we may still need to delete the vip // (although I don't believe the system should ever be in that state). pool, poolErr := pools.Get(lb.network, vip.PoolID).Extract() if poolErr != nil { detailedErr, ok := poolErr.(*gophercloud.UnexpectedResponseCodeError) if !ok || detailedErr.Actual != http.StatusNotFound { return poolErr } </ins> } <ins> poolExists := (poolErr == nil) </ins> <del> // Have to delete VIP before pool can be deleted err = vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil { </del> <ins> // We have to delete the VIP before the pool can be deleted, so we can't // continue on if this fails. // TODO(#8352): Only do this if the VIP exists once we can delete pools by // name rather than by ID. err := vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil && err != ErrNotFound { </ins> return err } // Ignore errors for everything following here <del> for _, monId := range pool.MonitorIDs { pools.DisassociateMonitor(lb.network, pool.ID, monId) </del> <ins> if poolExists { for _, monId := range pool.MonitorIDs { // TODO(#8352): Delete the monitor, don't just disassociate it. pools.DisassociateMonitor(lb.network, pool.ID, monId) } pools.Delete(lb.network, pool.ID) </ins> } <del> pools.Delete(lb.network, pool.ID) </del> return nil }

Related discussion:
text: The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.