Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The issue has been observed on Azure but should also apply to other providers as well. On Azure we observed that there are orphaned Public IPs that don't belong to any Kubernetes LoadBalancer service but from looking at the tags and the name it is clear that Kubernetes created them. In the Azure Activity Log, you can see that the Public IP was deleted when the K8s service was deleted but recreated a few minutes later. That re-creation is also visible in the K8s controller manager logs as a reconcile of the service. In our case, we add additional worker nodes and delete services at the same time. I think this leads to the re-creation here Because the service list is retrieved at the beginning of the reconcile loop, services that are deleted while the loop is reconciling a recreated by that loop and afterwards orphaned. What you expected to happen: No orphaned public IPs How to reproduce it (as minimally and precisely as possible): Create a few Kubernetes Load Balancer services (we have around 20). Then scale your worker nodes and delete one service at the same time. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0 Cloud provider or hardware configuration: Azure OS (e.g: ): Core OS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
Thanks for reporting the issue. I think this should be fixed during v1.16 release cycle. /milestone v1.16
In the nodeSyncLoop -updateLoadBalancerHosts -lockedUpdateLoadBalancerHosts call chain, I don't see where the mutex lock is obtained.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 4 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
The issue would probably not able to be fixed in v1.16. AFAIK, no one is working on it now.
Thank you for the update! I'll go ahead and remove the issue from the milestone. If anything changes in the meanwhile, please let us know. /milestone clear
/assign
/assign
GitHub didn't allow me to assign the following users: chewong. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Doesn't LB Finalizer support fix this? We enabled this by default starting v1.16
Initial PR: Bump to beta in v1.16:

Discussion:
clusterNameKey = "kubernetes-cluster-name" ) <del> // GetLoadBalancer returns whether the specified load balancer exists, and </del> <ins> // GetLoadBalancer returns whether the specified load balancer and its components exist, and </ins> // if so, what its status is. func (az *Cloud) GetLoadBalancer(ctx context.Context, clusterName string, service *v1.Service) (status *v1.LoadBalancerStatus, exists bool, err error) { <del> _, status, exists, err = az.getServiceLoadBalancer(service, clusterName, nil, false) </del> <ins> // Since public IP is not a part of the load balancer on Azure, // there is a chance that we could orphan public IP resources while we delete the load blanacer (kubernetes/kubernetes#80571). // We need to make sure the existence of the load balancer depends on the load balancer resource and public IP resource on Azure. existsPip := func() bool { pipName, _, err := az.determinePublicIPName(clusterName, service) if err != nil { return false } pipResourceGroup := az.getPublicIPAddressResourceGroup(service) _, existsPip, err := az.getPublicIPAddress(pipResourceGroup, pipName) if err != nil { return false } return existsPip }() _, status, existsLb, err := az.getServiceLoadBalancer(service, clusterName, nil, false) </ins> if err != nil { <del> return nil, false, err </del> <ins> return nil, existsPip, err </ins> } <del> if !exists { </del> <ins> // Return exists = false only if the load balancer and the public IP are not found on Azure if !existsLb && !existsPip { </ins> serviceName := getServiceName(service) klog.V(5).Infof("getloadbalancer (cluster:%s) (service:%s) - doesn't exist", clusterName, serviceName) return nil, false, nil } <ins> // Return exists = true if either the load balancer or the public IP (or both) exists </ins> return status, true, nil }

// UpdateLoadBalancer updates hosts under the specified load balancer. func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) error { <ins> if !az.shouldUpdateLoadBalancer(clusterName, service) { klog.V(2).Infof("UpdateLoadBalancer: skipping service %s because it is either being deleted or does not exist anymore", service.Name) return nil } </ins> _, err := az.EnsureLoadBalancer(ctx, clusterName, service, nodes) return err }

return service.Spec.LoadBalancerIP, nil } <del> lbStatus, existsLb, err := az.GetLoadBalancer(ctx, clusterName, service) </del> <ins> _, lbStatus, existsLb, err := az.getServiceLoadBalancer(service, clusterName, nil, false) </ins> if err != nil { return "", err }

return &sg, nil } <ins> func (az *Cloud) shouldUpdateLoadBalancer(clusterName string, service *v1.Service) bool { _, _, existsLb, _ := az.getServiceLoadBalancer(service, clusterName, nil, false) return existsLb && service.ObjectMeta.DeletionTimestamp == nil } </ins> func logSafe(s *string) string { if s == nil { return "(nil)"

"strconv" "strings" "testing" <ins> "time" </ins> "github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-06-01/network" "github.com/Azure/go-autorest/autorest/to"

assert.Equal(t, test.expectedError, err != nil, "TestCase[%d]: %s", i, test.desc) } } <ins> func TestShouldUpdateLoadBalancer(t *testing.T) { testCases := []struct { desc                   string lbHasDeletionTimestamp bool existsLb               bool expectedOutput         bool }{ { desc:                   "should update a load balancer that does not have a deletion timestamp and exists in Azure", lbHasDeletionTimestamp: false, existsLb:               true, expectedOutput:         true, }, { desc:                   "should not update a load balancer that is being deleted / already deleted in K8s", lbHasDeletionTimestamp: true, existsLb:               true, expectedOutput:         false, }, { desc:                   "should not update a load balancer that does not exist in Azure", lbHasDeletionTimestamp: false, existsLb:               false, expectedOutput:         false, }, { desc:                   "should not update a load balancer that has a deletion timestamp and does not exist in Azure", lbHasDeletionTimestamp: true, existsLb:               false, expectedOutput:         false, }, } for i, test := range testCases { az := getTestCloud() service := getTestService("test1", v1.ProtocolTCP, nil, 80) if test.lbHasDeletionTimestamp { service.ObjectMeta.DeletionTimestamp = &metav1.Time{time.Now()} } if test.existsLb { lb := network.LoadBalancer{ Name: to.StringPtr("lb1"), LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{ FrontendIPConfigurations: &[]network.FrontendIPConfiguration{ { Name: to.StringPtr("atest1"), FrontendIPConfigurationPropertiesFormat: &network.FrontendIPConfigurationPropertiesFormat{ PublicIPAddress: &network.PublicIPAddress{ID: to.StringPtr("id1")}, }, }, }, }, } _, err := az.LoadBalancerClient.CreateOrUpdate(context.TODO(), "rg", *lb.Name, lb, "") if err != nil { t.Fatalf("TestCase[%d] meets unexpected error: %v", i, err) } } shouldUpdateLoadBalancer := az.shouldUpdateLoadBalancer(testClusterName, &service) assert.Equal(t, test.expectedOutput, shouldUpdateLoadBalancer, "TestCase[%d]: %s", i, test.desc) } } </ins>