Change:
BUG REPORT: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: CC: What happened: There are resource leaks in the cloud with respect to PublicIp and LoadBalancer Frontend Ip Pool and HealthProbe What you expected to happen: All created cloud resources, associated with a deleted service, should be destroyed. This is causing the Azure Load Balancer to fill up its frontend IP Pool which is currently limited to 10 entries. After a few resource leak, the new kuberntes services cannot get valid External IP since the Azure Load Balancer is full. How to reproduce it (as minimally and precisely as possible): Create 5 namespaces and deploy the guestbook-all-in- example. (making sure you turn on the LoadBalancer type for the frontend service) Wait for all of the 5 PublicIP to be provisioned in the Azure Load Balancer. Delete all of the 5 namespaces using the Poll the namespaces until they are totally gone from kubernetes and recreate them. Redeploy the guestbook-all-in- in all 5 namespaces. You will probably end up with 5 new IPs BUT not all of the previous ones will be deleted. Anything else we need to know: Kube-Controller-Manager Logs Service IP after the first pod creation Service IP after the re-creation Azure frontend Pool Screen Shot &lt;img width="799" alt="screen shot 2017-03-27 at 22 17 38" src=""
The following PublicIP were never removed
Does this repro on a single master cluster?
Yes, to ensure that all the logs were in the same file, I stopped all services on the other two kube-master. (e.g. kube-controller-manager, kube-apiserver, kube-scheduler) The log might still show entries when the configuration was setup as multi master
Update: I have retested with the deletion of the service/deployment instead of the namespace. I still have the same issue. As you can see 3 IP should have been delete from Azure. 52.167.217.196 52.167.223.67 52.167.222.61 &lt;img width="721" alt="screen shot 2017-03-28 at 18 34 55" src=""Find the Debug Log for the kube-controller-manager here (start from the last Service Restart Entry)
I'm seeing that the controller is calling delete for only , which seems to indicate that the controller only processed a single delete callback for the load balancers. I don't have an azure cluster handy. -- if it's possible, can you repro with: Start with empty cluster Ceate guestbook example Delete guestbook example, but don't create again. see if the IP resources leak, and post the kube-controller-manager logs. I would like to understand if this occurs without having to recreate the objects immediately.
"delete the guestbook example" == "delete the guestbook namespace" or "manually delete the guestbook's svc"? (I only ask to clarify because I suspect that if simple "single service delete" were failing, we'd see a lot more noise/issues about it)
I can confirm that if I wait after the deletion that all of the cleanup job occurs there is no leak. The problem is when the deleted service is recreated with the same name under the same namespace.
The creation needs to occur while the deletion of the resource in Azure is still ongoing. The apiserver returns that the resources are gone and cleaned which allow us to recreate.
Can you post the kube-controller-manager log for the simplified repro? (create -delete -create) with one copy of the guestbook app)?
let me extract that information from the logs. I created test4 last, deleted it, then recreated in a continuous sequence. without deploying anything else at the same time.
(Not to interrupt any diagnosing, but FYI: I tried to intentionally use the value from the cloudprovider-levlel in order to avoid any sort of conflict like this: That value is used to name the LB rules/probes and NSG rules, with the hope that it would avoid these sorts of conflicts. But I'm not sure how is formed... if it's deterministic based on the names of things, then I can imagine there could be some conflicts occurring.)
do you have a preferred logging level ? it seems that level 10 logs too much... The journal output is missing a bunch of lines. My original Log looks better and doesn't have missing lines. I think the first one was set at level 3
Here are the Azure Logs for the first test where test5/frontend's IP leak for example azureloadbalancer The azureloadbalancer with test5/frontend only shows thats the cloud provider was involved only for the creation and not the deletion. If we look at the we can see an example of the logs when the azure_loadbalancer gets called for deletion Adding a bit of logs from the controller related to The First time the Service was created and configured in Azure was at this time and line number () The second time it was created was at this time and line number () All of the specific logs below
the UID you use looks different between each delete/create First create/delete for test1/frontend Second Create
I think that the answer lies between line and in the first log file I attached from the OP
More investigation on my side For the 's service, we see this in the log when the controller manager tries to see if it needs to delete the service For the 's service we see the following instead The main difference between the two are those two lines  When it cannot delete the code is here. Which simply seems to return false, and ignores the deletion of the service all together. The weird part that I'm not too sure why, is that the deletion failures occur before the namespace was recreated and repopulated with resources. this is the last trace I can find for the deletion of the test5 namespace at the Namespace and its content get recreated here, at which is almost 1m30m later
I guess the difference can be explained by the that looks global for the client. The first pass, it get sets to which then trigger the exit right away.... So from the code, it just seems that the call to ` returns nothing....
and I have reproduced the issue with the smallest possible interaction. The following manifest (2 services with type:LoadBalancer) was used The Logs of the kube-controller-manager at lvl4 logging level from the time range of the test only. The IPs after the second create are The following IP and FrontEnd configuration leaked The commands executed were PS: Note that I was not able to reproduce with a manifest containing a single service object. As soon as I the second to the manifest, I was able to reproduce the issue right away. Azure's LB snapshot <img width="788" alt="screen shot 2017-04-06 at 17 38 43" src=""
So this happens even outside of the namespace deletion? Is there anything obvious in the k-c-m logs? I'm very curious to know if this is a bug in the ServiceController or in the Azure cloudprovider code.
Also, do you see any errors in the Audit Log in the Portal for this resource group?
there are no issues in Azure' audit log nor the Activity logs for this RG. Regarding your question about the source of the issue, I would tend to say that the issue is in the ServiceController. Here's my reasoning. For the service we have the following lines Creation 1 Deletion Creation 2 Now for the service we have the following lines Creation 1 Deletion Creation 2 As you can see there is not even a call from the class to issue a delete based on this log entry ,hence the class is never called. Now, as to why the class forgets about the deletion and moves on to the recreation. So far, my best bet is that the deletion process and handled one at the time, and since it takes a while for the IP to be removed in Azure, the logic in the detect that frontend2 was created and is in need of an IP. Right before the frontend2 second creation I see this. Also, the order of creation/deletion is creates test/frontend creates test/frontend2 deletes test/frontend creates test/frontend2 creates test/frontend
So I looked at the code and I see that my hypothesis was true. (if I'm not mistaken) Here's the break down of my analysis. The Cloud-controller-manager calls the command with ConcurrentServiceSyncs hardcoded to (it cannot be change either from the command line) The service controller creates a single worker thread that uses a workingQueue to get the list of services to process. The worker's sole purpose is to synch the Service. Upon the deletion of both services, the queue gets filled up with two entries: and The worker pops out and starts processing the deletion. It takes around 3mins. Before the end of the deletion of , both services are recreated and the queue gets repopulated. After is deleted, is removed from the queue to be processed. In the method, the service is queried to see if it is present Since it was readded (with the same name) there are no Error returned, hence no deletion requested. From here, I'm not 100% sure what would be best. what is your take here ? should the SynchService stop querying for the Service existance in the apiserver and be given the Instruction to add/remove the service directly from the Informer's DeleteFunc method? Knowing that there is only a single worker. The queue could look something like this So all of the creation and deletion would be process in the right order. The only problem I see with this right now is that there seem to be a retry mechanism that could re-queue a delete action after a recreate... I guess the safest bet is really have a way to prevent the apiserver from accepting the same service to be created if the CloudProviderPublicIP has not been removed... This is going back to this Issue I guess(). Even though the issue is requesting that the user be able to query the deletion Status, here we would want the apiserver to not delete the service from his list until it is fully deleted...
This looks to be me like an A-B-A () problem with the cloud resource. Let me do some digging...
This looks to be another instance of If the reconciliation was performed periodically, the IPs will be garbage collected.
could you explain me what the reconciliation would provide to the serviceController ? I've read the issues you linked, but I'm still not 100% sure I understand the proposed logic. (probably because I still don't know how the service controller fully works -- mostly with respect to caching)
The idea is there would be a continual loop over the cloud load balancers to see if there are any configured that should be there. If there is a mismatch, the load balancer will be deleted. Admittedly, this is not a hugely satisfying answer. It's along the lines of: there is no sync up of the Kubernetes state with the cloud provider state in the service controller and this can lead to stranded/inconsistent resources.
I see. thanks for the explanation. What do you think about having the master prevent the creation of a service that was previously deleted until the cloud controller manager is done with his clean up job ? (e.g. all configuration are deleted in the cloud's LB) That should cover the use case where services are deleted and recreated almost back to back.
I still don't even understand why there's a conflict when they're deleted/recreated back to back. The Service UUID should be different. Or is it because the SerivceController stops trying to delete the previous service because ServiceController only tracks by name and not by the UUID?
I guess either way there's still agreement that this is a bug in ServiceController layer... please tell me if this isn't the case and we need to be more actively engaged from the Azure side.
-- as far as I can tell, it is tracked by name and not UUID
correct, the key used to track consist of the string:
This seems to be a result of using the namespace/service name in . I can reproduce this easily by causing the ServiceController to get busy syncing a new service and then deleting and recreating a different service like this: In the controller-manager I see logs like this (I'm using the Openstack provider in this case): You can see that the ServiceController essentially "lost" the delete because the name was the same. Because the calling syncService does not work concurrently (here ), by the time it finished handling and started on , the service had already been recreated. So when sync service checked it (here ) it already existed again and dropped into the case instead of the case like we would want. The end result is that the not only the public IP but the cloud provider's load balancer is orphaned.
Any updates here?
No time to take this up in this release -- punting.
/sig azure
/area platform/azure
remove from 1.7 milestone
Move to next milestone per sig/network's request.
any chance this could be picked up for 1.8 ?
I'll take a stab at this... Note I'm going to remove the sig/azure since this seems to be a global issue in the
I've run into this today (I think - at least the symptoms seem to be the same, with services perpetually stuck in for public IP, and I have been creating, deleting and recreating services perhaps too quickly). Is there a workaround? (I saw this problem on a test cluster, so I've resorted to deleting the cluster and re-creating it for now, but I'd like to avoid that in the future if I can, since it takes a while... Also, it's not done yet, so I don't know if it actually solves my problem - but if it doesn't, my problem is likely not related to this issue.) When I do there's only three IP:s listed for my subscription, so I know I'm not hitting the limit of simultaneously provisioned IP:s (the defaults allow 20) - do you know of other limits I could be hitting? Are there other resources except the public IP addresses that leak, that could be subject to similar caps? I'm running k8s 1.9.2 on the client, and 1.8.7 on the cluster.

Discussion:
// indicating whether processing should be retried; zero means no-retry; otherwise // we should retry in that Duration. func (s *ServiceController) processServiceUpdate(cachedService *cachedService, service *v1.Service, key string) (error, time.Duration) { <ins> if cachedService.state != nil { if cachedService.state.UID != service.UID { err, retry := s.processLoadBalancerDelete(cachedService, key) if err != nil { return err, retry } } } </ins> // cache the service, we need the info for service deletion cachedService.state = service err, retry := s.createLoadBalancerIfNeeded(key, service)

if !ok { return fmt.Errorf("Service %s not in cache even though the watcher thought it was. Ignoring the deletion.", key), doNotRetry } <ins> return s.processLoadBalancerDelete(cachedService, key) } func (s *ServiceController) processLoadBalancerDelete(cachedService *cachedService, key string) (error, time.Duration) { </ins> service := cachedService.state // delete load balancer info only if the service type is LoadBalancer if !wantsLoadBalancer(service) {