Change:
the kube-apiserver's log print this
/sig kube-apiserver
/triage support /remove-kind bug /sig cli
Hitting this too with 1.13.5
Your cluster was not clean before you installing kubernetes cluster, you should deployed kubernetes on your env before. Clean you ENV, and re-deploy again could resolve this issue.
I use the , It is always new
I hava seem issue. until now,not fix. kube-apiserver logs at below:
I use kubeadm install k8s on one vps. yesterday everything is ok.
I hava seem issue. until now,not fix. kube-apiserver logs at below: k8s-master-1 kube-apiserver[]: E0617 13:55:33. ] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.21.230.246, ResourceVersion: 0, AdditionalErrorMsg:
trouble resolved by set current apiserver
I have too, v1.14.2 v1.14.4 v1.15.1 apiserver exit and kubeadm hung.
The error likely came from leaseEndpointReconciler#doReconcile :
1 ] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.56.100, ResourceVersion: 0, AdditionalErrorMsg:
can you explain how to please? It will help me a lot.
on worker server run command
, you mean init with --apiserver-advertise-address on master, don't you?
no on worker node
thanks, i had been creating HA cluster, that should not be your situation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm running into the same issue. I'm able to run on an ubuntu box and it works but on Centos, I get the following error on kube-apiserver static pod and it restart: Kubeadm version: Docker version: Kubelet --version: kubeadm command:
Was there some clue from the log prior to 'Unable to remove old endpoints' ? thanks
Apiserver logs:
Considering 'StorageError: key not found', the endpoint might not exist. The subsequent "Create" seemed to succeed. Was there other symptom you noticed after 17:07:01 ? Thanks
No, it just crashes and restarts after that
Was there panic after 0120 17:07:10. ? Please share more log beyond 0120 17:07:10. if you have it. Thanks
Yes, the container panics and stops so no more logs after it. I tried querying etcd and there is no keys under , hence the key not found error but the question is why is it trying to remove a key that does not exist. Maybe, I have some old artifact remaining on my machine. I did to cleanup my old kubeadm setup
I take it back, there are more logs.
The issue in my case was k8s api-server was using ipv6 address ([::1]) which was not working. I disabled ipv6 and it fixed the issue for me
ipv6 has been turned off in my environment
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I have the same problem
I have the same problem. Have you solved it
Hi!. Had same issue, solved it right now by upgrading docker on master host. Was version 18.09, now 19.03, after upgrade works without any errors. Solution inspired by this question , but without reinstall docker. Just performed
I don't use kubeadm
I have a same problem, I used binary file Problem: Version:1.18.3
your second erroryou should disable the ipv6
I turned off IPv6But the error still exists I care more about the first question
I emailed you. qq mail
github
та же проблема
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
We see this error while we are having PLEG issues with kubelet. The below document says how to fix the PLEG issue. However, KOPS doesn't support in kubelet. There is already an issue open for this
not sure whats causing this. I had virtualbox VMs (3) on windows 10 amd64. and this issue happened after restarts, spent days to figure things out, including trying to set kubernetes up in different distros of linux. then i switched to vmware player, no issue so far. Again, my scenario is arbitrary, if anyone has the same setup as me, probably give vmware a try.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
same error on wsl2 when restarting the kube-apiserver
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten </details
hit this this morning after the node my kind cluster was on restarted. curious what the resolution could be.
It is incredible that this problem has existed for more than 2 years and has not been solved. I encountered this error when deploying k8s using binary files, I tried v1.18 - v1.21, and I checked all the search results on google. Unfortunately, it took me 3 days to still not be able to solve this problem.
Small probability is caused by ipv6 try this If not works, could try this:
/remove-sig cli
I got the same errors with binary in my cluster
/sig api-machinery /remove-kind support /kind bug
/assign
anybody can share the full logs of the apiserver when this happens?
hmm the error is just informative, it happens for a new installation, it doesn't exist, throws the error and later the lease is this error in the log is a red herring, most of the cases reported are related to deployment issues, wrong IP family, KIND cluster restarts, ... I'm going to close this to avoid converting this issue in a magnet, TL;DR is not a problem, if apiserver doesn't start check further in the logs, the reason has to be there /close
Closing this issue. <detailsIn response to : hmm the error is just informative, it happens for a new installation, it doesn't exist, throws the error and later the lease is this error in the log is a red herring, most of the cases reported are related to deployment issues, wrong IP family, KIND cluster restarts, ... TL;DR is not a problem, if apiserver doesn't start check further in the logs, the reason has to be there /close </details
Let's see if we can improve the error message
/triage accepted
showed up in kube-apiserver log even if configured because that's how golang worked, reproduceable code in: controllers in kube-apiserver accessed because net.InterfaceAddrs() returned (when ipv6 not disabled) and listening on [::]:6443 make that then following function returned [::1]
Faced the same issue today: And api-server didn't want to start. In my case it turned out to be that due to a network accident all our control-planes changed their ip addresses. I managed to start all api-servers and have them work as usual by changing all entries of old ip addresses to the new ones in . The error still presents, but everything works just fine. Maybe it is because of an old leases record with old ips somewhere in etcd (not sure).
Just ran into the same issue after updating from 1.23.17 to 1.24.15 (via kubeadm). The kube-api is unreachable. When we reboot, the API is visible shortly until the calico pods start (calico v 3.25.1). We tried both and with the same outcome.
following this guide everything just works fine on fresh ubuntu 20 and has the above issue on fresh ubuntu 22. did not get to the reason why this happens.

Discussion:
"k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" genericapiserver "k8s.io/apiserver/pkg/server" <ins> "k8s.io/apiserver/pkg/storage" </ins> corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/rest" "k8s.io/klog/v2"

// Reconcile during first run removing itself until server is ready. endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts) <del> if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil { klog.Errorf("Unable to remove old endpoints from kubernetes service: %v", err) </del> <ins> if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err == nil { klog.Error("Found stale data, removed previous endpoints on kubernetes service, apiserver didn't exit successfully previously") } else if !storage.IsNotFound(err) { klog.Errorf("Error removing old endpoints from kubernetes service: %v", err) </ins> } repairClusterIPs := servicecontroller.NewRepair(c.ServiceClusterIPInterval, c.ServiceClient, c.EventClient, &c.ServiceClusterIPRange, c.ServiceClusterIPRegistry, &c.SecondaryServiceClusterIPRange, c.SecondaryServiceClusterIPRegistry)

klog.Infof("Shutting down kubernetes service endpoint reconciler") c.EndpointReconciler.StopReconciling() if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil { <del> klog.Error(err) </del> <ins> klog.Errorf("Unable to remove endpoints from kubernetes service: %v", err) </ins> } }()