Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? no. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): service endpoints missing Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: manual Others: What happened: Service in namespace sees no endpoints that are available What you expected to happen: For it to see available endpoints How to reproduce it (as minimally and precisely as possible): Not sure I can. It appeared on the latest remove/replace of worker nodes, but was fine before Anything else we need to know: I have a traefik Ingress controller defined and a service for it: Selectors match, the pods are running just fine, yet the endpoints do not show up. I suspect if I delete the service and recreate it, it probably will pick it up, but I shouldn't have to, and am concerned it might occur again; rather actually debug it. How can I check what kube is doing to affiliate it? How often does it check? Where?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
The system is asking for a SIG, but it isn't clear which SIG is appropriate for this. Taking best guess, apologies if I am wrong. /sig sig/apps /sig sig/scheduling
/sig network
can you provide the following output?
I wish I could... in the week since, the cluster has been deleted and recreated several times. AWS instances may be cheap, but still aren't free. Client didn't want them sticking around too long for the cost.
Should we close this issue pending more information or keep it open for someone to test out?
we are killing and recreating clusters with some regularity over these weeks. I am happy if you want to close it since I cannot recreate it yet. Just let me reopen it if I manage to recreate and get more info. Just tell me, please, if it does come up again, what additional data should I get? There isn't much data in the docs on how the service resolves what endpoints it should use and how to figure out why it is misbehaving when it does. Happy to take pointers.
Please reopen when it happens again. Please record related endpoints objects and pod objects.
I'm facing same issue with Nginx by trying to expose the nodePort for Nginx-ingress-controller pod
Try .
Your service is using while your pods are using .
Sorry, didn't notice that
OS: ubuntu 16.04 kubernetes version: v1.8.4 hi, I have the same problem now. $ kc get endpoints -njenkins This situation has been going on for a long time and no solution has been found. Help out. Thanks
Same the problem, svc lost the endpoint after a few time. If deleting the pod, then the svc endpoint is show up again. How to debug this problem?
Im having the same problem on Windows Nodes, after some hours all svc on one node loses their endpoints. Kubernetes V1.15
May be fixed by this?
same issue for a custom helm that I created: kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T10:52:43Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-03T04:12:37Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} kubectl describe service/vnfdb-ui Name: vnfdb-ui Namespace: test-helm Labels: Annotations: <noneSelector: , Type: LoadBalancer IP: 10.105.168.158 Port: http 80/TCP TargetPort: http/TCP NodePort: http /TCP Endpoints: <noneSession Affinity: None External Traffic Policy: Cluster Events: <none$ kubectl get pods -l , NAME READY STATUS RESTARTS AGE vnfdb-ui--ttrwm 1/1 Running 0 9m $ kubectl get endpoints vnfdb-ui NAME ENDPOINTS AGE vnfdb-ui <none9m33s $ kubectl get endpoints vnfdb-ui -o yaml apiVersion: v1 kind: Endpoints metadata: creationTimestamp: "2019-08-24T21:28:58Z" labels: vnfdb Tiller ui "1.0" ui-0.1.0 name: vnfdb-ui namespace: test-helm resourceVersion: "" selfLink: /api/v1/namespaces/test-helm/endpoints/vnfdb-ui uid: -c6b6-11e9-a0d5- Any clue what is wrong here and how can I fix it?
Got it, apparently realized that its not only the selector that has to match labels. The port names should be the same in the pod definition and services definition as well. The latter was the culprit in my case.
Same the problem.My service and pod is running. But the service do not has endpoint,and this situation lasts for 10 hours. It is OK after I restart controller manager. My last operation was to delete the service and create the service again. I have not done any other operations since then. Kubernetes V1.11.5
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Good catch!
Happens to us once in a while: We have a process creating a service, then a test deleting that same service, which ends up being automatically recreated. Once in a while (every ~100 test), the re-created service never gets any endpoints, even after waiting for several hours. I'm fairly confident recreating the service again fixes it, but that's not a solution. It just happened on a GKE cluster in version , for which I don't have controller-manager logs available unfortunately. I'll try to reach GCP support, they may be able to troubleshoot more from the controller-manager logs. Edit: the endpoint magically appeared after 22 hours.
Thanks for the detailed bug report! I'll try to recreate this scenario with some new tests. /assign
A quick follow up to let you know we haven't forgotten about this issue. From what we can tell, this is being caused by an Endpoints informer cache that can potentially be out of date when syncService() runs. Essentially syncService() would be called twice - once for the delete and once for the create event. There's a chance that when the second run of syncService() happens the Endpoints cache may not have been updated with the latest run. If that happens, syncService() will not make any changes because it sees an endpoints resource that has all the right ports and addresses set. is working on a fix here that will mirror what we've done for the EndpointSlice controller with the . This will involve watching Endpoints changes and calling syncService() if the change has not already been accounted for by the controller. As far as I can tell, this should be a relatively rare bug. It should only happen if the cache is out of date, the recreated Service is identical, and the Pods have not changed at all. Given the relative complexity of the fix here, I'm not sure how far we'll be able to backport this. At the very least, we're hoping to have a fix in for 1.19.
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think is almost ready, sorry for not catching the lifecycle close earlier.
Dude, you saved my life.

Discussion:
e.podLister = podInformer.Lister() e.podsSynced = podInformer.Informer().HasSynced <ins> endpointsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ DeleteFunc: e.onEndpointsDelete, }) </ins> e.endpointsLister = endpointsInformer.Lister() e.endpointsSynced = endpointsInformer.Informer().HasSynced

e.queue.Add(key) } <ins> func (e *EndpointController) onEndpointsDelete(obj interface{}) { key, err := controller.KeyFunc(obj) if err != nil { utilruntime.HandleError(fmt.Errorf("Couldn't get key for object %+v: %v", obj, err)) return } e.queue.Add(key) } </ins> // worker runs a worker thread that just dequeues items, processes them, and // marks them done. You may run as many of these in parallel as you wish; the // workqueue guarantees that they will not end up processing the same service

return httptest.NewServer(mux), &fakeEndpointsHandler } <ins> // makeBlockingEndpointDeleteTestServer will signal the blockNextAction channel on endpoint "POST" & "DELETE" requests. All // block endpoint "DELETE" requestsi will wait on a blockDelete signal to delete endpoint. If controller is nil, a error will // be sent in the response. func makeBlockingEndpointDeleteTestServer(t *testing.T, controller *endpointController, endpoint *v1.Endpoints, blockDelete, blockNextAction chan struct{}, namespace string) *httptest.Server { handlerFunc := func(res http.ResponseWriter, req *http.Request) { if controller == nil { res.WriteHeader(http.StatusInternalServerError) res.Write([]byte("controller has not been set yet")) return } if req.Method == "POST" { controller.endpointsStore.Add(endpoint) blockNextAction <- struct{}{} } if req.Method == "DELETE" { go func() { // Delay the deletion of endoints to make endpoint cache out of sync <-blockDelete controller.endpointsStore.Delete(endpoint) controller.onEndpointsDelete(endpoint) }() blockNextAction <- struct{}{} } res.WriteHeader(http.StatusOK) res.Write([]byte(runtime.EncodeOrDie(clientscheme.Codecs.LegacyCodec(v1.SchemeGroupVersion), &v1.Endpoints{}))) } mux := http.NewServeMux() mux.HandleFunc("/api/v1/namespaces/"+namespace+"/endpoints", handlerFunc) mux.HandleFunc("/api/v1/namespaces/"+namespace+"/endpoints/", handlerFunc) mux.HandleFunc("/api/v1/namespaces/"+namespace+"/events", func(res http.ResponseWriter, req *http.Request) {}) mux.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) { t.Errorf("unexpected request: %v", req.RequestURI) http.Error(res, "", http.StatusNotFound) }) return httptest.NewServer(mux) } </ins> type endpointController struct { *EndpointController podStore       cache.Store

} } <ins> // TestMultipleServiceChanges tests that endpoints that are not created because of an out of sync endpoints cache are eventually recreated // A service will be created. After the endpoints exist, the service will be deleted and the endpoints will not be deleted from the cache immediately. // After the service is recreated, the endpoints will be deleted replicating an out of sync cache. Expect that eventually the endpoints will be recreated. func TestMultipleServiceChanges(t *testing.T) { ns := metav1.NamespaceDefault expectedSubsets := []v1.EndpointSubset{{ Addresses: []v1.EndpointAddress{ {IP: "1.2.3.4", NodeName: &emptyNodeName, TargetRef: &v1.ObjectReference{Kind: "Pod", Name: "pod0", Namespace: ns}}, }, }} endpoint := &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns, ResourceVersion: "1"}, Subsets:    expectedSubsets, } controller := &endpointController{} blockDelete := make(chan struct{}) blockNextAction := make(chan struct{}) stopChan := make(chan struct{}) testServer := makeBlockingEndpointDeleteTestServer(t, controller, endpoint, blockDelete, blockNextAction, ns) defer testServer.Close() *controller = *newController(testServer.URL, 0*time.Second) addPods(controller.podStore, ns, 1, 1, 0, ipv4only) go func() { controller.Run(1, stopChan) }() svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns}, Spec: v1.ServiceSpec{ Selector:  map[string]string{"foo": "bar"}, ClusterIP: "None", Ports:     nil, }, } controller.serviceStore.Add(svc) controller.onServiceUpdate(svc) // blockNextAction should eventually unblock once server gets endpoint request. waitForChanReceive(t, 1*time.Second, blockNextAction, "Service Add should have caused a request to be sent to the test server") controller.serviceStore.Delete(svc) controller.onServiceDelete(svc) waitForChanReceive(t, 1*time.Second, blockNextAction, "Service Delete should have caused a request to be sent to the test server") // If endpoints cache has not updated before service update is registered // Services add will not trigger a Create endpoint request. controller.serviceStore.Add(svc) controller.onServiceUpdate(svc) // Ensure the work queue has been processed by looping for up to a second to prevent flakes. wait.PollImmediate(50*time.Millisecond, 1*time.Second, func() (bool, error) { return controller.queue.Len() == 0, nil }) // Cause test server to delete endpoints close(blockDelete) waitForChanReceive(t, 1*time.Second, blockNextAction, "Endpoint should have been recreated") close(blockNextAction) close(stopChan) } func TestEndpointsDeletionEvents(t *testing.T) { ns := metav1.NamespaceDefault testServer, _ := makeTestServer(t, ns) defer testServer.Close() controller := newController(testServer.URL, 0) store := controller.endpointsStore ep1 := &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{ Name:            "foo", Namespace:       ns, ResourceVersion: "rv1", }, } // Test Unexpected and Expected Deletes store.Delete(ep1) controller.onEndpointsDelete(ep1) if controller.queue.Len() != 1 { t.Errorf("Expected one service to be in the queue, found %d", controller.queue.Len()) } } </ins> func stringVal(str *string) string { if str == nil { return "nil" } return *str } <ins> // waitForChanReceive blocks up to the timeout waiting for the receivingChan to receive func waitForChanReceive(t *testing.T, timeout time.Duration, receivingChan chan struct{}, errorMsg string) { timer := time.NewTimer(timeout) select { case <-timer.C: t.Errorf(errorMsg) case <-receivingChan: } } </ins>