Code change:
This issue was supposed to have fixed the issue where endpoint stay out of sync I also wrote a small script which would get me the out of sync endpoints compared to the endpointslices I expect the endpoints to eventually sync and reflect the most upto date information. I have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. No response Client Version: v1.29.7 Kustomize Version: v5.0.4-0.- Server Version: v1.29.7 <details</detailsalmalinux-9 <details</detailscri-o No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
from the logs, endpoint controller was retrying test1/test-qa in the expected way. It has retried 5 times and the next retry should be after 320ms, which should long enough for the informer cache to get the updated endpoints. If there is no more logs related to the failure, I think it may have synced it successfully. The python script used to detect the failure seems capturing diff between endpoints and endpointslice at a moment, but that could happen if there is any Pod change in the cluster, and technically the script is comparing endpoints of one moment with endpointslice of another moment. So did you see the endpoints not being synced after endpoint controller stopped retrying it?
Yeah i still see it and see similar messages continuously. Let me post some additional log messages after the one i posted above.
Observed the same issue in 1.29 and 1.30. I keep seeing the same error then new pods were behind a service. And the endpoint never catch up after all pods are settled.
could you provide complete log of kube-controller-manager and the yamls of the related endpoints and endpointslice?
/assign Are you able to triage this since you are already looking at this?
Yes i can get that over by EOD today
Sure, I'm waiting for more logs and the content of the endpoints to understand how it happened. Currently there are only retry logs which is expected in some cases.
We found a customer that triggered an edge case in your change of , which will cause Endpoints never updated. They have a mutatingWebhook on Endpoints object that automatically injects a label, let's say . syncServices, the endpoints controller , and they won't match since Endpoints have the extra "dummy-k: value". endpoints controller will by removing the label. mutatingWebhook will automatically adds back the label as part of the update request. From the APIServer's POV, the Endpoints object don't have any changes, thus the update request will succeed but returns the same resourceVersion as before. change in will mark that resourceVersion as stale. future pods events won't be handled correctly due to the step didn't result in a new resourceVersion. I think we should fix this by check the resulting resourceVersion from the update request, and only mark the old resourceVersion as stale when a new resourceVersion is generated. Not sure whether this is the same case others in this thread encountered :D
thanks for explaining the use case and the analysis. I did analyze the code after the issue was opened and suspected the only possibility is the returned resource version somehow equals to the one in the request but didn't know how it could happen after it has checked the desired endpoints and the current endpoints are different: Now I understand this could happen when the endpoints appear different on client side but are mutated to be same on server side. Do you plan to create a PR since you have proposed a fix? could you confirm if this is also your case, i.e. there is a mutatingWebhook for endpoints objects that could make the endpoints different from what endpoints controller expects?
Yeah, i can do a fix for this
- can we assign to you?
Sure, I'm already assigned. Given that we've had three user reports on this issue, I think it's important to address it as soon as possible. Could you please let me know when you plan to create the PR? If you're currently occupied, I'm happy to handle it and submit the PR.
Sorry, were occupied by some internal incidents last week. Just created a WIP PR for it. Will test it in kops for this specific webhook scenario in the following two days and and remove the WIP flag
/assign

Related discussion:
text: svc.Spec.ExternalName = "google.com" return svc } <ins> func TestEndpointTruncate(t *testing.T) { // Disable ServiceAccount admission plugin as we don't have serviceaccount controller running. server := kubeapiservertesting.StartTestServerOrDie(t, nil, framework.DefaultTestServerFlags(), framework.SharedEtcd()) defer server.TearDownFn() client, err := clientset.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("Error creating clientset: %v", err) } informers := informers.NewSharedInformerFactory(client, 0) tCtx := ktesting.Init(t) epController := endpoint.NewEndpointController( tCtx, informers.Core().V1().Pods(), informers.Core().V1().Services(), informers.Core().V1().Endpoints(), client, 0) // Start informer and controllers informers.Start(tCtx.Done()) go epController.Run(tCtx, 1) // Create namespace ns := framework.CreateNamespaceOrDie(client, "test-endpoints-truncate", t) defer framework.DeleteNamespaceOrDie(client, ns, t) // Create a pod with labels basePod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "test-pod", Labels: labelMap(), }, Spec: v1.PodSpec{ NodeName: "fake-node", Containers: []v1.Container{ { Name:  "fakename", Image: "fakeimage", Ports: []v1.ContainerPort{ { Name:          "port-443", ContainerPort: 443, }, }, }, }, }, Status: v1.PodStatus{ Phase: v1.PodRunning, Conditions: []v1.PodCondition{ { Type:   v1.PodReady, Status: v1.ConditionTrue, }, }, PodIP: "10.0.0.1", PodIPs: []v1.PodIP{ { IP: "10.0.0.1", }, }, }, } // create 1001 Pods to reach endpoint max capacity that is set to 1000 allPodNames := sets.New[string]() baseIP := netutils.BigForIP(netutils.ParseIPSloppy("10.0.0.1")) for i := 0; i < 1001; i++ { pod := basePod.DeepCopy() pod.Name = fmt.Sprintf("%s-%d", basePod.Name, i) allPodNames.Insert(pod.Name) podIP := netutils.AddIPOffset(baseIP, i).String() pod.Status.PodIP = podIP pod.Status.PodIPs[0] = v1.PodIP{IP: podIP} createdPod, err := client.CoreV1().Pods(ns.Name).Create(tCtx, pod, metav1.CreateOptions{}) if err != nil { t.Fatalf("Failed to create pod %s: %v", pod.Name, err) } createdPod.Status = pod.Status _, err = client.CoreV1().Pods(ns.Name).UpdateStatus(tCtx, createdPod, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update status of pod %s: %v", pod.Name, err) } } // Create a service associated to the pod svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-service", Namespace: ns.Name, Labels: map[string]string{ "foo": "bar", }, }, Spec: v1.ServiceSpec{ Selector: map[string]string{ "foo": "bar", }, Ports: []v1.ServicePort{ {Name: "port-443", Port: 443, Protocol: "TCP", TargetPort: intstr.FromInt32(443)}, }, }, } _, err = client.CoreV1().Services(ns.Name).Create(tCtx, svc, metav1.CreateOptions{}) if err != nil { t.Fatalf("Failed to create service %s: %v", svc.Name, err) } var truncatedPodName string // poll until associated Endpoints to the previously created Service exists if err := wait.PollUntilContextTimeout(tCtx, 1*time.Second, 10*time.Second, true, func(context.Context) (bool, error) { podNames := sets.New[string]() endpoints, err := client.CoreV1().Endpoints(ns.Name).Get(tCtx, svc.Name, metav1.GetOptions{}) if err != nil { return false, nil } for _, subset := range endpoints.Subsets { for _, address := range subset.Addresses { podNames.Insert(address.TargetRef.Name) } } if podNames.Len() != 1000 { return false, nil } truncated, ok := endpoints.Annotations[v1.EndpointsOverCapacity] if !ok || truncated != "truncated" { return false, nil } // There is only 1 truncated Pod. truncatedPodName, _ = allPodNames.Difference(podNames).PopAny() return true, nil }); err != nil { t.Fatalf("endpoints not found: %v", err) } // Update the truncated Pod several times to make endpoints controller resync the service. truncatedPod, err := client.CoreV1().Pods(ns.Name).Get(tCtx, truncatedPodName, metav1.GetOptions{}) if err != nil { t.Fatalf("Failed to get pod %s: %v", truncatedPodName, err) } for i := 0; i < 10; i++ { truncatedPod.Status.Conditions[0].Status = v1.ConditionFalse truncatedPod, err = client.CoreV1().Pods(ns.Name).UpdateStatus(tCtx, truncatedPod, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update status of pod %s: %v", truncatedPod.Name, err) } truncatedPod.Status.Conditions[0].Status = v1.ConditionTrue truncatedPod, err = client.CoreV1().Pods(ns.Name).UpdateStatus(tCtx, truncatedPod, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update status of pod %s: %v", truncatedPod.Name, err) } } // delete 501 Pods for i := 500; i < 1001; i++ { podName := fmt.Sprintf("%s-%d", basePod.Name, i) err = client.CoreV1().Pods(ns.Name).Delete(tCtx, podName, metav1.DeleteOptions{}) if err != nil { t.Fatalf("error deleting test pod: %v", err) } } // poll until endpoints for deleted Pod are no longer in Endpoints. if err := wait.PollUntilContextTimeout(tCtx, 1*time.Second, 10*time.Second, true, func(context.Context) (bool, error) { endpoints, err := client.CoreV1().Endpoints(ns.Name).Get(tCtx, svc.Name, metav1.GetOptions{}) if err != nil { return false, nil } numEndpoints := 0 for _, subset := range endpoints.Subsets { numEndpoints += len(subset.Addresses) } if numEndpoints != 500 { return false, nil } truncated, ok := endpoints.Annotations[v1.EndpointsOverCapacity] if ok || truncated == "truncated" { return false, nil } return true, nil }); err != nil { t.Fatalf("error checking for no endpoints with terminating pods: %v", err) } } </ins>