Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [sig-network] Services should have session affinity work for NodePort service [LinuxOnly] Testgrid link: Reason for failure: Dec 17 23:05:35.615: Connection to 10.0.207.245:80 timed out or not enough responses. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig testing
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I think this issue is resolved. my pipeline () passed and the issue that mentioned this one () has been merged. cc'ing Jordan just in case there is anything im missing. /cc
Until the root cause has been identified/fixed, or we see testgrid show this flake free for several days, we'll leave this open.
/assign these are four of the top-ten flakes according to if a fix is not identified shortly, we'll need to mark them as , which will move them out of the merge/release-blocking jobs
it seems that those errors are not happening when using KIND may it be a GCE environment issue?
Analyzing this occurence The tests fails timing out at 23:05:35.615 Dec 17 23:05:35.615: Connection to 10.0.207.245:80 timed out or not enough responses. The last update of the service port is at However, the endpoints are not present until 23:05:40. I1217 23:05:40. 1 ] Setting endpoints for "services-9616/affinity-clusterip-transition:" to [10.64.0.53:9376 10.64.1.114:9376] I1217 23:05:40. 1 ] sync-runner: 674.883613ms since last run, possible in 9.325116387s, scheduled in 9.325128618s During all that times it was logging That message belongs to the method func (hcs *server) SyncEndpoints(newEndpoints map[types.NamespacedName]int) error { do you have any clue why it takes so long to sync the endpoints or is just that the timeout should be raised?
Same happens in this one Dec 18 00:42:18.918: Connection to 10.0.60.18:80 timed out or not enough responses. endpoints are not ready until I1218 00:42:29. 1 ] Setting endpoints for "services-4166/affinity-clusterip-transition:" to [10.64.0.81:9376 10.64.1.71:9376 10.64.3.58:9376] I1218 00:42:29. 1 ] Setting endpoints for "services-4166/affinity-clusterip-transition:" to [10.64.1.71:9376 10.64.3.58:9376]
This is the constant that handles the timeout, however the comment says it should be enough time is it possible that there is a performance regression hidden somewhere?
The timeline shows that failures started around Dec 14th EDIT :man_facepalming: :smile: how could I forget this
That's when silent flake suppression/retry was turned off. I suspect this has been flaky for far longer than that.
Some interesting things, the apiserver is using the [::1] address. per example more entries in the log file I've opened an issue because I've observed that the apiserver mixing IP families caused retries and errors in kube-proxy
I'm starting to think that this may be a performance regression because the endpoints are not being updated in a timely manner, based on this analysis and the following entries and seems that these issues are related: and that has submited PR to fix this
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
four of the top ten flakes according to
any update on status of this? if a fix is not imminent, these tests will be marked and will not run in blocking presubmit.
Taking a look
cc:
Finding so far: This error message should be redherring. It should not slow down or block kube-proxy iptables sync.
I opened to tag as [Flaky], I know these are hard to track down but they've been in the top 10 flakes for about a month now
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
/assign
I've tried to reproduce it locally but I can't find entries for the iptables module, that AFAIK is the one that tracks the connections. I've created a deployment and a service and dumpled the iptables. After updating the service to use sessionAffinity and a timeout of 2 seconds, it seems that the iptables rules are correct: after several to the service for more than 2 seconds, it always hits the same backend. The problem is that I can't find entries in the /proc/net/xt_recent folder can it be something related to the iptables module and the switch to xtables?
Also, based on it seems that the test is not covering this scenario, it should be failing because it has to check that it switches the backend once the affinity times out
/assign
/cc
I couldn't find errors like this anymore and seems that the flakiness was reduced, maybe this problem was fixed by Analyzing one of the last occurrences it shows that the timeout is not enough In order to pass the test it should have 15 hits to the same backend in a row and during this time it was only able to get 13, there is a huge delay from the time the command is executed until the answer is received, i.e.
There is also another problem, in one of the occurrences the first 2 connections go to one backend and the other 13 goes to the another one it fails because it should have deleted the tracked hosts before retrying, otherwise is taking into account the wrong hosts to check teh affinity
ah, that was a problem before with the "affinity works at all" tests () but I had thought when I looked at this that the problem was that the affinity actually wasn't switching, not just that we weren't getting enough samples
also, suggested to me that there was actually a problem here that wasn't just "we aren't taking long enough to get 15 samples"
I think that this is another scenario we are not covering after 2 seconds we should get answers from a different backend, right?
:thinking: this is working to me from a pod ~I think that sessionAffinity only applies to pod-svc communication, that's why session affinity is not working in my example and in ~ my bad EDIT ok one mystery solved, scratch all previous theories, it seems that with a short time between requests the tracking entries never expire and it sticks forever, using 2 seconds between requests it gets requests from other backends
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍

Discussion:
// return false only in case of unexpected errors. func checkAffinity(execPod *v1.Pod, serviceIP string, servicePort int, shouldHold bool) bool { serviceIPPort := net.JoinHostPort(serviceIP, strconv.Itoa(servicePort)) <del> cmd := fmt.Sprintf(`curl -q -s --connect-timeout 2 http://%s/`, serviceIPPort) </del> <ins> curl := fmt.Sprintf(`curl -q -s --connect-timeout 2 http://%s/`, serviceIPPort) cmd := fmt.Sprintf("for i in $(seq 0 %d); do echo; %s ; done", AffinityConfirmCount, curl) </ins> timeout := AffinityTimeout if execPod == nil { timeout = LoadBalancerPollTimeout } var tracker affinityTracker <del> if pollErr := wait.PollImmediate(framework.Poll, timeout, func() (bool, error) { </del> <ins> // interval considering a maximum of 2 seconds per connection interval := 2 * AffinityConfirmCount * time.Second if pollErr := wait.PollImmediate(interval, timeout, func() (bool, error) { </ins> if execPod != nil { stdout, err := framework.RunHostCmd(execPod.Namespace, execPod.Name, cmd) if err != nil { framework.Logf("Failed to get response from %s. Retry until timeout", serviceIPPort) return false, nil } <del> tracker.recordHost(stdout) </del> <ins> hosts := strings.Split(stdout, "n") for _, host := range hosts { tracker.recordHost(strings.TrimSpace(host)) } </ins> } else { rawResponse := GetHTTPContent(serviceIP, servicePort, timeout, "") tracker.recordHost(rawResponse.String())

}) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for service with type clusterIP [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity work for service with type clusterIP [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-clusterip") svc.Spec.Type = v1.ServiceTypeClusterIP execAffinityTestForNonLBService(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity timeout work for service with type clusterIP [LinuxOnly]", func() { svc := getServeHostnameService("affinity-clusterip-timeout") svc.Spec.Type = v1.ServiceTypeClusterIP execAffinityTestForSessionAffinityTimeout(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should be able to switch session affinity for service with type clusterIP [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-clusterip-transition") svc.Spec.Type = v1.ServiceTypeClusterIP execAffinityTestForNonLBServiceWithTransition(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for NodePort service [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity work for NodePort service [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-nodeport") svc.Spec.Type = v1.ServiceTypeNodePort execAffinityTestForNonLBService(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for NodePort service [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity timeout work for NodePort service [LinuxOnly]", func() { svc := getServeHostnameService("affinity-nodeport-timeout") svc.Spec.Type = v1.ServiceTypeNodePort execAffinityTestForSessionAffinityTimeout(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should be able to switch session affinity for NodePort service [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-nodeport-transition") svc.Spec.Type = v1.ServiceTypeNodePort execAffinityTestForNonLBServiceWithTransition(f, cs, svc)

return pausePod.Status.PodIP, host } <ins> // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of // affinity test for non-load-balancer services. Session afinity will be // enabled when the service is created and a short timeout will be configured so // session affinity must change after the timeout expirese. func execAffinityTestForSessionAffinityTimeout(f *framework.Framework, cs clientset.Interface, svc *v1.Service) { ns := f.Namespace.Name numPods, servicePort, serviceName := 3, defaultServeHostnameServicePort, svc.ObjectMeta.Name ginkgo.By("creating service in namespace " + ns) serviceType := svc.Spec.Type // set an affinity timeout equal to the number of connection requests svcSessionAffinityTimeout := int32(AffinityConfirmCount) svc.Spec.SessionAffinity = v1.ServiceAffinityClientIP svc.Spec.SessionAffinityConfig = &v1.SessionAffinityConfig{ ClientIP: &v1.ClientIPConfig{TimeoutSeconds: &svcSessionAffinityTimeout}, } _, _, err := StartServeHostnameService(cs, svc, ns, numPods) framework.ExpectNoError(err, "failed to create replication controller with service in the namespace: %s", ns) defer func() { StopServeHostnameService(cs, ns, serviceName) }() jig := e2eservice.NewTestJig(cs, ns, serviceName) svc, err = jig.Client.CoreV1().Services(ns).Get(context.TODO(), serviceName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch service: %s in namespace: %s", serviceName, ns) var svcIP string if serviceType == v1.ServiceTypeNodePort { nodes, err := e2enode.GetReadySchedulableNodes(cs) framework.ExpectNoError(err) addrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) gomega.Expect(len(addrs)).To(gomega.BeNumerically(">", 0), "ginkgo.Failed to get Node internal IP") svcIP = addrs[0] servicePort = int(svc.Spec.Ports[0].NodePort) } else { svcIP = svc.Spec.ClusterIP } execPod := e2epod.CreateExecPodOrFail(cs, ns, "execpod-affinity", nil) defer func() { framework.Logf("Cleaning up the exec pod") err := cs.CoreV1().Pods(ns).Delete(context.TODO(), execPod.Name, nil) framework.ExpectNoError(err, "failed to delete pod: %s in namespace: %s", execPod.Name, ns) }() err = jig.CheckServiceReachability(svc, execPod) framework.ExpectNoError(err) // the service should be sticky until the timeout expires framework.ExpectEqual(checkAffinity(execPod, svcIP, servicePort, true), true) // but it should return different hostnames after the timeout expires // try several times to avoid the probability that we hit the same pod twice hosts := sets.NewString() cmd := fmt.Sprintf(`curl -q -s --connect-timeout 2 http://%s/`, net.JoinHostPort(svcIP, strconv.Itoa(servicePort))) for i := 0; i < 10; i++ { hostname, err := framework.RunHostCmd(execPod.Namespace, execPod.Name, cmd) if err == nil { hosts.Insert(hostname) if hosts.Len() > 1 { return } time.Sleep(time.Duration(svcSessionAffinityTimeout) * time.Second) } } framework.Fail("Session is sticky after reaching the timeout") } </ins> func execAffinityTestForNonLBServiceWithTransition(f *framework.Framework, cs clientset.Interface, svc *v1.Service) { execAffinityTestForNonLBServiceWithOptionalTransition(f, cs, svc, true) }