Change:
On a single node kubernetes cluster, deployed using minikube: [root ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME control-plane.minikube.internal Ready control-plane,master 46h v1.22.2 10.45.253.76 <noneRed Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-305.el8.x8664 docker://20.10.6 We have two pods handling SIP traffic. One pod is sending an OPTIONS to the other on port 5060 using a service. ea1-tasftapp ClusterIP 10.97.122.198 <none5060/UDP 46h tas-sip-b2bua-tasftapp--7nvsm 1/1 Running 0 13m 10.244.0.35 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--t9b2j 1/1 Running 0 45h 10.244.0.24 control-plane.minikube.internal <none<noneConnection on conntrack looks like this: udp 17 119 src=10.244.0.24 dst=10.97.122.198 sport=5060 dport=5060 src=10.244.0.35 dst=10.244.0.24 sport=5060 dport=5060 [ASSURED] mark=0 use=1 If we delete the pod behind the ea1-tasftapp service by using kubectl delete pod, a new one is created: tas-sip-b2bua-tasftapp--7rdgm 1/1 Running 0 47s 10.244.0.36 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--t9b2j 1/1 Running 0 45h 10.244.0.24 control-plane.minikube.internal <none<noneand conntrack is updated correctly: udp 17 119 src=10.244.0.24 dst=10.97.122.198 sport=5060 dport=5060 src=10.244.0.36 dst=10.244.0.24 sport=5060 dport=5060 [ASSURED] mark=0 use=1 If we delete the pod using force, a new pod is created: tas-sip-b2bua-tasftapp--gvx9g 1/1 Running 0 22s 10.244.0.37 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--t9b2j 1/1 Running 0 45h 10.244.0.24 control-plane.minikube.internal <none<nonebut the conntrack is updated using the node ip instead of the new pods. Due to this, connectivity between the two pods doesn't work. udp 17 29 src=10.244.0.24 dst=10.97.122.198 sport=5060 dport=5060 [UNREPLIED] src=10.97.122.198 dst=10.45.253.76 sport=5060 dport=9817 mark=0 use=1 Problem is solved only by restarting the pod initiated the connection (tas-sip-b2bua-tassnf--t9b2j) or by restarting kube-proxy. Issue appears only on 1.22 kubernetes version. On version 1.21 everything is working properly. After pod is killed, a new conntrack entry should be created towards the IP of the new pod. Having udp connectivity between a pod and a service and force delete the pod where the traffic is routed through the service. No response Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-15T21:32:41Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} On premises deployment. NAME="Red Hat Enterprise Linux" VERSION="8.4 (Ootpa)" ID="rhel" IDLIKE="fedora" VERSIONID="8.4" PLATFORMID="platform:el8" PRETTYNAME="Red Hat Enterprise Linux 8.4 (Ootpa)" ANSICOLOR="0;31" CPENAME="cpe:/o:redhat:enterpriselinux:8.4:GA" HOMEURL="" DOCUMENTATIONURL="" BUGREPORTURL="" REDHATBUGZILLAPRODUCT="Red Hat Enterprise Linux 8" REDHATBUGZILLAPRODUCTVERSION=8.4 REDHATSUPPORTPRODUCT="Red Hat Enterprise Linux" REDHATSUPPORTPRODUCTVERSION="8.4" Linux control-plane.minikube.internal 4.18.0-305.el8.x8664 SMP Thu Apr 29 08:54:30 EDT 2021 x8664 x8664 x86_64 GNU/Linux <details</detailsminikube version: v1.23.2 commit: [root ~]# calicoctl version Client Version: v3.20.2 Git commit: Cluster Version: v3.20.2 Cluster Type: k8s,kdd,bgp,kubeadm
/sig network
/area kube-proxy
can you share the manifests and steps to try to reproduce it? Is not clear to me what pods belong to the service in this case The conntrack entry using the nodeIP depends on the iptables rules you have configured in the host, if you target a service that doesn't exist most probably the CNI source nat the Pod to the hostIP
Let me try to describe our setup, as reproducing our deployment would not be possible. The issue here is between the services "ea1-tassnf" and "ea1-tasftapp". Both these services have 1 pod behind them. The 'tas-sip-b2bua-tassnf' pod sends periodic heartbeats (SIP protocol, over UDP) to 'tas-sip-b2bua-tasftapp'. The heartbeats are addressed to the service name 'ea1-tasftapp'. Everything works fine until we try to simulate a failure. We try to simulate a failure by force deleting the pod 'tas-sip-b2bua-tasftapp' (). This results in a new pod for 'tas-sip-b2bua-tasftapp'. But this new pod never receives the heartbeat, while we see the pod 'tas-sip-b2bua-tasfsnf' keeps sending heartbeats, but it times out. The root cause seems to be the conntrack entries (see the explanation in the issue description). If we restart 'kube-proxy' , the heartbeats start working. Also as stated earlier, the exact same scenario works fine on kubernetes v1.21. This issue appears from v1.22.
most of the conntrack issues with UDP were fixed, I'm surprised this worked before and not now :/ ... can you run kube-proxy with verbosity 7 and attach the logs after reproduce the issue (please share more or less the time when you delete the pod and the name of the pods so I can find them easily in the logs)
I have the logs exactly from when I start our test scenario and we deleted the pod. The verbosity is 9 hope this is not a problem. Please find them attached! I attach also the same exactly test on kubernetes 1.21 (where everything works as expected). I can see some differences but to be honest, wasn't able to find what exactly is creating the issue. Since we can always replicate the issue, I can provide whatever else is needed/may help on the troubleshooting.
EDIT WRONG ~on 1.22~ ~on 1.21~ ~on 1.21 it deletes the conntrack entry for service~ ~on 1.22 doesn't detect the service stale :thinking: and call the slicecache twice but the handler once~ ~ does this ring a bell?~
do I read it correctly and the problem only happens when using --force to delete the pod, without force it works well?
actually our latest tests show otherwise, let me explain. The pod used to take time to terminate (in its handler), and this issue was observed only when we deleted the pod with , and not with graceful termination or with a large grace period (e.g. ). Now, we have corrected a bug in our application in its handler, and the application terminates very fast. With this update in the application, we observe this issue for all cases i.e. regardless of whether the pod is gracefully deleted or with force. Maybe this gives a clue? Because when the application used to take time to terminate, we dont get this issue when the pod was deleted with , but if we shorten the grace period (e.g. ) or use , this issue appears.
I don't have clear this is related to the kubernetes version, the logs are missing some information but in the 1.22 logs the logic seem correct iptables rule created stale endpoint detected entries deleted less than one second later one new endpoint to the service and stale service detected and its entries are deleted I don't know if you may have a race somewhere, in the application or with the CNI, I suggest you to try that with KIND that has less "networking overhead"
We already tried with KIND, actually this is the CNI used on the environment that my colleague is using. Behavior is exactly the same. As I said I'm always available to provide further logs from any kubernetes version (1.20,21 or 22). Right now, Behavior is the same using Calico or KIND. The scenario is working on 1.20 and 1.21 but not on 1.22.
wow, can you attach the full logs of the kube-proxy? I'd like to check the configuration options and the api calls
Thank you again for the support and effort. Find bellow all the information: Kubernetes 1.21 [root ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME control-plane.minikube.internal Ready control-plane,master 2d23h v1.21.5 10.45.253.76 <noneRed Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-305.el8.x8664 docker://20.10.6 [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} [root ~]# kubectl get svc --namespace tas -o wide --selector platform=tas-scif-sip NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR ea1-redis ClusterIP 10.109.170.31 <none6379/TCP 21s app=redis,chart=redis,environment=dev,platform=tas-scif-sip,release=ea1 ea1-tasftapp ClusterIP 10.110.133.170 <none5060/UDP 21s app=tasftapp,chart=tas-sip-b2bua,environment=dev,platform=tas-scif-sip,release=ea1 ea1-tassnf NodePort 10.96.49.152 <none5060:/UDP 21s app=snf,chart=tas-sip-b2bua,environment=dev,platform=tas-scif-sip,release=ea1 ea1-telegraf ClusterIP 10.109.214.39 <none8186/TCP,9273/TCP 21s app=telegraf,chart=metrics,environment=dev,platform=tas-scif-sip,release=ea1 [root ~]# kubectl get pods --namespace tas -o wide --selector platform=tas-scif-sip NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES metrics-telegraf--c2lq8 1/1 Running 0 35s 10.244.0.17 control-plane.minikube.internal <none<noneredis-tasredis--rssh4 1/1 Running 0 35s 10.244.0.19 control-plane.minikube.internal <none<nonetas-sip-b2bua-tasftapp--nqhxb 1/1 Running 0 35s 10.244.0.22 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--c6j6t 1/1 Running 0 35s 10.244.0.18 control-plane.minikube.internal <none<none[root ~]# conntrack -L | grep 10.110.133.170 udp 17 119 src=10.244.0.18 dst=10.110.133.170 sport=5060 dport=5060 src=10.244.0.22 dst=10.244.0.18 sport=5060 dport=5060 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 294 flow entries have been shown. After force delete: [root ~]# kubectl get pods --namespace tas -o wide --selector platform=tas-scif-sip NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES metrics-telegraf--c2lq8 1/1 Running 0 2m3s 10.244.0.17 control-plane.minikube.internal <none<noneredis-tasredis--rssh4 1/1 Running 0 2m3s 10.244.0.19 control-plane.minikube.internal <none<nonetas-sip-b2bua-tasftapp--mtsrp 1/1 Running 0 15s 10.244.0.23 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--c6j6t 1/1 Running 0 2m3s 10.244.0.18 control-plane.minikube.internal <none<none[root ~]# conntrack -L | grep 10.110.133.170 udp 17 119 src=10.244.0.18 dst=10.110.133.170 sport=5060 dport=5060 src=10.244.0.23 dst=10.244.0.18 sport=5060 dport=5060 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 315 flow entries have been shown. Kube-proxy logs with verbosity 9 Kubernets 1.22 [root ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME control-plane.minikube.internal Ready control-plane,master 4d1h v1.22.2 10.45.253.76 <noneRed Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-305.el8.x8664 docker://20.10.6 [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"" [root ~]# kubectl get svc --namespace tas -o wide --selector platform=tas-scif-sip NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR ea1-redis ClusterIP 10.100.207.51 <none6379/TCP 2m17s app=redis,chart=redis,environment=dev,platform=tas-scif-sip,release=ea1 ea1-tasftapp ClusterIP 10.102.145.128 <none5060/UDP 2m17s app=tasftapp,chart=tas-sip-b2bua,environment=dev,platform=tas-scif-sip,release=ea1 ea1-tassnf NodePort 10.107.207.247 <none5060:/UDP 2m17s app=snf,chart=tas-sip-b2bua,environment=dev,platform=tas-scif-sip,release=ea1 ea1-telegraf ClusterIP 10.109.139.216 <none8186/TCP,9273/TCP 2m17s app=telegraf,chart=metrics,environment=dev,platform=tas-scif-sip,release=ea1 [root ~]# kubectl get pods --namespace tas -o wide --selector platform=tas-scif-sip NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES metrics-telegraf--tqbb8 1/1 Running 0 2m38s 10.244.0.41 control-plane.minikube.internal <none<noneredis-tasredis--x2fs7 1/1 Running 0 2m38s 10.244.0.43 control-plane.minikube.internal <none<nonetas-sip-b2bua-tasftapp--jhdv4 1/1 Running 0 2m38s 10.244.0.42 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--sb447 1/1 Running 0 2m38s 10.244.0.40 control-plane.minikube.internal <none<none[root ~]# conntrack -L | grep 10.102.145.128 udp 17 119 src=10.244.0.40 dst=10.102.145.128 sport=5060 dport=5060 src=10.244.0.42 dst=10.244.0.40 sport=5060 dport=5060 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 284 flow entries have been shown. After force delete: [root ~]# kubectl get pods --namespace tas -o wide --selector platform=tas-scif-sip NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES metrics-telegraf--tqbb8 1/1 Running 0 5m44s 10.244.0.41 control-plane.minikube.internal <none<noneredis-tasredis--x2fs7 1/1 Running 0 5m44s 10.244.0.43 control-plane.minikube.internal <none<nonetas-sip-b2bua-tasftapp--xjwtg 1/1 Running 0 103s 10.244.0.44 control-plane.minikube.internal <none<nonetas-sip-b2bua-tassnf--sb447 1/1 Running 0 5m44s 10.244.0.40 control-plane.minikube.internal <none<none[root ~]# conntrack -L | grep 10.102.145.128 udp 17 29 src=10.244.0.40 dst=10.102.145.128 sport=5060 dport=5060 [UNREPLIED] src=10.102.145.128 dst=10.45.253.76 sport=5060 dport=7845 mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 318 flow entries have been shown. Kube-proxy logs with verbosity 9 [] () This is again from my environment, using Calico but as I said, the behavior is exactly the same with kindnet
I can't reproduce it :/ Install an UDP service with a server Get the new service IP Run another pod polling the udp service constantly In another terminal watch the conntrack entries inside the node After doing to the service pod, the conntrack entries are correct and the app keeps working
Thank you again for the effort. We are working to isolate the issue and provide what is needed in order for you to simulate it. We will come back as soon as possible with an update.
Sorry for coming back late on this. We were trying our best to isolate this problem, and provide you a way to reproduce this. We now have a way to consistently reproduce this problem without requiring all of our containers. And, I think I know what triggers this issue. Our pod had an that just did a sleep for 5 secs. The wrong iptable entry occurs only when this pause is present. As soon as I remove this , we do not see this issue anymore, no matter how many times I delete the pod (--force or gracefully). I have written 2 small Java applications, which you can deploy to simulate the problem in your k8s cluster. I have built container images for these 2 applications which you can pull into your environment - options-responder: This listens for SIP OPTIONS over UDP (port 5060), and just responds for every incoming request. options-sender: This sends SIP OPTIONS request every 1 sec to . The images are available on docker hub - I've attached 2 yaml files that will deploy these into k8s - We can check the logs from pod when it is exchanging packets successfully - Now we kill the pod - The options-sender pod soon starts getting timeouts, and this continues - I monitor the conntrack entries during the time I delete the pod - For the UNREPLIED udp entries, we still see wrong IP (node IP) on the return path and this is never cleared. I think if you deploy this into your k8s cluster, you should be able to reproduce this problem. And also figure out why adding an with a small pause introduces wrong iptable entry which is never deleted! Let me know if you face any issues in deploying this.
/assign
is still the kubernetes version making any difference or was a red herring?
Using the pods provided by my colleague, initial statement is still valid. The scenario works fine on kubernetes v1.21:
/triage accepted I can reproduce it, there is definitively something odd here
great catch, this is something worse than the udp conntrack problem
and cherry=pick to 1.22 branch

Discussion:
// Equal is part of proxy.Endpoint interface. func (info *BaseEndpointInfo) Equal(other Endpoint) bool { <del> return info.String() == other.String() && info.GetIsLocal() == other.GetIsLocal() </del> <ins> return info.String() == other.String() && info.GetIsLocal() == other.GetIsLocal() && info.IsReady() == other.IsReady() </ins> } // GetNodeName returns the NodeName for this endpoint.

// detectStaleConnections modifies <staleEndpoints> and <staleServices> with detected stale connections. <staleServiceNames> // is used to store stale udp service in order to clear udp conntrack later. func detectStaleConnections(oldEndpointsMap, newEndpointsMap EndpointsMap, staleEndpoints *[]ServiceEndpoint, staleServiceNames *[]ServicePortName) { <ins> // Detect stale endpoints: an endpoint can have stale conntrack entries if it was receiving traffic // and then goes unready or changes its IP address. </ins> for svcPortName, epList := range oldEndpointsMap { if svcPortName.Protocol != v1.ProtocolUDP { continue } for _, ep := range epList { <ins> // if the old endpoint wasn't ready is not possible to have stale entries // since there was no traffic sent to it. if !ep.IsReady() { continue } </ins> stale := true <ins> // Check if the endpoint has changed, including if it went from ready to not ready. // If it did change stale entries for the old endpoint has to be cleared. </ins> for i := range newEndpointsMap[svcPortName] { if newEndpointsMap[svcPortName][i].Equal(ep) { stale = false

} } <ins> // Detect stale services // For udp service, if its backend changes from 0 to non-0 ready endpoints. // There may exist a conntrack entry that could blackhole traffic to the service. </ins> for svcPortName, epList := range newEndpointsMap { if svcPortName.Protocol != v1.ProtocolUDP { continue } <del> // For udp service, if its backend changes from 0 to non-0. There may exist a conntrack entry that could blackhole traffic to the service. if len(epList) > 0 && len(oldEndpointsMap[svcPortName]) == 0 { </del> <ins> epReady := 0 for _, ep := range epList { if ep.IsReady() { epReady++ } } oldEpReady := 0 for _, ep := range oldEndpointsMap[svcPortName] { if ep.IsReady() { oldEpReady++ } } if epReady > 0 && oldEpReady == 0 { </ins> *staleServiceNames = append(*staleServiceNames, svcPortName) } }

return e.Endpoint == o.Endpoint && e.IsLocal == o.IsLocal && e.protocol == o.protocol && <del> e.chainName == o.chainName </del> <ins> e.chainName == o.chainName && e.Ready == o.Ready </ins> } // Returns the endpoint chain name for a given endpointsInfo.

eps.AddressType = discovery.AddressTypeIPv4 eps.Endpoints = []discovery.Endpoint{{ Addresses: []string{epIP}, <ins> Conditions: discovery.EndpointConditions{ Ready: utilpointer.Bool(false), }, }} eps.Ports = []discovery.EndpointPort{{ Name:     utilpointer.StringPtr(svcPortName.Port), Port:     utilpointer.Int32(int32(svcPort)), Protocol: &udpProtocol, }} }), ) fp.syncProxyRules() if fexec.CommandCalls != 0 { t.Fatalf("Updated UDP service with not ready endpoints must not clear UDP entries") } populateEndpointSlices(fp, makeTestEndpointSlice(svcPortName.Namespace, svcPortName.Name, 1, func(eps *discovery.EndpointSlice) { eps.AddressType = discovery.AddressTypeIPv4 eps.Endpoints = []discovery.Endpoint{{ Addresses: []string{epIP}, Conditions: discovery.EndpointConditions{ Ready: utilpointer.Bool(true), }, </ins> }} eps.Ports = []discovery.EndpointPort{{ Name:     utilpointer.StringPtr(svcPortName.Port),

} }) <ins> // Regression test for #105657 // 1. Create an UDP Service // 2. Client Pod sending traffic to the UDP service // 3. Create an UDP server associated to the Service created in 1. with an init container that sleeps for some time // The init container makes that the server pod is not ready, however, the endpoint slices are created, it is just // that the Endpoint conditions Ready is false. // If the kube-proxy conntrack logic doesn't check readiness, it will delete the conntrack entries for the UDP server // when the endpoint slice has been created, however, the iptables rules will not installed until at least one // endpoint is ready. If some traffic arrives to since kube-proxy clear the entries (see the endpoint slice) and // installs the corresponding iptables rules (the endpoint is ready), a conntrack entry will be generated blackholing // subsequent traffic. ginkgo.It("should be able to preserve UDP traffic when initial unready endpoints get ready", func() { // Create a ClusterIP service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the ClusterIP service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := e2epod.NewAgnhostPod(ns, podClient, nil, nil, nil) nodeSelection := e2epod.NodeSelection{Name: clientNodeInfo.name} e2epod.SetNodeSelection(&clientPod.Spec, nodeSelection) cmd := fmt.Sprintf(`date; for i in $(seq 1 3000); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := e2epod.NewAgnhostPod(ns, podBackend1, nil, nil, nil, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels nodeSelection = e2epod.NodeSelection{Name: serverNodeInfo.name} // Add an init container to hold the pod to be ready for 15 seconds serverPod1.Spec.InitContainers = []v1.Container{ { Name:    "init", Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"/bin/sh", "-c", "echo Pausing start. && sleep 15"}, }, } e2epod.SetNodeSelection(&serverPod1.Spec, nodeSelection) fr.PodClient().CreateSync(serverPod1) // wait until the endpoints are ready validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) // Note that the fact that Endpoints object already exists, does NOT mean // that iptables (or whatever else is used) was already programmed. // Additionally take into account that UDP conntract entries timeout is // 30 seconds by default. // Based on the above check if the pod receives the traffic. ginkgo.By("checking client pod connected to the backend on Node IP " + serverNodeInfo.nodeIP) if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend1, podClient)); err != nil { logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) framework.Failf("Failed to connect to backend pod") } }) </ins> // Regression test for #74839, where: // Packets considered INVALID by conntrack are now dropped. In particular, this fixes // a problem where spurious retransmits in a long-running TCP connection to a service