Code change:
framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

Related discussion:
text: I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.