Code change:
return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Related discussion:
text: 3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.