Code change:
return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Related discussion:
text: Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.