Code change:
return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Related discussion:
text: so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.