Code change:
return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Related discussion:
text: ‍
just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job: