Change:
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Two hunks (2x 10 "x"s) are not sent back. extends the logging. At least it will tell us whether the sender gets an error or not.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
No more insights here. Pod log doesn't show anything special.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
I analysed this flake instance: https://k8s- It looks like the pfpod container terminates, the kubelet realizes that and wants to remove the network interface from the net ns using CNI via kubenet. This fails two times (somehow TeardownNetwork is called multiple times, succeeds twice, but fails after that; compare about multiple TeardownNetwork calls). These failures interrupt the kubelet's tries to kill the pod. It succeeds 30 seconds later during reconciliation. But at that point the e2e test timed out already waiting for the pod to terminate.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
This is the real problem, not another flake: This happens roughly once per week, pretty stable.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-multizone kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gci-gke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
any updates on this issue
No, no further insights. Feel free to invest some time here with some new ideas.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-multizone kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This needs to be triaged as a release-blocker or not for 1.5
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
or Can you help us triage this as blocking or non-blocking for 1.5. It was originally marked as non-blocking for the 1.4 release.
The initial triage done by sttts@ is still valid even though we changed something in TeardownNetwork logic in 1.5 release. I don't think this should be release blocker with the same reason provided by sttts@ for 1.4 release. But instead of asking timestclair@ to take further look, I am assigning this to cc/
I marked this as release-blocker just for triage here. Thanks!
Thanks can you take a look and retriage if needed.
We have two flakes here: -- that's what and is about, i.e. Teardown logic. happened 3 times in last 8 days, much less the days before. IMO this is not a blocker. -- this seems to happen multiple times a day when a lot of e2e tests are running. This is the actual network flake. I will try to look into one of those to understand whether there is anything non-network related we can fix.
Here is a debug PR running the flaking test 100 times: None failed until now. Moreover, 200 runs locally do not fail.
Another observation: the number of returned is always a multiple of 10. The test container sleep for 100ms after each 10 . So, this might be a hint that there is some kind of race closing the connection.
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gke-multizone ci-kubernetes-e2e-gci-gke-prod-parallel ci-kubernetes-e2e-gci-gke-staging-parallel ci-kubernetes-e2e-gci-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gce-release-1.5 ci-kubernetes-e2e-gci-gce-proto ci-kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Looks like this got a lot worse on 12/3 - https://k8s- Any chance this is related to
yes, it is most probably related to . In fact, was meant to make connection closing much less aggressive on the server side. It looks like our half-open connection support using socat is far from stable. I have in flight which is blocked by broken socat. Maybe that will help eventually to fix this flake.
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce ci-kubernetes-soak-gke-test Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Bumping to P0 - this is happening pretty much all the time now. yes, it is most probably related to . - can we revert that PR then? Or have a fix really soon, this is the biggest trigger of failures now.
we marked both tests as last night: This shouldn't happen anymore after rebasing PRs.
Here is a first attempt for a real fix:
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
We will re-enable this test via This is also supposed to include a fix. Revert if the issue reappears.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
Builds: ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gci-gce-release-1.5 Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-release-1.4 ci-kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:

Discussion:
conn.Close() By("Waiting for the target pod to stop running") <del> waitErr := f.WaitForPodNoLongerRunning(pod.Name) </del> <ins> if err := f.WaitForPodNoLongerRunning(pod.Name); err != nil { framework.Failf("Pod did not stop running: %v", err) } </ins> <del> By("Retrieving logs from the target pod") </del> <ins> By("Verifying logs") </ins> logOutput, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { <del> framework.Failf("Error retrieving logs: %v", err) } if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) </del> <ins> framework.Failf("Error retrieving pod logs: %v", err) </ins> } <del> By("Verifying logs") </del> verifyLogMessage(logOutput, "Accepted client connection") verifyLogMessage(logOutput, "Expected to read 3 bytes from client, but got 0 instead") })

} By("Waiting for the target pod to stop running") <del> waitErr := f.WaitForPodNoLongerRunning(pod.Name) </del> <ins> if err := f.WaitForPodNoLongerRunning(pod.Name); err != nil { framework.Failf("Pod did not stop running: %v", err) } </ins> <del> By("Retrieving logs from the target pod") </del> <ins> By("Verifying logs") </ins> logOutput, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { <del> framework.Failf("Error retrieving logs: %v", err) </del> <ins> framework.Failf("Error retrieving pod logs: %v", err) </ins> } <del> if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) } By("Verifying logs") </del> verifyLogMessage(logOutput, "^Accepted client connection$") verifyLogMessage(logOutput, "^Received expected client data$") verifyLogMessage(logOutput, "^Done$")

if err := f.WaitForPodRunning(pod.Name); err != nil { framework.Failf("Pod did not start running: %v", err) } <ins> defer func() { logs, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { framework.Logf("Error getting pod log: %v", err) } else { framework.Logf("Pod log:n%s", logs) } }() </ins> By("Running 'kubectl port-forward'") cmd := runPortForward(f.Namespace.Name, pod.Name, 80)

} By("Waiting for the target pod to stop running") <del> waitErr := f.WaitForPodNoLongerRunning(pod.Name) </del> <ins> if err := f.WaitForPodNoLongerRunning(pod.Name); err != nil { framework.Failf("Pod did not stop running: %v", err) } </ins> <del> By("Retrieving logs from the target pod") </del> <ins> By("Verifying logs") </ins> logOutput, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { <del> framework.Failf("Error retrieving logs: %v", err) } if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) </del> <ins> framework.Failf("Error retrieving pod logs: %v", err) </ins> } <del> By("Verifying logs") </del> verifyLogMessage(logOutput, "Accepted client connection") verifyLogMessage(logOutput, "Done") })