Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-e2e-kops-aws Which test(s) are failing: [sig-network] Network should set TCP CLOSEWAIT timeout Test (that's what the name appears as, I don't have any more info) Since when has it been failing: 11-28-2018 Testgrid link: Reason for failure: [sig-network] Network should set TCP CLOSEWAIT timeout Test Anything else we need to know: Filing this as a flake as subsequent test executions succeeded. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test /kind flake
/sig network
Checking at the logs we can see that the entry disappeared from the conntrack table It couldn' timeout because there is only a 1-second gap between both reads, most probably something closed the connection, but it's hard to know from the log files :thinking:
If this issue has been triaged, please comment . Meta: /athenabot mark-triage-reminder I am a bot run by ‍
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
seen in
/reopen /assign Looking at one job seems that the pod that is acting as a server is restarted continuously, I need to check if it happens in more jobs, that can be a possible explanation of the flakiness, but is too soon to take conclusions
Reopened this issue. <detailsIn response to : /reopen /assign </details
/open
/reopen it seems it failed yesterday :rolleyes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
just hit it now seems it too fast checking for the pod to be running
/reopen Until this is clean :)
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
seems there is progress, there was no failure during more than one day in the jobs , however, the jobs there are still registering failures. Is it possible that the are running with older versions?
They all pull from Just to use the latest failure listed on triage as an example: v1.19.0-alpha.2.8+ Looking at the commits in (eg: ) I see that it landed before v1.19.0-alpha.2, so your fix is in there
ok, let it run a few days more and see how it evolves ...
I'm going to close this because there are no errors on the jobs in the last 5 days, and only one error in the that I can't explain, because theoretically, the loop to find the close_wait connection has to run every second during 30 seconds and, in that failure, it only runs 3 times. There are other errors of this test with the output , but those jobs have a considerable amount of test failed (100), so I'm concerned if that can be something environmental causing these failures. /close feel free to reopen
Closing this issue. <detailsIn response to : I'm going to close this because there are no errors on the jobs in the last 5 days, and only one error in the that I can't explain, because theoretically, the loop to find the close_wait connection has to run every second during 30 seconds and, in that failure, it only runs 3 times. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" </del> "k8s.io/kubernetes/test/images/agnhost/net/nat" imageutils "k8s.io/kubernetes/test/utils/image"

var _ = SIGDescribe("Network", func() { const ( <del> testDaemonHTTPPort    = 11301 testDaemonTCPPort     = 11302 timeoutSeconds        = 10 postFinTimeoutSeconds = 5 </del> <ins> testDaemonHTTPPort     = 11301 testDaemonTCPPort      = 11302 deadlineTimeoutSeconds = 10 postFinTimeoutSeconds  = 30 </ins> ) fr := framework.NewDefaultFramework("network")

zero := int64(0) <ins> // Create a pod to check the conntrack entries on the host node // It mounts the host /proc/net folder to be able to access // the nf_conntrack file with the host conntrack entries privileged := true hostExecPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "e2e-net-exec", Namespace: fr.Namespace.Name, Labels:    map[string]string{"app": "e2e-net-exec"}, }, Spec: v1.PodSpec{ HostNetwork: true, NodeName:    clientNodeInfo.name, Containers: []v1.Container{ { Name:            "e2e-net-exec", Image:           kubeProxyE2eImage, ImagePullPolicy: "Always", Args:            []string{"pause"}, VolumeMounts: []v1.VolumeMount{ { Name:      "proc-net", MountPath: "/rootfs/proc/net", ReadOnly:  true, }, }, SecurityContext: &v1.SecurityContext{ Privileged: &privileged, }, }, }, Volumes: []v1.Volume{ { Name: "proc-net", VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: "/proc/net", }, }, }, }, TerminationGracePeriodSeconds: &zero, }, } fr.PodClient().CreateSync(hostExecPod) </ins> // Some distributions (Ubuntu 16.04 etc.) don't support the proc file. <del> _, err = e2essh.IssueSSHCommandWithResult( "ls /proc/net/nf_conntrack", framework.TestContext.Provider, clientNodeInfo.node) </del> <ins> _, err = framework.RunHostCmd(fr.Namespace.Name, "e2e-net-exec", "ls /rootfs/proc/net/nf_conntrack") </ins> if err != nil && strings.Contains(err.Error(), "No such file or directory") { <del> framework.Skipf("The node %s does not support /proc/net/nf_conntrack", clientNodeInfo.name) </del> <ins> framework.Skipf("The node %s does not support /proc/net/nf_conntrack", clientNodeInfo.name) </ins> } framework.ExpectNoError(err) <ins> // Create the client and server pods </ins> clientPodSpec := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "e2e-net-client",

Image:           kubeProxyE2eImage, ImagePullPolicy: "Always", Args: []string{ <del> "net", "--serve", fmt.Sprintf("0.0.0.0:%d", testDaemonHTTPPort), </del> <ins> "net", "--serve", fmt.Sprintf(":%d", testDaemonHTTPPort), </ins> }, }, },

"net", "--runner", "nat-closewait-server", "--options", <del> fmt.Sprintf(`{"LocalAddr":"0.0.0.0:%v", "PostFindTimeoutSeconds":%v}`, </del> <ins> fmt.Sprintf(`{"LocalAddr":":%v", "PostFinTimeoutSeconds":%v}`, </ins> testDaemonTCPPort, postFinTimeoutSeconds), },

options := nat.CloseWaitClientOptions{ RemoteAddr: fmt.Sprintf("%v:%v", serverNodeInfo.nodeIP, testDaemonTCPPort), <del> TimeoutSeconds:        timeoutSeconds, PostFinTimeoutSeconds: 0, </del> <ins> TimeoutSeconds:        deadlineTimeoutSeconds, PostFinTimeoutSeconds: postFinTimeoutSeconds, </ins> LeakConnection:        true, }

`'%v' 2>/dev/null`, testDaemonHTTPPort, string(jsonBytes)) <del> framework.RunHostCmdOrDie(fr.Namespace.Name, "e2e-net-client", cmd) </del> <ins> // Run the closewait command in a subroutine so it keeps waiting during postFinTimeoutSeconds // otherwise the pod is deleted and the connection is closed loosing the conntrack entry go func() { framework.RunHostCmdOrDie(fr.Namespace.Name, "e2e-net-client", cmd) }() </ins> <-time.After(time.Duration(1) * time.Second) ginkgo.By("Checking /proc/net/nf_conntrack for the timeout") <del> // If test flakes occur here, then this check should be performed // in a loop as there may be a race with the client connecting. e2essh.IssueSSHCommandWithResult( fmt.Sprintf("sudo cat /proc/net/nf_conntrack | grep 'dport=%v'", testDaemonTCPPort), framework.TestContext.Provider, clientNodeInfo.node) // Timeout in seconds is available as the fifth column from // /proc/net/nf_conntrack. result, err := e2essh.IssueSSHCommandWithResult( fmt.Sprintf( "sudo cat /proc/net/nf_conntrack "+ "| grep 'CLOSE_WAIT.*dst=%v.*dport=%v' "+ "| tail -n 1"+ "| awk '{print $5}' ", serverNodeInfo.nodeIP, testDaemonTCPPort), framework.TestContext.Provider, clientNodeInfo.node) framework.ExpectNoError(err) timeoutSeconds, err := strconv.Atoi(strings.TrimSpace(result.Stdout)) framework.ExpectNoError(err) </del> // These must be synchronized from the default values set in // pkg/apis/../defaults.go ConntrackTCPCloseWaitTimeout. The // current defaults are hidden in the initialization code. const epsilonSeconds = 60 const expectedTimeoutSeconds = 60 * 60 <del> framework.Logf("conntrack entry timeout was: %v, expected: %v", timeoutSeconds, expectedTimeoutSeconds) gomega.Expect(math.Abs(float64(timeoutSeconds - expectedTimeoutSeconds))).Should( gomega.BeNumerically("<", (epsilonSeconds))) </del> <ins> // Obtain the corresponding conntrack entry on the host checking // the nf_conntrack file from the pod e2e-net-exec. // It retries in a loop if the entry is not found. cmd = fmt.Sprintf("cat /rootfs/proc/net/nf_conntrack "+ "| grep -m 1 'CLOSE_WAIT.*dst=%v.*dport=%v' ", serverNodeInfo.nodeIP, testDaemonTCPPort) if err := wait.PollImmediate(5*time.Second, 30*time.Second, func() (bool, error) { result, err := framework.RunHostCmd(fr.Namespace.Name, "e2e-net-exec", cmd) // retry if we can't obtain the conntrack entry if err != nil { framework.Logf("failed to obtain conntrack entry: %v %v", result, err) return false, nil } framework.Logf("conntrack entry for node %v and port %v:  %v", serverNodeInfo.nodeIP, testDaemonTCPPort, result) // Timeout in seconds is available as the fifth column of // the matched entry in /proc/net/nf_conntrack. line := strings.Fields(result) if len(line) < 5 { return false, fmt.Errorf("conntrack entry does not have a timeout field: %v", line) } timeoutSeconds, err := strconv.Atoi(line[4]) if err != nil { return false, fmt.Errorf("failed to convert matched timeout %s to integer: %v", line[4], err) } if math.Abs(float64(timeoutSeconds-expectedTimeoutSeconds)) < epsilonSeconds { return true, nil } return false, fmt.Errorf("wrong TCP CLOSE_WAIT timeout: %v expected: %v", timeoutSeconds, expectedTimeoutSeconds) }); err != nil { framework.Failf("no conntrack entry for port %d on node %s", testDaemonTCPPort, serverNodeInfo.nodeIP) } </ins> }) // Regression test for #74839, where: