Code change:
t.Errorf("Expect command executed %d times, but got %d", svcCount, fexec.CommandCalls) } } <ins> func TestClearUDPConntrackForPortNAT(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeCombinedOutputAction{ func() ([]byte, error) { return []byte("1 flow entries have been deleted"), nil }, func() ([]byte, error) { return []byte(""), fmt.Errorf("conntrack v1.4.2 (conntrack-tools): 0 flow entries have been deleted") }, func() ([]byte, error) { return []byte("1 flow entries have been deleted"), nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, LookPathFunc: func(cmd string) (string, error) { return cmd, nil }, } testCases := []struct { name string port int dest string }{ { name: "IPv4 success", port: 30211, dest: "1.2.3.4", }, } svcCount := 0 for i, tc := range testCases { err := ClearEntriesForPortNAT(&fexec, tc.dest, tc.port, v1.ProtocolUDP) if err != nil { t.Errorf("%s test case: unexpected error: %v", tc.name, err) } expectCommand := fmt.Sprintf("conntrack -D -p udp --dport %d --dst-nat %s", tc.port, tc.dest) + familyParamStr(utilnet.IsIPv6String(tc.dest)) execCommand := strings.Join(fcmd.CombinedOutputLog[i], " ") if expectCommand != execCommand { t.Errorf("%s test case: Expect command: %s, but executed %s", tc.name, expectCommand, execCommand) } svcCount++ } if svcCount != fexec.CommandCalls { t.Errorf("Expect command executed %d times, but got %d", svcCount, fexec.CommandCalls) } } </ins>

Related discussion:
text: from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.