Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g. from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.
I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5

Discussion:
for _, epSvcPair := range connectionMap { if svcInfo, ok := proxier.serviceMap[epSvcPair.ServicePortName]; ok && svcInfo.GetProtocol() == v1.ProtocolUDP { endpointIP := utilproxy.IPPart(epSvcPair.Endpoint) <del> err := conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) </del> <ins> nodePort := svcInfo.GetNodePort() var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, v1.ProtocolUDP) } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) } </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

return info.HealthCheckNodePort } <ins> // GetNodePort is part of the ServicePort interface. func (info *BaseServiceInfo) GetNodePort() int { return info.NodePort } </ins> func (sct *ServiceChangeTracker) newBaseServiceInfo(port *v1.ServicePort, service *v1.Service) *BaseServiceInfo { onlyNodeLocalEndpoints := false if apiservice.RequestsOnlyLocalTraffic(service) {

GetProtocol() v1.Protocol // GetHealthCheckNodePort returns service health check node port if present.  If return 0, it means not present. GetHealthCheckNodePort() int <ins> // GetNodePort returns a service Node port if present. If return 0, it means not present. GetNodePort() int </ins> } // Endpoint in an interface which abstracts information about an endpoint.

} return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

t.Errorf("Expect command executed %d times, but got %d", svcCount, fexec.CommandCalls) } } <ins> func TestClearUDPConntrackForPortNAT(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeCombinedOutputAction{ func() ([]byte, error) { return []byte("1 flow entries have been deleted"), nil }, func() ([]byte, error) { return []byte(""), fmt.Errorf("conntrack v1.4.2 (conntrack-tools): 0 flow entries have been deleted") }, func() ([]byte, error) { return []byte("1 flow entries have been deleted"), nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, LookPathFunc: func(cmd string) (string, error) { return cmd, nil }, } testCases := []struct { name string port int dest string }{ { name: "IPv4 success", port: 30211, dest: "1.2.3.4", }, } svcCount := 0 for i, tc := range testCases { err := ClearEntriesForPortNAT(&fexec, tc.dest, tc.port, v1.ProtocolUDP) if err != nil { t.Errorf("%s test case: unexpected error: %v", tc.name, err) } expectCommand := fmt.Sprintf("conntrack -D -p udp --dport %d --dst-nat %s", tc.port, tc.dest) + familyParamStr(utilnet.IsIPv6String(tc.dest)) execCommand := strings.Join(fcmd.CombinedOutputLog[i], " ") if expectCommand != execCommand { t.Errorf("%s test case: Expect command: %s, but executed %s", tc.name, expectCommand, execCommand) } svcCount++ } if svcCount != fexec.CommandCalls { t.Errorf("Expect command executed %d times, but got %d", svcCount, fexec.CommandCalls) } } </ins>