Change:
Hi, We have a device (router in this case) that sends long living stream of udp packets to our kubernetes cluster. We then create a service and pods that expose a node port to receive the incoming udp packets. However, packets were not reach our containers. Further digging and found that because UDP packets already stream into our host (therefore conntrack entry already created) BEFORE our service is created (iptable rules ), the iptable rule didn't take effect. Since our UDP stream is long living, the conntrack entry never get timeout neither. To workaround the issue, we have to stop the UDP stream for sometime after the service is created so that the conntrack entry times out. If we use "conntrack -D" to clear all entries after service is deployed also workaround the issue. Though it is the behavior of the linux iptable that new rules doesn't affect already existing connection, we are trying to see if kubernetes proxy have any config that can clear the related conntrack entries when a new service is deployed?
I am interested in your problem. Can you give some more details how did you add iptables rules before service started. Did you use the PREROUTING chain to forward port to pod ?
OK I was able to reproduce the problem
I think the issue is more problematic and complex. It is explained here:
Yes I think it is the same conntrack issue as in the one reported in docker issue. But I also think that the kubernetes might have a better control over it, since it knows the service up/down. I think k8s already deleting the conntrack entries when removing services. Just wondering if the same can apply to adding services?
Yes you are right. Kubernetes as a Docker manager should take care of this. Deleting the conntrack entries is done in very simple way: Just call the application
But there is another problem because I removed conntrack entries and then I created service and still have problem. I had to clear conntrack entries after I started my service. I will try add this to source code and then try it.
In code it looks much better. I assume you used for your service to use some particular port. The fix is in place when node port is binded. Do you think there is any other scenario where conntrack entries should be cleared? When I create PR I will let you know
The PR created: If you want verify it you can easily do it using local cluster Then clone kube-deploy project to create local k8s cluster with image you already created with fixes
Thanks for the fix. This is what we did manually to workaround so it should be good enough for us. It would be even better if you can clear the conntrack entry only for the affected entry, such as destination port, to minimizes the impact. Wondering if that possible? Will try out as well. Thanks.
I wanted do it like you mentioned but unfortunately conntract command doesn't give that possibilities. In parameters you can only pass the source and destination IP addresses.
Sorry didn't get back for a while. I tried the conntrack tool and seems like you can use --dport option: conntrack -D -s 172.27.209.165 -p udp --dport 53 udp 17 63 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 udp 17 33 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 conntrack v1.4.3 (conntrack-tools): 2 flow entries have been deleted.
Thanks for info I will try it
I've updated my PR for flag It works well
It's not clear to me why there would be a conntrack rule for your service port before we add any rules?
We have a router, which configured to export statistics using UDP to our host. That UDP stream is long live. If our host restarted, or we re-deploy our cluster (mainly during development), then udp packets received by host before K8S is provisioned.
But why do you have conntrack rules - simply receiving packets will not cause conntrack to create records.. , dachulin <:

Discussion:
glog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err) continue } <ins> if lp.protocol == "udp" { proxier.clearUdpConntrackForPort(lp.port) } </ins> replacementPortsMap[lp] = socket } // We're holding the port, so it's OK to install iptables rules.

proxier.portsMap = replacementPortsMap } <ins> // Clear UDP conntrack for port or all conntrack entries when port equal zero. // When a packet arrives, it will not go through NAT table again, because it is not "the first" packet. // The solution is clearing the conntrack. Known issus: // https://github.com/docker/docker/issues/8795 // https://github.com/kubernetes/kubernetes/issues/31983 func (proxier *Proxier) clearUdpConntrackForPort(port int) { var err error = nil glog.V(2).Infof("Deleting conntrack entries for udp connections") if port > 0 { err = proxier.execConntrackTool("-D", "-p", "udp", "--dport", strconv.Itoa(port)) if err != nil && !strings.Contains(err.Error(), noConnectionToDelete) { glog.Errorf("conntrack return with error: %v", err) } } else { glog.Errorf("Wrong port number. The port number must be greater than zero") } } </ins> // Join all words with spaces, terminate with newline and write to buf. func writeLine(buf *bytes.Buffer, words ...string) { buf.WriteString(strings.Join(words, " ") + "n")