Code change:
What happened: We're running pod behind and sending logs via UDP to it. It is k8s v1.13.3 with ipvs services. When pod gets restarted logs getting blackholed. Simplified example on a node: So, old pod had IP 192.168.129.116, new pod - 192.168.128.193. Weight correctly forwards all new traffic to new pod, but there are some which are still sending traffic to non-existing IP: Minimal value for UDP connection expiration is 1s: But logs are being sent at much faster rates, so these broken connections are never expire. Issue introduced by Looks like this supposed to keep old pod running, while there are still connections to it But is there a reason to still keep these realServer entries, while old pod already cleaned up and no such IP exist? What you expected to happen: would cleanup all non-existing realServers from ipvs entries. How to reproduce it (as minimally and precisely as possible): Repro steps are same as for host-port issue: Anything else we need to know?: sysctls are default: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on-premise OS (e.g: ): Debian 9 Kernel (e.g. ): 4.9.0-8-amd64 Install tools: Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by ‍
cc
We observe the same behavior in k8s v1.12.7 and it hits us hard. In our case Nginx in a pod uses coredns service IP as DNS resolver address and every time coredns pods get new addresses DNS queries are blackholed. Query rate is higher than ipvs UDP connection timeout (30s in our case), so the connection to a non-existent ip address stays forever. Kube-proxy keeps non-existent real server entry in ipvs service and conntrack entry is never getting removed either. Kube-proxy logs: ipvsadm -ln:
I've been putting some thoughts on the issue (we actually have the same one). Currently, we handle graceful termination the same way for udp as we do for tcp. However on heavily loaded UDP services, (DNS for instance) udp ports are reused quickly and old backends are not deleted fast enough. There are two possible solutions: use more aggressive timeouts (30s instead of the default 300s). Not perfect but better immediately delete real servers and associated connections when a backend is not ready anymore (very good for DNS queries but this means no graceful termination for udp) / what do you think? I'm not sure what the exact behavior with iptables I can create a detailed issue with options to have a wider discussion. Or even directly send an email to the sig-network group
I like the idea. BTW, iptables also does not support graceful termination for UDP connections.
this is very easy to implement: change the test here: make sure is set (which we do : )
If we agree this is the way to go, I can implement it (if iptables works this way this is the best solution, I agree) We could still tweak timeouts for TCP (defaults are very long: 900s, which can lead to complete blackholing for 15mn on a crash of a target because the connection is never closed)
I agree.
/area ipvs
If you have a test environment I have a test image with the fix discussed just above: Commit here: This is not perfect, but I believe it is much better. What happens is the following: udp target pod becomes NotReady IP is removed from RealServers immediately Because of any new packet reusing the port will generate a write error and the connection will be removed from the ipvs connection table (if the application retries you are fine) I think in addition to that we should set much more aggressive UDP timeouts (5s? instead of the default: 300s) and make it configurable with a flag for specific use-case (I haven't seen a use case for long lived udp "connections" with very low amount of traffic)
Sorry, i was unable to quick test this on local dev cluster. Tried to replace image with but it seems it does not work with k8s 1.13. It then generates a lot of the same messages, and no any ipvs VIPs are created. I've also tried on fresh new cluster with the same result. Then tried but was unable to even start the cluster (with modified config from our 1.13) Can you please cherry-pick this to 1.13 or 1.14 or am I doing something wrong?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
We tested this patch to kubernetes v1.12.7 and it works well in production. BTW, I don't know much about SCTP. But question is do we need to threat it as UDP or TCP in this case? But it seems more close to UDP.
Thanks for testing! The PR is limiting graceful termination to TCP which seems the safest. I don't know SCTP but making it behave like TCP would be an easy change.
This feels like a fundamental problem w/ endpoints... ideally the proxier would remove the realserver when the pod is nonexistent rather than relying on arbitrary timeouts. Otherwise, there is no way to support graceful termination with UDP based protocols. Not a big deal for most UDP things, but I imagine e.g. QUIC users would be affected. Immediately terminating udp connections is in line with the iptables behavior, which seems to be okay for most people, so the change seems reasonable until there is a better way to determine pod state.
The PR that was just merged will improve things a lot for UDP but I completely agree with you on the fundamental problem with endpoints. I think we should have 3 different states: Ready (accepting traffic), in which case we should route traffic to the endpoint NotReady (terminating for instance), in which case we should stop routing new traffic to the endpoint but should keep existing connections "Gone" (no in the endpoint object anymore), in which case we should remove the endpoint altogether Today the endpoint object is too simple to handle these differences. I hope the new EndpointCondition status in the EndpointSlice API will help address this problem. More than happy to integrate this in IPVS when it is released (cc / )

Related discussion:
text: } for _, rs := range rss { if rsToDelete.RealServer.Equal(rs) { <del> // Delete RS with no connections // For UDP, ActiveConn is always 0 // For TCP, InactiveConn are connections not in ESTABLISHED state if rs.ActiveConn+rs.InactiveConn != 0 { </del> <ins> // For UDP traffic, no graceful termination, we immediately delete the RS //     (existing connections will be deleted on the next packet because sysctlExpireNoDestConn=1) // For other protocols, don't delete until all connections have expired) if rsToDelete.VirtualServer.Protocol != "udp" && rs.ActiveConn+rs.InactiveConn != 0 { </ins> klog.Infof("Not deleting, RS %v: %v ActiveConn, %v InactiveConn", rsToDelete.String(), rs.ActiveConn, rs.InactiveConn) return false, nil }