Code change:
What happened: On pods with a lot of DNS traffic, many DNS queries to the DNS ClusterIP fail during a rolling update of the DNS pods (in our case, we use coredns). What you expected to happen: Almost all DNS queries should succeed during a rolling update. How to reproduce it (as minimally and precisely as possible): Create a coredns (this would probably be the same with kube-dns) service backed by 2 pods Access this service by creating a lot of DNS queries, for example: Wait for the IPVS connection table to contain a lot of DNS connections (a few minutes) Delete a DNS pod A significant number of queries will timeout. After 5mn (depending in the IPVS timeout configuration) everything will work ok again. Anything else we need to know?: This comes from the current logic in the graceful termination code: The current logic is to delete a RS if the number of active connections is 0. This makes sense for TCP but for UDP the number of active connections is always 0. This is an issue for DNS queries because the RS will be deleted but the IPVS connection will remain until it expires (5mn by default) and if there are a lot of DNS queries, the port will be reused and queries blackholed. In addition, will not always work here because IPVS only removes connections when a packet is processed (in the case of DNS once the client receives an answer, no more packets are sent). Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign
I think there is also an issue with the way we clear the conntrack for UDP We call in : If I understand correctly this will clear conntrack entries matching and before graceful termination is done. I think we should only remove the conntrack entries when we delete the real server. In addition, we only clear the conntrack entries for <clusterIP, I think we should also clean them for Node IPs (Nodeports) and External IPs. What do you think?
I think we still need to keep this issue open even though we have merged - I assume they are different issues.
yes I think the PR is a great improvement but we probably need a few other small fixes (like the conntrack thing I mentioned)
The fix from the PR is not perfect yet for DNS traffic when qps is high (but it's definitely an improvement for UDP traffic which didn't support any kind of graceful termination). For example: Will generate 5000qps and ~500 new entries in the IPVS table per second. With a UDP timeout of 30s this leads to entries maximum. If we have 2 pods, when we delete one we have to wait for the ~7500 entries to expire, given the timeout (30s) many ports will be reused (which resets the timeout) and when the coredns pod finishes its lameduck period, we lose queries. 5000 qps against 2 coredns pods is high and unlikely but to decrease the probability of DNS queries failing, we can either: decrease the UDP timeout (in this example if set to 10s, we dont lose a single packet) increase the lameduck period (I tested with UDP timeout 30s and lameduck 180s and it worked great) (of course with more pods the impact is less important: what matters is the number of ports assigned to Terminating pods) I think being able to control the UDP timeout is very important, the default (300s) is very high (see: ) (I forked dnstrace to add the parameter: , I'll try and get it merged in the main repository)
can you please help confirm our issue? in our system, our dns query is quite often, like 1 query per 10 seconds. in this case, we found that if we restart coredns, the old IPVS rules are never deleted on some of our nodes, which caused a lot of failed dns queries. is this because the graceful termination mechanism introduced here? In order to delete the RS and remove the ipvs rules, we have to wait the DNS query expire, but before the old dns query expires, new dns query came in, so grace termination will never see and old ipvs rules are never cleaned up, causing dns query issue. if this is the case, I would suggest to introduce a force termination mechanism
: it depends how you query DNS: if you have a persistent resolution socket, the IPVS backend may never expire but in that case your resolver should timeout and reconnect which will use an active backend. If you use a standard resolver (glibc for instance) each query will get a new connection so you will reach an active coredns unless you query frequently enough so you end up reusing a same source port associated with a removed backend (but below 100 queries/second this is very unlikely). I have ideas to address both use cases: configure more aggressive timeouts (in particular for UDP) and make them configurable (see issue: ) forcefully remove real servers (coredns in your case) that are still tracked by IPVS after X minutes (today we only delete them when they have 0 active and 0 inactive connections which may never happen depending on your workload) may have ideas on this topic too
DNS resolution through kube-proxy is tricky for several reasons regardless of the proxy mode (for iptables for instance, many people have conntrack issues when doing a lot of queries). In our case, we are experimenting with node local coredns caches (you can also have a look at which is an implementation of this idea)
isn't 100 queries/second quite common for a site? QPS can be thousands nowadays. we did not do special configuration for our dns so it should be using glibc. I think what you mentioned about reusing the same source port should be our issue here. by checking the output of ipvsadmin, it seems the source port is reused for different udp queries <img width="517" alt="" src="https://user-"do you have any other suggestions on this issue?
btw, forcefully remove real servers is what I want here.
I meant 100 qps for a single client (which can definitely happen though) Yes it is very likely that you have the issue I mentioned Until we implement the solution I mentioned natively, you can try changing the ipvs udp timeout on your instances. The default are: You can change udp timeout to 30s (or even lower depending on your connection rate): (0 here means do not touch tcp/tcpfin)

Related discussion:
text: uniqueRS := rs.String() if _, ok := q.list[uniqueRS]; ok { <del> return false </del> <ins> delete(q.list, uniqueRS) return true </ins> } <del> delete(q.list, uniqueRS) return true </del> <ins> return false </ins> } func (q *graceTerminateRSList) flushList(handler func(rsToDelete *listItem) (bool, error)) bool {