Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: After upgrading to kube-proxy , the ClusterIP will be disconnected when restarting kube-proxy. When a service uses ClusterIP to connect to other services, it will be affected. As shown in the figure, during the kube-proxy restart, the response time is obviously slower. ! When back to kube-proxy , no this problem. this is the log, attention the time, kube-proxy restart at , see the picture above, response time at the time is normal. when use kube-proxy , ClusterIP of will also be disconnected, name resolve will failed ! What you expected to happen: ClusterIP won't lost connection when kube-proxy restart How to reproduce it (as minimally and precisely as possible): Use kube-proxy Anything else we need to know?: Add flag , same result ! Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10 Install tools: manual Others:
/area ipvs
Reiterating the mentions to trigger a notification: <detailsIn response to : /area ipvs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
perhaps it was introduced by graceful termination
I'll have a look this week (probably not today) It could be caused by the VS being cleaned up on startup (not likely) or the clusterIP being unbound and rebound
Is there any progress on this issue?
I've started looking into this Just to understand well: you notice an increased latency for new connections to a clusterIP during the restart phase? Things are fine during the deletion period ? (and while kube-proxy is not running?)
It seems you mean that for a short while the clusterIP is not even working (given what you mentioned on coredns). Is this correct?
Yes, seems ipvs rule will be recreated when restart kube-proxy
I think I have reproduced this (using wrk2 and deleting the kube-proxy process while it's running) I wonder if it is related to this part of the code that is run at start-up: It will delete all RS that have no connections and set the weight of the other to 0, which means we won't be able to establish connections until syncProxyRules runs I don't know the reason for doing this at initialization. / you probably have more context. I can still create a build to validate this hypothesis
: I tested my idea (not deleting RS at start-up) and initial tests look promising: max latency does not change during a restart of the kube-proxy process (in my very simple wrk test) VS and RS reconciliation still work fine without the deletion (very limited testing there: I deleted a service and scaled down a deployment backing another while kube-proxy was not running and at restart everything was immediately resynced) Here is the diff against the branch: If you can try on a test cluster, I have a public test image here: Again, I don't know the reason for deleting all RS at start-up in the first place, so be careful (and let me know of you notice anything weird).
I had compiled kube-proxy use the new code in , and tested with , When restart kube-proxy, all DNS queries have succeeded while original kube-proxy will have some failed query. Thanks!
this looks really promising I'm creating a PR. If there is any doubt on the impacts we can discuss there
Turns out the iptable proxier has similar issues (for different reasons):

Discussion:
// Run start a goroutine to try to delete rs in the graceful delete rsList with an interval 1 minute func (m *GracefulTerminationManager) Run() { <del> // before start, add leftover in delete rs to graceful delete rsList vss, err := m.ipvs.GetVirtualServers() if err != nil { klog.Errorf("IPVS graceful delete manager failed to get IPVS virtualserver") } for _, vs := range vss { rss, err := m.ipvs.GetRealServers(vs) if err != nil { klog.Errorf("IPVS graceful delete manager failed to get %v realserver", vs) continue } for _, rs := range rss { m.GracefulDeleteRS(vs, rs) } } </del> go wait.Until(m.tryDeleteRs, rsCheckDeleteInterval, wait.NeverStop) }