Change:
While investigating a socket leak in our kubernetes-deployed application, I have found out that the kube-proxy also seems to have a serious leak of its own. In two of my three nodes, out of the ~44K connections of the kube-proxy .... ... about 65% are legit established connections caused by the aforementioned leak ... ... but ~14K are in FINWAIT2 state ... ... and ~1K are in CLOSEWAIT. Here's some debug info: Gist with service descriptions, node descriptions and iptables rules from node (we are running kubernetes 1.1 in AWS): Goroutine dump of kube-proxy at node , obtained with : at node : has a value of 60 in all my nodes. After deploying the fix for application-level leak and waiting 10 minutes, according to the number of ESTABLISHED connections is ~100 in all nodes as expected but, in the two affected nodes: The CLOSEWAIT connections only halve from 1K to 500 The FINWAIT2 ones also halve from 14K to 7K I obtained new goroutine and netstat dumps, in case it helps: Also, it may be valuable to know that the application-level leaks were caused by a websocket proxy (part of the pods) which failed to clean up connections when the client-side closed the underlying TCP socket. Both the app-mapper and the destination are accessed through kubernetes services, which means that the kube-proxy is caught up in-between: The two kube-proxy instances affected live in the same nodes where the two app-mapper pods live: and
Uhm, I have noticed that netstat somehow wasn't reporting all the sockets. Whilst lsof does: The extra entries seem to correspond to : So, I think my analysis above still applies. Here's the lsof listing in case it helps:
Can you post anything interesting from kube- Does anything appear to be spinning? Maybe try bumping the kube-proxy --v flag to 4 and reproducing , Alfonso Acosta wrote:
For record: The majority of hung goroutines (about 2/3 of them) are at go copyBytes("from backend", in, out, &amp;wg)go copyBytes("to backend", out, in, &amp;wg), both of which call down to . I note that we half-close (CloseWrite(), CloseRead()) connections in each direction. I'm not sure I recall why exactly. I could try some tweaks around here to get more info - are you willing to run a test binary for me? , Tim Hockin wrote:
The kube-proxy logs were empty (rotated by I guess) but I can try to reproduce in a separate cluster. It would be good to know, 7K are a lot plus they escape netstat (and I would bet that the same happens with most monitoring systems). Sure
I have reproduced it with our unfixed, leaky app and kube-proxy running with (in fact I can reproduce it systematically). I can confirm that, for each descriptor pair our application leaks, kube-proxy gets one socket in CLOSEWAIT and another one in FINWAIT2. Again, to give more context on our application: What's happening is that the frontend closes the client socket, but the app-mapper (which is proxying the frontend connection to the destination) doesn't close its connection pair. The CLOSEWAIT socket in kube-proxy corresponds to the frontend ("from" side) socket and the FINWAIT2 to the app-mapper ("to" side). My theory is that, when cleaning up a connection pair (due to the client closing its connection), kube-proxy expects the other end in the "to" connection to close it right away, but when it doesn't (due to the leak in this case) it stays in the FINWAIT2 state forever and leaving the "from" side in CLOSEWAIT. I think that, unlike what stated in , kube-proxy should probably timeout the connections on the "to" end after certain time. I don't see a point to wait for the "to" end if the "from" end was closed. I am not familiar with kube-proxy's internals so my reasoning may be an oversimplification or simply wrong. Here is the netstat listing from one of the affected nodes: Here are the logs of the kube-proxy running in the same node: Here is the kube-proxy go-routine dump: Here is the node description: Here are the service descriptions:
After having a look at the code, causing a couple of leaks and removing all pods to reduce the connections of the kube-proxy to a minimum, I have a pretty good idea of what's happening. The exits, but its doesn't. I need to confirm this, but my guess is that the in the copier are not enough to guarantee that in the routine returns if the remote end doesn't close the socket. You may find the full go-routing stack here:
Good news - I have a clue what is happening (I think). I am pretty sure it is the half-close logic we have - it's technically possible to close just one direction of a TCP session. If the client closes the socket, the proxy gets a signal that the client read-side closed and closes the backend's write-side. But it does not get a signal that the client's write side closed (because POSIX, that's why). If the backend responds by closing the socket (since it knows if it has more data to send or not), the proxy gets a signal that the backend read-side closed and closes the client's write side. If the backend DOES NOT close the socket, the proxy waits forever. It does not get a signal that the client's write side closed. So the easy fix is to not try the half-close stuff and just say that closed means closed, lingering data be damned. The more robust fix might be to start a timeout on the backend once the client disconnects. Here is a kube proxy binary which just closes the connection - try it out? , Alfonso Acosta wrote:
Specifically run with --v=4 and let me see logs :) , Tim Hockin wrote:
Yep, that matches my observations above. I haven't run your binary but in parallel to your investigations I have come up with a simple repro which does exactly that. For every leak created by the service I get a / pair in the kube-proxy: I can run your binary tomorrow if you still need me to, but I think this confirms the theory. I believe that the opposite timer (on the client once the backend disconnects) would also be necessary
I found some time and got the logs from a recent unmodified kube-proxy binary and the one provided while causing a leak in the leaker service: They confirm that the problem comes from the current use of and .
Thanks for the confirmation. I can't say for sure whether these are the exact same problem, but they are suspiciously similar. I'll try to work up a real patch tomorrow. , Alfonso Acosta wrote:
Patch is out for review. Thanks for the help!

Discussion:
go copyBytes("from backend", in, out, &wg) go copyBytes("to backend", out, in, &wg) wg.Wait() <del> in.Close() out.Close() </del> } func copyBytes(direction string, dest, src *net.TCPConn, wg *sync.WaitGroup) {

glog.V(4).Infof("Copying %s: %s -> %s", direction, src.RemoteAddr(), dest.RemoteAddr()) n, err := io.Copy(dest, src) if err != nil { <del> glog.Errorf("I/O error: %v", err) </del> <ins> if !isClosedError(err) { glog.Errorf("I/O error: %v", err) } </ins> } glog.V(4).Infof("Copied %d bytes %s: %s -> %s", n, direction, src.RemoteAddr(), dest.RemoteAddr()) <del> dest.CloseWrite() src.CloseRead() </del> <ins> dest.Close() src.Close() </ins> } // udpProxySocket implements proxySocket.  Close() is implemented by net.UDPConn.  When Close() is called,