Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When opening port-forward requests to a pod, kubelet memory usage increases without releasing it again when the port-forward is stopped. What you expected to happen: Kubelet should cleanup the port-forward when it is closed How to reproduce it (as minimally and precisely as possible): Using the following script I can observe an increase in Kubelet memory usage on the node where the pod runs: 300 connections allocates roughly 30mb by kubelet We first noticed this on the node where Tiller (Helm) runs. The node regularly runs out of memory because of the Kubelet memory usage. see also We don't experience the helm issue on all clusters, but I can reproduce the memory leak with the above script on all clusters. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian jessie Kernel (e.g. ): Install tools: Kops 1.8.0 Others:
/sig node In my testing (v1.8.0), I observed create a TCP connection between apiserver and kubelet for each port-forward (and one between apiserver and kubectl). This apiserver<-kubelet TCP connection was not torn down when the corresponding command was killed. I have not tried against a newer apiserver/kubelet, but if someone has one handy, it's easy to just compare output on the apiserver or kubelet host before and after running several commands. Be aware that the CLOSE_WAIT connections can take minutes to expire after the kubectl<-apiserver commands exit, and this is normal and fine - we're interested in the ones that stay in state ESTABLISHED.
I'm not aware of any changes in 1.9 or 1.10 that would have fixed this. Not that we shouldn't fix this issue, but something like would be a good backstop to leaking connections like this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale behaviour is still observable for us on 1.9
On v1.11 I see plenty of and connections that hang around after many minutes.
I some logging to (among many other places) After closing a port-forward from kubectl, I see immediately the reader close on the apiserver, and the writer close on the kubelet. I do not see the other side of the connections close. Maybe 1 on each side when the pod ends. Is there a flag to enable pprof so we can dump some goroutine stacktraces? /sig apimachinery
kubelet stacktrace, the important parts
/sig api-machinery
apiserver stacktrace, the important parts
Lastly, if you restart on one side, the other side does clean up. At least if it's the kubelet. I restarted the kubelet, and apiserver shows Which is the ~600 waiting goroutines.
Based on some conversation with I will look into the approach for this issue. That issue () seems like a similar problem.

Discussion:
"net/http/httputil" "net/url" "strings" <del> "sync" </del> "time" "k8s.io/apimachinery/pkg/api/errors"

} } <del> // Proxy the connection. wg := &sync.WaitGroup{} wg.Add(2) </del> <ins> // Proxy the connection. This is bidirectional, so we need a goroutine // to copy in each direction. Once one side of the connection exits, we // exit the function which performs cleanup and in the process closes // the other half of the connection in the defer. writerComplete := make(chan struct{}) readerComplete := make(chan struct{}) </ins> go func() { var writer io.WriteCloser

if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from client to backend: %v", err) } <del> wg.Done() </del> <ins> close(writerComplete) </ins> }() go func() {

if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from backend to client: %v", err) } <del> wg.Done() </del> <ins> close(readerComplete) </ins> }() <del> wg.Wait() </del> <ins> // Wait for one half the connection to exit. Once it does the defer will // clean up the other half of the connection. select { case <-writerComplete: case <-readerComplete: } glog.V(6).Infof("Disconnecting from backend proxy %sn  Headers: %v", &location, clone.Header) </ins> return true }