Change:
Forking from In kubernetes-soak-continuous-e2e-gke, the kubelet memory usage continued going up I checked one of the nodes , and there were over 4000 goroutines (and increasing) in kubelet. The same applies to the kubernetes-soak-continuous-e2e-gke-gci nodes. The version of kubelet on the node: Kubernetes v1.5.0-alpha.1.483+ Relevant goroutines: The cluster was deployed on 10/18. Not sure if this has already been fixed. /cc a couple of people who may have more ideas. /cc
Nope this is new.
Just for clarity, that flake was occurring since Aug 6th which was well before we enabled http2. I wonder if there is some issue inside of the client tooling. /cc
the test was flaking since Aug 6th because of some other reasons (docker leaking memory, a known issue). I didn't trace all the test failures, but I believe this is relatively recent.
Maybe this is the TLS deadlock? :) Joking.
Could you take a look at this one? mentioned that the issue only occur to gke cluster, not gce. I also quickly checked gke 1.4 soaking cluster, and didn't see the same issue. It looks like a new regression. Thanks!
This may be obvious, but: net/http.(http2serverConn).serve calls net/http.(http2serverConn).readFrames. So whatever is leaking goroutines is calling one of (from net/http): func (srv Server) Serve(l net.Listener) error func Serve(l net.Listener, handler Handler) error func (srv Server) ListenAndServe() func ListenAndServe(addr string, handler Handler) error func (srv *Server) ListenAndServeTLS(certFile, keyFile string) error func ListenAndServeTLS(addr, certFile, keyFile string, handler Handler) error
The other stack traces that call one of these functions originate at:
Are connections being left open, or only go routines? It wouldn't surprise me if this was related to the streaming functions (exec/attach/port-forward).
Looks like connections are being left open. We have around 2k goroutines that are in the middle of serve(), and around 2k open connections. They are all on localhost:. Based on default ports, this is the Kubelet Server Port
Take a look at the API server. If the streaming connections are being left open, there might be a lot of go routines here:
This is in the API server logs:
And I see this one as well. Both errors seem to occur about every hour or so. logging error output: "Error: 'EOF'nTrying to reach: 'http://10.240.0.4:4194/containers/'"
""""" logging error output: "Error: 'EOF'nTrying to reach: ''"
there are a few lines in the api server log regarding kube- 21:35:38. 5 ] Error proxying data from client to backend: read tcp 10.240.0.56:443-104.197.45.156:: read: connection reset by peer
netstat output includes 2k+ lines like these: tcp6 0 0 localhost: localhost: ESTABLISHED tcp6 0 0 localhost: localhost: ESTABLISHED tcp6 0 0 localhost: localhost:xxxxx ESTABLISHED The port is the Kubelet's default port. Looking for any logs related to the other ports shows they are all used by healthz calls: sudo grep -r : 08:06:50. 3719 ] GET /healthz: (50.055µs) 200 [[curl/7.26.0] 127.0.0.1:] 17:01:46. 3719 ] GET /healthz: (53.715µs) 200 [[curl/7.26.0] 127.0.0.1:] sudo grep -r : 01:17:18. 3719 ] GET /healthz: (30.595µs) 200 [[curl/7.26.0] 127.0.0.1:] 19:28:03. 3719 ] GET /healthz: (41.365µs) 200 [[Go-http-client/2.0] 127.0.0.1:]
Snippet from the on a gke node: On a gce node: You can clearly see that there are only healthz checks (from a monitoring script) on GCE, while there are healthz requests from Go-http-client on GKE. As mentioned in , there are a lot of open ports on the GKE cluster. All these ports are used by the Go-http-client healthz requests. These ports are used by ssh connections (which probably didn't close the http connections?). /cc
Thanks for looking into this. This is what I suspected since I know managed node team is working on node repairing logic here. could you take a look at this?
I think this may be caused by the ssh tunnel health check. Similar to kubelet, the apiserver has over 10k goroutines hanging. /cc

Discussion:
Dial: e.Tunnel.Dial, // TODO(cjcullen): Plumb real TLS options through. TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, <ins> // We don't reuse the clients, so disable the keep-alive to properly // close the connection. DisableKeepAlives: true, </ins> }) client := &http.Client{Transport: transport} resp, err := client.Get(l.healthCheckURL.String())