Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

Related discussion:
text: } } <ins> // 1. connect to https server with http1.1 using a TCP proxy // 2. the connection has keepalive enabled so it will be reused // 3. break the TCP connection stopping the proxy // 4. close the idle connection to force creating a new connection // 5. count that there are 2 connection to the server (we didn't reuse the original connection) func TestReconnectBrokenTCP_HTTP1(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello, %s", r.Proto) })) ts.EnableHTTP2 = false ts.StartTLS() defer ts.Close() u, err := url.Parse(ts.URL) if err != nil { t.Fatalf("failed to parse URL from %q: %v", ts.URL, err) } lb := newLB(t, u.Host) defer lb.ln.Close() stopCh := make(chan struct{}) go lb.serve(stopCh) transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatal("failed to assert *http.Transport") } config := &Config{ Host:      "https://" + lb.ln.Addr().String(), Transport: utilnet.SetTransportDefaults(transport), // large timeout, otherwise the broken connection will be cleaned by it Timeout: wait.ForeverTestTimeout, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } config.TLSClientConfig.NextProtos = []string{"http/1.1"} client, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } data, err := client.Get().AbsPath("/").DoRaw(context.TODO()) if err != nil { t.Fatalf("unexpected err: %s: %v", data, err) } if string(data) != "Hello, HTTP/1.1" { t.Fatalf("unexpected response: %s", data) } // Deliberately let the LB stop proxying traffic for the current // connection. This mimics a broken TCP connection that's not properly // closed. close(stopCh) stopCh = make(chan struct{}) go lb.serve(stopCh) // Close the idle connections utilnet.CloseIdleConnectionsFor(client.Client.Transport) // If the client didn't close the idle connections, the broken connection // would still be in the connection pool, the following request would // then reuse the broken connection instead of creating a new one, and // thus would fail. data, err = client.Get().AbsPath("/").DoRaw(context.TODO()) if err != nil { t.Fatalf("unexpected err: %v", err) } if string(data) != "Hello, HTTP/1.1" { t.Fatalf("unexpected response: %s", data) } dials := atomic.LoadInt32(&lb.dials) if dials != 2 { t.Fatalf("expected %d dials, got %d", 2, dials) } } // 1. connect to https server with http1.1 using a TCP proxy making the connection to timeout // 2. the connection has keepalive enabled so it will be reused // 3. close the in-flight connection to force creating a new connection // 4. count that there are 2 connection on the LB but only one succeeds func TestReconnectBrokenTCPInFlight_HTTP1(t *testing.T) { done := make(chan struct{}) defer close(done) received := make(chan struct{}) ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/hang" { conn, _, _ := w.(http.Hijacker).Hijack() close(received) <-done conn.Close() } fmt.Fprintf(w, "Hello, %s", r.Proto) })) ts.EnableHTTP2 = false ts.StartTLS() defer ts.Close() u, err := url.Parse(ts.URL) if err != nil { t.Fatalf("failed to parse URL from %q: %v", ts.URL, err) } lb := newLB(t, u.Host) defer lb.ln.Close() stopCh := make(chan struct{}) go lb.serve(stopCh) transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatal("failed to assert *http.Transport") } config := &Config{ Host:      "https://" + lb.ln.Addr().String(), Transport: utilnet.SetTransportDefaults(transport), // Use something extraordinary large to not hit the timeout Timeout: wait.ForeverTestTimeout, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } config.TLSClientConfig.NextProtos = []string{"http/1.1"} client, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } // The request will connect, hang and eventually time out // but we can use a context to close once the test is done // we are only interested in have an inflight connection ctx, cancel := context.WithCancel(context.Background()) reqErrCh := make(chan error, 1) defer close(reqErrCh) go func() { _, err = client.Get().AbsPath("/hang").DoRaw(ctx) reqErrCh <- err }() // wait until it connect to the server select { case <-received: case <-time.After(wait.ForeverTestTimeout): t.Fatal("Test timed out waiting for first request to fail") } // Deliberately let the LB stop proxying traffic for the current // connection. This mimics a broken TCP connection that's not properly // closed. close(stopCh) stopCh = make(chan struct{}) go lb.serve(stopCh) // New request will fail if tries to reuse the connection data, err := client.Get().AbsPath("/").DoRaw(context.Background()) if err != nil { t.Fatalf("unexpected err: %v", err) } if string(data) != "Hello, HTTP/1.1" { t.Fatalf("unexpected response: %s", data) } dials := atomic.LoadInt32(&lb.dials) if dials != 2 { t.Fatalf("expected %d dials, got %d", 2, dials) } // cancel the in-flight connection cancel() select { case <-reqErrCh: if err == nil { t.Fatal("Connection succeeded but was expected to timeout") } case <-time.After(10 * time.Second): t.Fatal("Test timed out waiting for the request to fail") } } </ins> func TestRestClientTimeout(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { time.Sleep(2 * time.Second)