Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

Related discussion:
text: }) } } <ins> func TestReuseRequest(t *testing.T) { var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(r.RemoteAddr)) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() ctx, cancel := context.WithCancel(context.Background()) defer cancel() c := testRESTClient(t, ts) req1, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } req2, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } if string(req1) != string(req2) { t.Fatalf("Expected %v to be equal to %v", string(req1), string(req2)) } }) } } func TestHTTP1DoNotReuseRequestAfterTimeout(t *testing.T) { var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { done := make(chan struct{}) ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Logf("TEST Connected from %v on %vn", r.RemoteAddr, r.URL.Path) if r.URL.Path == "/hang" { t.Logf("TEST hanging %vn", r.RemoteAddr) <-done } w.Write([]byte(r.RemoteAddr)) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() // close hanging connection before shutting down the http server defer close(done) ctx, cancel := context.WithCancel(context.Background()) defer cancel() transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatalf("failed to assert *http.Transport") } config := &Config{ Host:      ts.URL, Transport: utilnet.SetTransportDefaults(transport), Timeout:   100 * time.Millisecond, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } if !tt.enableHTTP2 { config.TLSClientConfig.NextProtos = []string{"http/1.1"} } c, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } req1, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } _, err = c.Verb("GET"). Prefix("/hang"). DoRaw(ctx) if err == nil { t.Fatalf("Expected error") } req2, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } // http1 doesn't reuse the connection after it times if tt.enableHTTP2 != (string(req1) == string(req2)) { if tt.enableHTTP2 { t.Fatalf("Expected %v to be the same as %v", string(req1), string(req2)) } else { t.Fatalf("Expected %v to be different to %v", string(req1), string(req2)) } } }) } } func TestTransportConcurrency(t *testing.T) { const numReqs = 10 var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Logf("Connected from %v %v", r.RemoteAddr, r.URL) fmt.Fprintf(w, "%v", r.FormValue("echo")) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() var wg sync.WaitGroup wg.Add(numReqs) c := testRESTClient(t, ts) reqs := make(chan string) defer close(reqs) for i := 0; i < 4; i++ { go func() { for req := range reqs { res, err := c.Get().Param("echo", req).DoRaw(context.Background()) if err != nil { t.Errorf("error on req %s: %v", req, err) wg.Done() continue } if string(res) != req { t.Errorf("body of req %s = %q; want %q", req, res, req) } wg.Done() } }() } for i := 0; i < numReqs; i++ { reqs <- fmt.Sprintf("request-%d", i) } wg.Wait() }) } } </ins>