Code change:
// localAddr indicates how many TCP connection set up localAddr := make([]string, 0) <del> // init HTTP2 client client := http.Client{ Transport: &http2.Transport{ TLSClientConfig: tlsConfig, DialTLS: func(network, addr string, cfg *tls.Config) (conn net.Conn, err error) { conn, err = tls.Dial(network, addr, cfg) if err != nil { t.Fatalf("unexpect connection err: %v", err) } localAddr = append(localAddr, conn.LocalAddr().String()) return }, }, </del> <ins> client, err := newGOAWAYClient(func(network, addr string, cfg *tls.Config) (conn net.Conn, err error) { conn, err = tls.Dial(network, addr, cfg) if err != nil { t.Fatalf("unexpect connection err: %v", err) } localAddr = append(localAddr, conn.LocalAddr().String()) return }) if err != nil { t.Fatalf("failed to set-up client, err: %v", err) </ins> } <del> watchChs := make([]chan int, 0) </del> <ins> watchChs := make([]<-chan watchResponse, 0) </ins> for _, url := range tc.reqs { <del> req, err := http.NewRequest(http.MethodGet, s.URL+url, nil) if err != nil { t.Fatalf("unexpect new request error: %v", err) } resp, err := client.Do(req) </del> <ins> w, err := requestGOAWAYServer(client, s.URL, url) </ins> if err != nil { <del> t.Fatalf("failed request test server, err: %v", err) </del> <ins> t.Fatalf("failed to request server, err: %v", err) </ins> } <del> // encounter watch bytes received, does not expect to be broken if url == urlWatch || url == urlWatchWithGoaway { ch := make(chan int) watchChs = append(watchChs, ch) go func() { count := 0 for { buffer := make([]byte, 1) n, err := resp.Body.Read(buffer) if err != nil { // urlWatch will receive io.EOF, // urlWatchWithGoaway will receive http2.GoAwayError if err != io.EOF { if _, ok := err.(http2.GoAwayError); !ok { t.Errorf("watch received not EOF err: %v", err) } } ch <- count return } count += n } }() </del> <ins> if w != nil { watchChs = append(watchChs, w) </ins> } }

Related discussion:
text: What happened: We enabled feature in our large-scale cluster, and we found client(Kubelet) may got error(500 http code response) like after received GOAWAY frame. I had already re-produce this bug and open an in golang repo. So far I think this bug only occur with a small probability when client request server in high concurrency use a same httpclient instance, and it's a golang sdk bug in http client. Before I solved this issue, I suggest do NOT enable GOAWAY() feature. As it's a golang sdk bug(currently I known and doubt) in http client, revert GOAWAY feature(codes) is not necessary.
cc
/sig api-machinery
Thanks for the report.
Based on not setting the field, I think it defaults that way, so that's good. We will leave the option and filter so that when golang updates its client, we will have a simple way to turn it on and execute it.
Sure. I am trying to fix the golang http client issue and will update progress at . After golang http client issue fixed, I will add more concurrent test cases to see if there are any other bugs may occur by GOAWAY feature. BTW, we still enable GOAWAY feature in our production cluster, because there is no any evidence told us this error will influence SLO, as all most components(kubelet, controller-manager, scheduler) is controller work mode and will re-try after got error. /assign
Golang version or required, which will release at 07/01/2020.
Kubernetes v1.19(or master branch) now is using golang 1.15, and golang 1.15 has fixed the issue. And is on going. Let me add more test cases to ensure the error will not happen again.
I think we should re-open this issue because of . /re-open