Change:
Before merged, TestClientReceivedGOAWAY failed with data race on . However, as far as I know, it's not the root cause of TestClientReceivedGOAWAY flake, because dial a new connection only when client send a new request, and in this case, requests are sent one by one. In other words, if data race on , then the additional unexpected connection set up, which means the test case or the configuration has problem. After merged, TestClientReceivedGOAWAY still failed in many jobs, such as , , which may be a good proof of my suspect. And I got a failed job with data race issue that read at (main testing goroutine) and write at , which told us the http client try to set-up a new connection even when we had already finished all requests(may caused by goroutine leak/block in http client). I'm still debug the root cause of TestClientReceivedGOAWAY flake, for now, I doubt that I used an incorrect http2 server/client configuration( or SETTINGSMAXCONCURRENT_STREAMS limited), or the dial function ignored context.Context which will cause more than one goroutine dial the connection when the first canceled(not found any where will cancel the requests). I still can not reproduce this failed case and it's hard to find the root cause. Couple of days may be needed to find the root cause, and I will post a PR to skip TestClientReceivedGOAWAY before I solved the problem. Sorry for any inconvenient.
/assign /sig testing
cc
Thank you for working on this!! How often does it flake? I'm just wondering if we should disable it in the mean time to avoid angry contributors.
On my PC, about 1/1000. I will post a PR to disable it for now.
Thank you so much! That's not a bad rate at all, but we run a lot of tests. This is super helpful!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle stale I'm still working on it, a new golang version(1.16 or 1.15.7+) release required
/sig api-machinery /remove-sig testing AFAICT this is specific to these tests, not the test framework or CI infrastructure.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
the tests re-enabled in started flaking again immediately Example failure: /milestone v1.21 the tests should be deflaked or disabled for 1.21
/cc
/triage accepted
+1 in recent PR tests
Sorry for my slow response. I'm working on it, looks like I need to debug golang HTTP client again :-P
Shall we revert in the meantime?
No, has already fixed by golang 1.15.8 I think. This one is caused by another problem.
I will disable TestClientReceivedGOAWAY case again if I can not find the root cause this night.
Can not reproduce on my PC, disable this test case before I figure out the root cause:
I run this test case with this command and can not reproduce. Does any thing different with command ?
moving out of the milestone and lowering priority becuase merged, leaving open because the fundamental issue is unresolved /milestone clear
will do something comparable under make. There's a bunch of stuff in there, generally it's not relevant for a unit test bug like this but the answer to what exactly it does is not easily concise. Note that CI may have more load than your local machine running it once, and also currently runs as root.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle active /assign ref:

Discussion:
err error } <del> // newGOAWAYClient return a configured http client which used to request test GOAWAY server, a dial may specified // to encounter the TCP connection. func newGOAWAYClient(dial func(network, addr string, cfg *tls.Config) (net.Conn, error)) (*http.Client, error) { tlsConfig := &tls.Config{ InsecureSkipVerify: true, NextProtos:         []string{http2.NextProtoTLS}, } tr := &http.Transport{ Proxy:               http.ProxyFromEnvironment, TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, MaxIdleConnsPerHost: 25, DialTLSContext: func(ctx context.Context, network, addr string) (net.Conn, error) { if dial != nil { return dial(network, addr, tlsConfig) } return tls.Dial(network, addr, tlsConfig) }, } if err := http2.ConfigureTransport(tr); err != nil { return nil, err } client := &http.Client{ Transport: tr, } return client, nil } </del> // requestGOAWAYServer request test GOAWAY server using specified method and data according to the given url. // A non-nil channel will be returned if the request is watch, and a watchResponse can be got from the channel when watch done. func requestGOAWAYServer(client *http.Client, serverBaseURL, url string) (<-chan watchResponse, error) {

// TestClientReceivedGOAWAY tests the in-flight watch requests will not be affected and new requests use a new // connection after client received GOAWAY. func TestClientReceivedGOAWAY(t *testing.T) { <del> t.Skip("disabled because of https://github.com/kubernetes/kubernetes/issues/94622") </del> s, err := newTestGOAWAYServer() if err != nil { t.Fatalf("failed to set-up test GOAWAY http server, err: %v", err)

for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { <del> var mu sync.Mutex </del> // localAddr indicates how many TCP connection set up localAddr := make([]string, 0) <del> client, err := newGOAWAYClient(func(network, addr string, cfg *tls.Config) (conn net.Conn, err error) { </del> <ins> // create the http client dialFn := func(network, addr string, cfg *tls.Config) (conn net.Conn, err error) { </ins> conn, err = tls.Dial(network, addr, cfg) if err != nil { t.Fatalf("unexpect connection err: %v", err) } <del> mu.Lock() </del> localAddr = append(localAddr, conn.LocalAddr().String()) <del> mu.Unlock() </del> return <del> }) if err != nil { t.Fatalf("failed to set-up client, err: %v", err) </del> <ins> } tlsConfig := &tls.Config{ InsecureSkipVerify: true, NextProtos:         []string{http2.NextProtoTLS}, } tr := &http.Transport{ TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, // Disable connection pooling to avoid additional connections // that cause the test to flake MaxIdleConnsPerHost: -1, DialTLSContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialFn(network, addr, tlsConfig) }, } if err := http2.ConfigureTransport(tr); err != nil { t.Fatalf("failed to configure http transport, err: %v", err) } client := &http.Client{ Transport: tr, </ins> } watchChs := make([]<-chan watchResponse, 0)

s.StartTLS() defer s.Close() <del> client, err := newGOAWAYClient(nil) </del> <ins> // create the http client tlsConfig := &tls.Config{ InsecureSkipVerify: true, NextProtos:         []string{http2.NextProtoTLS}, } tr := &http.Transport{ TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, MaxIdleConnsPerHost: 25, } if err := http2.ConfigureTransport(tr); err != nil { t.Fatalf("failed to configure http transport, err: %v", err) } client := &http.Client{ Transport: tr, } </ins> if err != nil { t.Fatalf("failed to set-up client, err: %v", err) }