Change:
prefix := fmt.Sprintf("port forwarding for %s", addr) ctx, cancel := context.WithCancel(ctx) l := &listener{ <del> connections: make(chan *connection), ctx:         ctx, cancel:      cancel, addr:        addr, </del> <ins> ctx:    ctx, cancel: cancel, addr:   addr, </ins> } <del> var connectionsCreated, connectionsClosed int32 </del> <ins> var connectionsCreated int </ins> runForwarding := func() { klog.V(2).Infof("%s: starting connection polling", prefix) defer klog.V(2).Infof("%s: connection polling ended", prefix) <del> // This delay determines how quickly we notice when someone has // connected inside the cluster. With socat, we cannot make this too small // because otherwise we get many rejected connections. With the mock // driver as proxy that doesn't happen as long as we don't // ask for too many concurrent connections because the mock driver // keeps the listening port open at all times and the Linux // kernel automatically accepts our connection requests. tryConnect := time.NewTicker(100 * time.Millisecond) </del> <ins> tryConnect := time.NewTicker(connectionPollInterval) </ins> defer tryConnect.Stop() for { select { case <-ctx.Done(): return case <-tryConnect.C: <del> currentClosed := atomic.LoadInt32(&connectionsClosed) openConnections := connectionsCreated - currentClosed if openConnections >= maxConcurrentConnections { break } klog.V(5).Infof("%s: trying to create a new connection #%d, %d open", prefix, connectionsCreated, openConnections) stream, err := dial(ctx, fmt.Sprintf("%s #%d", prefix, connectionsCreated), dialer, addr.Port) if err != nil { klog.Errorf("%s: no connection: %v", prefix, err) break } // Make the connection available to Accept below. klog.V(5).Infof("%s: created a new connection #%d", prefix, connectionsCreated) l.connections <- &connection{ stream:  stream, addr:    addr, counter: connectionsCreated, closed:  &connectionsClosed, } connectionsCreated++ </del> <ins> func() { l.mutex.Lock() defer l.mutex.Unlock() for i, c := range l.connections { if c == nil { klog.V(5).Infof("%s: trying to create a new connection #%d", prefix, connectionsCreated) stream, err := dial(ctx, fmt.Sprintf("%s #%d", prefix, connectionsCreated), dialer, addr.Port) if err != nil { klog.Errorf("%s: no connection: %v", prefix, err) return } // Make the connection available to Accept below. klog.V(5).Infof("%s: created a new connection #%d", prefix, connectionsCreated) c := &connection{ l:       l, stream:  stream, addr:    addr, counter: connectionsCreated, } l.connections[i] = c connectionsCreated++ return } } }() </ins> } } }

Discussion:
ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver.

In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see