Change:
ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver. In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see

Discussion:
"net" "net/http" "sync" <del> "sync/atomic" </del> "time" v1 "k8s.io/api/core/v1"

// need more than one per sidecar and kubelet. Keeping this reasonably // small ensures that we don't establish connections through the apiserver // and the remote kernel which then aren't needed. <ins> // // The proxy code below establishes this many connections in advance, // without waiting for a client on the remote side. On the local side // a gRPC server will accept the same number of connections and then wait // for data from a future client. // // This approach has the advantage that a client on the remote side can // immediately start communicating, without the delay caused by establishing // the connection. That delay is large enough that clients like the // node-driver-registrar with a very small timeout for gRPC did indeed // time out unnecessarily. </ins> const maxConcurrentConnections = 10 <ins> // This delay determines how quickly we notice when someone has // connected inside the cluster. With socat, we cannot make this too small // because otherwise we get many rejected connections. With the mock // driver as proxy that doesn't happen as long as we don't // ask for too many concurrent connections because the mock driver // keeps the listening port open at all times and the Linux // kernel automatically accepts our connection requests. const connectionPollInterval = 100 * time.Millisecond </ins> // Listen creates a listener which returns new connections whenever someone connects // to a socat or mock driver proxy instance running inside the given pod. //

prefix := fmt.Sprintf("port forwarding for %s", addr) ctx, cancel := context.WithCancel(ctx) l := &listener{ <del> connections: make(chan *connection), ctx:         ctx, cancel:      cancel, addr:        addr, </del> <ins> ctx:    ctx, cancel: cancel, addr:   addr, </ins> } <del> var connectionsCreated, connectionsClosed int32 </del> <ins> var connectionsCreated int </ins> runForwarding := func() { klog.V(2).Infof("%s: starting connection polling", prefix) defer klog.V(2).Infof("%s: connection polling ended", prefix) <del> // This delay determines how quickly we notice when someone has // connected inside the cluster. With socat, we cannot make this too small // because otherwise we get many rejected connections. With the mock // driver as proxy that doesn't happen as long as we don't // ask for too many concurrent connections because the mock driver // keeps the listening port open at all times and the Linux // kernel automatically accepts our connection requests. tryConnect := time.NewTicker(100 * time.Millisecond) </del> <ins> tryConnect := time.NewTicker(connectionPollInterval) </ins> defer tryConnect.Stop() for { select { case <-ctx.Done(): return case <-tryConnect.C: <del> currentClosed := atomic.LoadInt32(&connectionsClosed) openConnections := connectionsCreated - currentClosed if openConnections >= maxConcurrentConnections { break } klog.V(5).Infof("%s: trying to create a new connection #%d, %d open", prefix, connectionsCreated, openConnections) stream, err := dial(ctx, fmt.Sprintf("%s #%d", prefix, connectionsCreated), dialer, addr.Port) if err != nil { klog.Errorf("%s: no connection: %v", prefix, err) break } // Make the connection available to Accept below. klog.V(5).Infof("%s: created a new connection #%d", prefix, connectionsCreated) l.connections <- &connection{ stream:  stream, addr:    addr, counter: connectionsCreated, closed:  &connectionsClosed, } connectionsCreated++ </del> <ins> func() { l.mutex.Lock() defer l.mutex.Unlock() for i, c := range l.connections { if c == nil { klog.V(5).Infof("%s: trying to create a new connection #%d", prefix, connectionsCreated) stream, err := dial(ctx, fmt.Sprintf("%s #%d", prefix, connectionsCreated), dialer, addr.Port) if err != nil { klog.Errorf("%s: no connection: %v", prefix, err) return } // Make the connection available to Accept below. klog.V(5).Infof("%s: created a new connection #%d", prefix, connectionsCreated) c := &connection{ l:       l, stream:  stream, addr:    addr, counter: connectionsCreated, } l.connections[i] = c connectionsCreated++ return } } }() </ins> } } }

} type listener struct { <del> addr        Addr connections chan *connection ctx         context.Context cancel      func() </del> <ins> addr   Addr ctx    context.Context cancel func() mutex       sync.Mutex connections [maxConcurrentConnections]*connection </ins> } var _ net.Listener = &listener{}

func (l *listener) Close() error { klog.V(5).Infof("forward listener for %s: closing", l.addr) l.cancel() <ins> l.mutex.Lock() defer l.mutex.Unlock() for _, c := range l.connections { if c != nil { c.stream.Close() } } </ins> return nil } func (l *listener) Accept() (net.Conn, error) { <del> select { case <-l.ctx.Done(): return nil, errors.New("listening was stopped") case c := <-l.connections: klog.V(5).Infof("forward listener for %s: got a new connection #%d", l.addr, c.counter) return c, nil </del> <ins> tryAccept := time.NewTicker(connectionPollInterval) defer tryAccept.Stop() for { select { case <-l.ctx.Done(): return nil, errors.New("listening was stopped") case <-tryAccept.C: conn := func() net.Conn { l.mutex.Lock() defer l.mutex.Unlock() for _, c := range l.connections { if c != nil && !c.accepted { klog.V(5).Infof("forward listener for %s: got a new connection #%d", l.addr, c.counter) c.accepted = true return c } } return nil }() if conn != nil { return conn, nil } } </ins> } } type connection struct { <del> stream  *stream addr    Addr counter int32 closed  *int32 mutex   sync.Mutex </del> <ins> l                *listener stream           *stream addr             Addr counter          int mutex            sync.Mutex accepted, closed bool </ins> } var _ net.Conn = &connection{}

func (c *connection) Close() error { c.mutex.Lock() defer c.mutex.Unlock() <del> if c.closed != nil { </del> <ins> if !c.closed { </ins> // Do the logging and book-keeping only once. The function itself may be called more than once. klog.V(5).Infof("forward connection #%d for %s: closing our side", c.counter, c.addr) <del> atomic.AddInt32(c.closed, 1) c.closed = nil </del> <ins> c.l.mutex.Lock() defer c.l.mutex.Unlock() for i, c2 := range c.l.connections { if c2 == c { c.l.connections[i] = nil break } } </ins> } c.stream.Close() return nil }