Code change:
nm.registeredNodesLock.RLock() node := nm.registeredNodes[convertToString(nodeName)] nm.registeredNodesLock.RUnlock() <del> if node == nil { return v1.Node{}, vclib.ErrNoVMFound </del> <ins> if node != nil { klog.V(4).Infof("Node %s found in vSphere cloud provider cache", nodeName) return *node, nil } if nm.nodeLister != nil { klog.V(4).Infof("Node %s missing in vSphere cloud provider cache, trying node informer") node, err := nm.nodeLister.Get(convertToString(nodeName)) if err != nil { if !errors.IsNotFound(err) { return v1.Node{}, err } // Fall through with IsNotFound error and try to get the node from the API server } else { node := node.DeepCopy() nm.addNode(node) klog.V(4).Infof("Node %s found in vSphere cloud provider node informer", nodeName) return *node, nil } </ins> } <del> return *node, nil </del> <ins> if nm.nodeGetter != nil { klog.V(4).Infof("Node %s missing in vSphere cloud provider caches, trying the API server") node, err := nm.nodeGetter.Nodes().Get(context.TODO(), convertToString(nodeName), metav1.GetOptions{}) if err != nil { if !errors.IsNotFound(err) { return v1.Node{}, err } // Fall through with IsNotFound error to keep the code consistent with the above } else { nm.addNode(node) klog.V(4).Infof("Node %s found in the API server", nodeName) return *node, nil } } klog.V(4).Infof("Node %s not found in vSphere cloud provider", nodeName) return v1.Node{}, vclib.ErrNoVMFound </ins> } func (nm *NodeManager) getNodes() map[string]*v1.Node {

Related discussion:
text: I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.