Change:
I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.
/assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.

Discussion:
"github.com/vmware/govmomi/object" "github.com/vmware/govmomi/vim25/mo" <del> "k8s.io/klog/v2" </del> v1 "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> k8stypes "k8s.io/apimachinery/pkg/types" <ins> coreclients "k8s.io/client-go/kubernetes/typed/core/v1" corelisters "k8s.io/client-go/listers/core/v1" </ins> cloudprovider "k8s.io/cloud-provider" <ins> "k8s.io/klog/v2" </ins> "k8s.io/legacy-cloud-providers/vsphere/vclib" )

//CredentialsManager credentialManager *SecretCredentialManager <ins> nodeLister corelisters.NodeLister nodeGetter coreclients.NodesGetter </ins> // Mutexes registeredNodesLock   sync.RWMutex nodeInfoLock          sync.RWMutex

nm.registeredNodesLock.RLock() node := nm.registeredNodes[convertToString(nodeName)] nm.registeredNodesLock.RUnlock() <del> if node == nil { return v1.Node{}, vclib.ErrNoVMFound </del> <ins> if node != nil { klog.V(4).Infof("Node %s found in vSphere cloud provider cache", nodeName) return *node, nil } if nm.nodeLister != nil { klog.V(4).Infof("Node %s missing in vSphere cloud provider cache, trying node informer") node, err := nm.nodeLister.Get(convertToString(nodeName)) if err != nil { if !errors.IsNotFound(err) { return v1.Node{}, err } // Fall through with IsNotFound error and try to get the node from the API server } else { node := node.DeepCopy() nm.addNode(node) klog.V(4).Infof("Node %s found in vSphere cloud provider node informer", nodeName) return *node, nil } </ins> } <del> return *node, nil </del> <ins> if nm.nodeGetter != nil { klog.V(4).Infof("Node %s missing in vSphere cloud provider caches, trying the API server") node, err := nm.nodeGetter.Nodes().Get(context.TODO(), convertToString(nodeName), metav1.GetOptions{}) if err != nil { if !errors.IsNotFound(err) { return v1.Node{}, err } // Fall through with IsNotFound error to keep the code consistent with the above } else { nm.addNode(node) klog.V(4).Infof("Node %s found in the API server", nodeName) return *node, nil } } klog.V(4).Infof("Node %s not found in vSphere cloud provider", nodeName) return v1.Node{}, vclib.ErrNoVMFound </ins> } func (nm *NodeManager) getNodes() map[string]*v1.Node {

klog.V(4).Infof("GetHostsInZone %v returning: %v", zoneFailureDomain, hosts) return hosts, nil } <ins> func (nm *NodeManager) SetNodeLister(nodeLister corelisters.NodeLister) { nm.nodeLister = nodeLister } func (nm *NodeManager) SetNodeGetter(nodeGetter coreclients.NodesGetter) { nm.nodeGetter = nodeGetter } </ins>

<ins> //go:build !providerless // +build !providerless /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package vsphere import ( "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake" "k8s.io/legacy-cloud-providers/vsphere/vclib" ) // Annotation used to distinguish nodes in node cache / informer / API server const nodeAnnotation = "test" func getNode(annotation string) *v1.Node { return &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node1", Annotations: map[string]string{ nodeAnnotation: annotation, }, }, } } func TestGetNode(t *testing.T) { tests := []struct { name           string cachedNodes    []*v1.Node informerNodes  []*v1.Node // "nil" means that the NodeManager has no nodeLister apiServerNodes []*v1.Node // "nil" means that the NodeManager has no nodeGetter expectedNodeAnnotation string expectNotFound         bool }{ { name:           "No cached node anywhere", cachedNodes:    []*v1.Node{}, informerNodes:  []*v1.Node{}, apiServerNodes: []*v1.Node{}, expectNotFound: true, }, { name:           "No lister & getter", cachedNodes:    []*v1.Node{}, informerNodes:  nil, apiServerNodes: nil, expectNotFound: true, }, { name:                   "cache is used first", cachedNodes:            []*v1.Node{getNode("cache")}, informerNodes:          []*v1.Node{getNode("informer")}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "cache", }, { name:                   "informer is used second", cachedNodes:            []*v1.Node{}, informerNodes:          []*v1.Node{getNode("informer")}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "informer", }, { name:                   "API server is used last", cachedNodes:            []*v1.Node{}, informerNodes:          []*v1.Node{}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "apiserver", }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { // local NodeManager cache cache := make(map[string]*v1.Node) for _, node := range test.cachedNodes { cache[node.Name] = node } // Client with apiServerNodes objs := []runtime.Object{} for _, node := range test.apiServerNodes { objs = append(objs, node) } client := fake.NewSimpleClientset(objs...) nodeGetter := client.CoreV1() // Informer + nodeLister. Despite the client already has apiServerNodes, they won't appear in the // nodeLister, because the informer is never started. factory := informers.NewSharedInformerFactory(client, 0 /* no resync */) nodeInformer := factory.Core().V1().Nodes() for _, node := range test.informerNodes { nodeInformer.Informer().GetStore().Add(node) } nodeLister := nodeInformer.Lister() nodeManager := NodeManager{ registeredNodes: cache, } if test.informerNodes != nil { nodeManager.SetNodeLister(nodeLister) } if test.apiServerNodes != nil { nodeManager.SetNodeGetter(nodeGetter) } node, err := nodeManager.GetNode("node1") if test.expectNotFound && err != vclib.ErrNoVMFound { t.Errorf("Expected NotFound error, got: %v", err) } if !test.expectNotFound && err != nil { t.Errorf("Unexpected error: %s", err) } if test.expectedNodeAnnotation != "" { if node.Annotations == nil { t.Errorf("Expected node with annotation %q, got nil", test.expectedNodeAnnotation) } else { if ann := node.Annotations[nodeAnnotation]; ann != test.expectedNodeAnnotation { t.Errorf("Expected node with annotation %q, got %q", test.expectedNodeAnnotation, ann) } } } }) } } </ins>

// Initialize passes a Kubernetes clientBuilder interface to the cloud provider func (vs *VSphere) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) { vs.kubeClient = clientBuilder.ClientOrDie("vsphere-legacy-cloud-provider") <ins> vs.nodeManager.SetNodeGetter(vs.kubeClient.CoreV1()) </ins> } // Initialize Node Informers

cache.ResourceEventHandlerFuncs{UpdateFunc: vs.syncNodeZoneLabels}, zoneLabelsResyncPeriod, ) <ins> nodeLister := informerFactory.Core().V1().Nodes().Lister() vs.nodeManager.SetNodeLister(nodeLister) </ins> klog.V(4).Infof("Node informers in vSphere cloud provider initialized") }