Code change:
} func (f *fakeAttachDetachVolumeHost) CSINodeLister() storagelistersv1.CSINodeLister { <del> // not needed for testing return nil </del> <ins> csiNode := &storagev1.CSINode{ ObjectMeta: metav1.ObjectMeta{Name: f.nodeName}, Spec: storagev1.CSINodeSpec{ Drivers: []storagev1.CSINodeDriver{}, }, } enableMigrationOnNode(csiNode, csilibplugins.GCEPDInTreePluginName) return getFakeCSINodeLister(csiNode) } func enableMigrationOnNode(csiNode *storagev1.CSINode, pluginName string) { nodeInfoAnnotations := csiNode.GetAnnotations() if nodeInfoAnnotations == nil { nodeInfoAnnotations = map[string]string{} } newAnnotationSet := sets.NewString() newAnnotationSet.Insert(pluginName) nas := strings.Join(newAnnotationSet.List(), ",") nodeInfoAnnotations[v1.MigratedPluginsAnnotationKey] = nas csiNode.Annotations = nodeInfoAnnotations </ins> } func (f *fakeAttachDetachVolumeHost) CSIDriverLister() storagelistersv1.CSIDriverLister {

Related discussion:
text: After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.