Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Context: When a cluster has dangling s caused by restarts of kube-controller-manager (like described in ), the attach/detach controller tries to clean them up on startup (ref ). For this, it iterates over all s and finds the responsible volume plugin for retrieving the unique volume name. Problem: However, if the volume was provisioned by an in-tree plugin, then later migrated to CSI and now the / feature gate is set, the volume is skipped during , so the will stay in the system and the volume will not be detached from the undesired node and potentially block a new attach operation to a different node. Analysis: The / feature gate causes the respective in-tree volume plugins not to be registered: The spec only contains the in-tree fields (e.g or ), but not the fields: This causes the call in to find no responsible volume plugin for the , which will skip the cleanup: That's why we see a lot of logs like this on startup of kube-controller-manager and dangling s on our clusters: The attach/detach controller should cleanup dangling s for migrated in-tree plugins with unregistered in-tree plugins as well. ref , addresses a similar issue. Though, the fix doesn't come into effect, if the respective / feature gate is enabled. Kubernetes version (use ): / Cloud provider or hardware configuration: AWS, GCP
/sig storage cc
Hey, thanks for reporting this bug!! I will try to come up with a fix. BTW, wondering what's the symptom you observed except for the log. Is there any volume operation(attaching/detaching/mounting/unmounting/creation/deletion) not working? This, to my understanding, will only make dangling attachment not working properly but I guess normal operation should work, is it?
The consequence of the leftover s in our case is that volumes are not detached on infrastructure provider side when moving pods with volumes between nodes, e.g. because of rolling updates/eviction/etc.. This in turn causes the new s on the new nodes to fail with logs similar to this one: (we observe the same for GCP) I can also work on a fix tomorrow, if you want. I think, we should be able to reuse the fix from and additionally determine the from or similar, instead of only relying on .
That would be nice! Really appreciate it. Yep, sounds like a good idea. If you could also help to verify the fix in your environment that would be awesome
Ok, great. Will try to come up with a fix. /assign Definitely! I'll try my best and let you know if I need some help :)
Yep, we observed this after experiencing the bug. We ended up manually creating a cleanup script to check EC2 <-K8s VA if it was actually attached or not, otherwise delete VA. My understanding is with fixed we will no longer have this issue, but it would be nice to have the garbage collection logic work for migrated/non-migrated volumes regardless
/triage accepted

Discussion:
klog.Errorf("Unable to lookup pv object for: %q, err: %v", *pvName, err) continue } <ins> var plugin volume.AttachableVolumePlugin </ins> volumeSpec := volume.NewSpecFromPersistentVolume(pv, false) <del> plugin, err := adc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil || plugin == nil { // Currently VA objects are created for CSI volumes only. nil plugin is unexpected, generate a warning klog.Warningf( "Skipping processing the volume %q on nodeName: %q, no attacher interface found. err=%v", *pvName, nodeName, err) continue </del> <ins> // Consult csiMigratedPluginManager first before querying the plugins registered during runtime in volumePluginMgr. // In-tree plugins that provisioned PVs will not be registered anymore after migration to CSI, once the respective // feature gate is enabled. if inTreePluginName, err := adc.csiMigratedPluginManager.GetInTreePluginNameFromSpec(pv, nil); err == nil { if adc.csiMigratedPluginManager.IsMigrationEnabledForPlugin(inTreePluginName) { // PV is migrated and should be handled by the CSI plugin instead of the in-tree one plugin, _ = adc.volumePluginMgr.FindAttachablePluginByName(csi.CSIPluginName) // podNamespace is not needed here for Azurefile as the volumeName generated will be the same with or without podNamespace volumeSpec, err = csimigration.TranslateInTreeSpecToCSI(volumeSpec, "" /* podNamespace */, adc.intreeToCSITranslator) if err != nil { klog.Errorf( "Failed to translate intree volumeSpec to CSI volumeSpec for volume:%q, va.Name:%q, nodeName:%q: %s. Error: %v", *pvName, va.Name, nodeName, inTreePluginName, err) continue } } </ins> } <del> pluginName := plugin.GetPluginName() if adc.csiMigratedPluginManager.IsMigrationEnabledForPlugin(pluginName) { plugin, _ = adc.volumePluginMgr.FindAttachablePluginByName(csi.CSIPluginName) // podNamespace is not needed here for Azurefile as the volumeName generated will be the same with or without podNamespace volumeSpec, err = csimigration.TranslateInTreeSpecToCSI(volumeSpec, "" /* podNamespace */, adc.intreeToCSITranslator) if err != nil { klog.Errorf( "Failed to translate intree volumeSpec to CSI volumeSpec for volume:%q, va.Name:%q, nodeName:%q: %v", </del> <ins> if plugin == nil { plugin, err = adc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil || plugin == nil { // Currently VA objects are created for CSI volumes only. nil plugin is unexpected, generate a warning klog.Warningf( "Skipping processing the volume %q on nodeName: %q, no attacher interface found. err=%v", </ins> *pvName, <del> va.Name, </del> nodeName, <del> pluginName, </del> err) continue } } volumeName, err := volumeutil.GetUniqueVolumeNameFromSpec(plugin, volumeSpec) if err != nil { klog.Errorf(

"k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/csi" <del> "k8s.io/kubernetes/pkg/volume/gcepd" </del> "k8s.io/kubernetes/pkg/volume/util" )

} type vaTest struct { <del> testName          string volName           string podName           string podNodeName       string pvName            string vaName            string vaNodeName        string vaAttachStatus    bool csiMigration      bool expected_attaches map[string][]string expected_detaches map[string][]string </del> <ins> testName               string volName                string podName                string podNodeName            string pvName                 string vaName                 string vaNodeName             string vaAttachStatus         bool csiMigration           bool expected_attaches      map[string][]string expected_detaches      map[string][]string expectedASWAttachState cache.AttachState </ins> } func Test_ADC_VolumeAttachmentRecovery(t *testing.T) {

expected_attaches: map[string][]string{}, expected_detaches: map[string][]string{"mynode-1": {"vol1"}}, }, <del> { testName:       "CSI Migration", volName:        "vol1", podNodeName:    "mynode-1", pvName:         "pv1", vaName:         "va1", vaNodeName:     "mynode-1", vaAttachStatus: false, csiMigration:   true, </del> <ins> { // pod is scheduled, volume is migrated, attach status:false, verify volume is marked as attached testName:               "Scheduled Pod with migrated PV", volName:                "vol1", podNodeName:            "mynode-1", pvName:                 "pv1", vaName:                 "va1", vaNodeName:             "mynode-1", vaAttachStatus:         false, csiMigration:           true, expectedASWAttachState: cache.AttachStateAttached, }, { // pod is deleted, volume is migrated, attach status:false, verify volume is marked as uncertain testName:               "Deleted Pod with migrated PV", volName:                "vol1", pvName:                 "pv1", vaName:                 "va1", vaNodeName:             "mynode-1", vaAttachStatus:         false, csiMigration:           true, expectedASWAttachState: cache.AttachStateUncertain, </ins> }, } { t.Run(tc.testName, func(t *testing.T) {

defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.CSIMigrationGCE, tc.csiMigration)() defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InTreePluginGCEUnregister, tc.csiMigration)() <del> plugins = gcepd.ProbeVolumePlugins() </del> <ins> // if InTreePluginGCEUnregister is enabled, only the CSI plugin is registered but not the in-tree one </ins> plugins = append(plugins, csi.ProbeVolumePlugins()...) } else { plugins = controllervolumetesting.CreateTestPlugin()

// Since csi migration is turned on, the attach state for the PV should be in CSI format. attachedState := adc.actualStateOfWorld.GetAttachState( v1.UniqueVolumeName(csiPDUniqueNamePrefix+tc.volName), types.NodeName(tc.vaNodeName)) <del> if attachedState != cache.AttachStateAttached { t.Fatalf("Expected attachedState %v, but it is %v", cache.AttachStateAttached, attachedState) </del> <ins> if attachedState != tc.expectedASWAttachState { t.Fatalf("Expected attachedState %v, but it is %v", tc.expectedASWAttachState, attachedState) </ins> } // kubernetes.io/gce-pd/<volName> should not be marked when CSI Migration is on