Change:
I migrated an existing EBS volume from in-tree plugin to external aws-ebs-csi-driver (). Resizing PVC twice in 6 hour window causes kube-controller-manager to crash. Exception message: Kube-controller-manager shouldn't crash in such case. If external aws-ebs-csi-driver fails to provision the pvc it should just log an warning or error and continue operating as usual running cluster version 1.22.9 using in-tree aws ebs driver. pvc: Install aws-ebs-csi-driver following featureGate CSIMigrationAWS on all cluster components (with draining nodes before enabling featureGate on kubelets - according to warning ) we have ebs volume that has been migrated to aws-ebs-csi-driver. pvc by increasing .spec.resources.request.storage to 2Gi. PVC, PV and EBS volume are correctly resized. once again not later than 6 hours after first resize. Increase .spec.resources.request.storage to 3Gi. Now kube-controller-manager throws an exception and crashes: EBS CSI Driver logs: I tried also enabling alpha featureGate InTreePluginAWSUnregister but the issue still occurs. <details</details<detailsAWS </details<details</details<details</details<detailsdocker </details<detailsaws-ebs-csi-driver version 1.10.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/sig storage some situation where the sync loop passes a nil parameter to
/assign
SetClaimResizer may return nil pvc and non-nil error, ClaimToClaimKey naively assumes non-nil as described above, will create PR
But the posted bug is I think 1.22 specific and should never happen on master or 1.24 and hence this fix against master is kind of okay and should fix the issue but at the same time code in is still kinda broken in 1.22 because it returns if patching fails.

Discussion:
return nil } if err != nil { <del> klog.V(5).Infof("Error getting PVC %q (uid: %q) from informer : %v", util.GetPersistentVolumeClaimQualifiedName(pvc), pvc.UID, err) </del> <ins> klog.V(5).Infof("Error getting PVC %q from informer : %v", key, err) </ins> return err } pv, err := expc.getPersistentVolume(ctx, pvc) if err != nil { <del> klog.V(5).Infof("Error getting Persistent Volume for PVC %q (uid: %q) from informer : %v", util.GetPersistentVolumeClaimQualifiedName(pvc), pvc.UID, err) </del> <ins> klog.V(5).Infof("Error getting Persistent Volume for PVC %q (uid: %q) from informer : %v", key, pvc.UID, err) </ins> return err } if pv.Spec.ClaimRef == nil || pvc.Namespace != pv.Spec.ClaimRef.Namespace || pvc.UID != pv.Spec.ClaimRef.UID { <del> err := fmt.Errorf("persistent Volume is not bound to PVC being updated : %s", util.ClaimToClaimKey(pvc)) </del> <ins> err := fmt.Errorf("persistent Volume is not bound to PVC being updated : %s", key) </ins> klog.V(4).Infof("%v", err) return err }

volumeSpec := volume.NewSpecFromPersistentVolume(pv, false) migratable, err := expc.csiMigratedPluginManager.IsMigratable(volumeSpec) if err != nil { <del> klog.V(4).Infof("failed to check CSI migration status for PVC: %s with error: %v", util.ClaimToClaimKey(pvc), err) </del> <ins> klog.V(4).Infof("failed to check CSI migration status for PVC: %s with error: %v", key, err) </ins> return nil } // handle CSI migration scenarios before invoking FindExpandablePluginBySpec for in-tree

expc.recorder.Event(pvc, v1.EventTypeNormal, events.ExternalExpanding, msg) csiResizerName, err := expc.translator.GetCSINameFromInTreeName(inTreePluginName) if err != nil { <del> errorMsg := fmt.Sprintf("error getting CSI driver name for pvc %s, with error %v", util.ClaimToClaimKey(pvc), err) </del> <ins> errorMsg := fmt.Sprintf("error getting CSI driver name for pvc %s, with error %v", key, err) </ins> expc.recorder.Event(pvc, v1.EventTypeWarning, events.ExternalExpanding, errorMsg) return fmt.Errorf(errorMsg) } pvc, err := util.SetClaimResizer(pvc, csiResizerName, expc.kubeClient) if err != nil { <del> errorMsg := fmt.Sprintf("error setting resizer annotation to pvc %s, with error %v", util.ClaimToClaimKey(pvc), err) </del> <ins> errorMsg := fmt.Sprintf("error setting resizer annotation to pvc %s, with error %v", key, err) </ins> expc.recorder.Event(pvc, v1.EventTypeWarning, events.ExternalExpanding, errorMsg) return fmt.Errorf(errorMsg) }

eventType = v1.EventTypeWarning } expc.recorder.Event(pvc, eventType, events.ExternalExpanding, fmt.Sprintf("Ignoring the PVC: %v.", msg)) <del> klog.Infof("Ignoring the PVC %q (uid: %q) : %v.", util.GetPersistentVolumeClaimQualifiedName(pvc), pvc.UID, msg) </del> <ins> klog.Infof("Ignoring the PVC %q (uid: %q) : %v.", key, pvc.UID, msg) </ins> // If we are expecting that an external plugin will handle resizing this volume then // is no point in requeuing this PVC. return nil

processed bool } <del> // ClaimToClaimKey return namespace/name string for pvc func ClaimToClaimKey(claim *v1.PersistentVolumeClaim) string { return fmt.Sprintf("%s/%s", claim.Namespace, claim.Name) } </del> // UpdatePVSize updates just pv size after cloudprovider resizing is successful func UpdatePVSize( pv *v1.PersistentVolume,