Change:
shouldFail      bool watchTimeout    time.Duration }{ <ins> // PV agnostic path positive test cases </ins> { <del> testName:        "normal, json file exists", </del> <ins> testName:        "success, json file exists", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        `{"driverName": "csi", "volumeHandle":"project/zone/test-vol1"}`, <del> createPV:        false, </del> stageUnstageSet: true, }, { <del> testName:        "normal, json file doesn't exist -> use PV", </del> <ins> testName:        "stage_unstage not set, PV agnostic path, unmount device is skipped", deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, // PV agnostic path negative test cases { testName:        "fail: missing json, fail to retrieve driver and volumeID from globalpath", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        "", <del> createPV:        true, </del> stageUnstageSet: true, <ins> shouldFail:      true, </ins> }, { <del> testName:        "invalid json ->  use PV", </del> <ins> testName:        "fail: invalid json, fail to retrieve driver and volumeID from globalpath", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        `{"driverName"}}`, <del> createPV:        true, </del> stageUnstageSet: true, <ins> shouldFail:      true, </ins> }, <ins> // Old style PV based path positive test cases </ins> { <del> testName:        "no json, no PV.volID", volID:           "", </del> <ins> testName:        "success: json file doesn't exist, old style pv based global path -> use PV", volID:           "project/zone/test-vol1", </ins> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", <ins> stageUnstageSet: true, </ins> createPV:        true, <del> shouldFail:      true, </del> }, { <del> testName:        "no json, no PV", </del> <ins> testName:        "success: invalid json file, old style pv based global path -> use PV", </ins> volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", <del> jsonFile:        "", createPV:        false, </del> <ins> jsonFile:        `{"driverName"}}`, </ins> stageUnstageSet: true, <del> shouldFail:      true, </del> <ins> createPV:        true, </ins> }, { <del> testName:        "stage_unstage not set no vars should not fail", </del> <ins> testName:        "stage_unstage not set, PV based path, unmount device is skipped", </ins> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, <ins> // Old style PV based path negative test cases { testName:        "fail: json file doesn't exist, old style pv based device path, missing PV", volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", stageUnstageSet: true, shouldFail:      true, }, { testName:        "fail: no json, no PV.volID, old style pv based global path", volID:           "", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", shouldFail:      true, }, </ins> } for _, tc := range testCases {

Discussion:
When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.