Change:
When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me. staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need. It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.

Discussion:
func makeDeviceMountPath(plugin *csiPlugin, spec *volume.Spec) (string, error) { if spec == nil { <del> return "", errors.New(log("makeDeviceMountPath failed, spec is nil")) </del> <ins> return "", errors.New("makeDeviceMountPath failed, spec is nil") </ins> } <del> pvName := spec.PersistentVolume.Name if pvName == "" { return "", errors.New(log("makeDeviceMountPath failed, pv name empty")) </del> <ins> driver, err := GetCSIDriverName(spec) if err != nil { return "", err } if driver == "" { return "", errors.New("makeDeviceMountPath failed, csi source driver name is empty") } csiSource, err := getPVSourceFromSpec(spec) if err != nil { return "", errors.New(log("makeDeviceMountPath failed to get CSIPersistentVolumeSource: %v", err)) } if csiSource.VolumeHandle == "" { return "", errors.New("makeDeviceMountPath failed, CSIPersistentVolumeSource volume handle is empty") </ins> } <del> return filepath.Join(plugin.host.GetPluginDir(plugin.GetPluginName()), persistentVolumeInGlobalPath, pvName, globalMountInGlobalPath), nil </del> <ins> result := sha256.Sum256([]byte(fmt.Sprintf("%s", csiSource.VolumeHandle))) volSha := fmt.Sprintf("%x", result) return filepath.Join(plugin.host.GetPluginDir(plugin.GetPluginName()), driver, volSha, globalMountInGlobalPath), nil </ins> } func getDriverAndVolNameFromDeviceMountPath(k8s kubernetes.Interface, deviceMountPath string) (string, string, error) {

import ( "context" <ins> "crypto/sha256" </ins> "fmt" "io/ioutil" "os"

csiAttacher := getCsiAttacherFromVolumeAttacher(attacher, testWatchTimeout) pluginDir := csiAttacher.plugin.host.GetPluginDir(plug.GetPluginName()) testCases := []struct { <del> testName          string pvName            string expectedMountPath string shouldFail        bool </del> <ins> testName           string pvName             string volumeId           string skipPVCSISource    bool // The test clears PV.Spec.CSI shouldFail         bool addVolSource       bool // The test adds a Volume.VolumeSource.CSI. removeVolumeHandle bool // The test force removes CSI volume handle. </ins> }{ { <del> testName:          "normal test", pvName:            "test-pv1", expectedMountPath: pluginDir + "/pv/test-pv1/globalmount", </del> <ins> testName: "success test", pvName:   "test-pv1", volumeId: "test-vol1", }, { testName:        "fail test, failed to create device mount path due to missing CSI source", pvName:          "test-pv1", volumeId:        "test-vol1", skipPVCSISource: true, shouldFail:      true, }, { testName:     "fail test, failed to create device mount path, CSIVolumeSource found", pvName:       "test-pv1", volumeId:     "test-vol1", addVolSource: true, shouldFail:   true, </ins> }, { <del> testName:          "no pv name", pvName:            "", expectedMountPath: pluginDir + "/pv/test-pv1/globalmount", shouldFail:        true, </del> <ins> testName:           "fail test, failed to create device mount path, missing CSI volume handle", pvName:             "test-pv1", volumeId:           "test-vol1", shouldFail:         true, removeVolumeHandle: true, </ins> }, }

var spec *volume.Spec // Create spec <del> pv := makeTestPV(tc.pvName, 10, testDriver, "testvol") spec = volume.NewSpecFromPersistentVolume(pv, pv.Spec.PersistentVolumeSource.CSI.ReadOnly) </del> <ins> pv := makeTestPV(tc.pvName, 10, testDriver, tc.volumeId) if tc.removeVolumeHandle { pv.Spec.PersistentVolumeSource.CSI.VolumeHandle = "" } if tc.addVolSource { spec = volume.NewSpecFromVolume(makeTestVol(tc.pvName, testDriver)) } else { spec = volume.NewSpecFromPersistentVolume(pv, pv.Spec.PersistentVolumeSource.CSI.ReadOnly) if tc.skipPVCSISource { spec.PersistentVolume.Spec.CSI = nil } } </ins> // Run mountPath, err := csiAttacher.GetDeviceMountPath(spec)

if err != nil && !tc.shouldFail { t.Errorf("test should not fail, but error occurred: %v", err) } else if err == nil { <ins> expectedMountPath := filepath.Join(pluginDir, testDriver, generateSha(tc.volumeId), globalMountInGlobalPath) </ins> if tc.shouldFail { t.Errorf("test should fail, but no error occurred") <del> } else if mountPath != tc.expectedMountPath { t.Errorf("mountPath does not equal expectedMountPath. Got: %s. Expected: %s", mountPath, tc.expectedMountPath) </del> <ins> } else if mountPath != expectedMountPath { t.Errorf("mountPath does not equal expectedMountPath. Got: %s. Expected: %s", mountPath, expectedMountPath) </ins> } } }

shouldFail      bool watchTimeout    time.Duration }{ <ins> // PV agnostic path positive test cases </ins> { <del> testName:        "normal, json file exists", </del> <ins> testName:        "success, json file exists", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        `{"driverName": "csi", "volumeHandle":"project/zone/test-vol1"}`, <del> createPV:        false, </del> stageUnstageSet: true, }, { <del> testName:        "normal, json file doesn't exist -> use PV", </del> <ins> testName:        "stage_unstage not set, PV agnostic path, unmount device is skipped", deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, // PV agnostic path negative test cases { testName:        "fail: missing json, fail to retrieve driver and volumeID from globalpath", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        "", <del> createPV:        true, </del> stageUnstageSet: true, <ins> shouldFail:      true, </ins> }, { <del> testName:        "invalid json ->  use PV", </del> <ins> testName:        "fail: invalid json, fail to retrieve driver and volumeID from globalpath", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        `{"driverName"}}`, <del> createPV:        true, </del> stageUnstageSet: true, <ins> shouldFail:      true, </ins> }, <ins> // Old style PV based path positive test cases </ins> { <del> testName:        "no json, no PV.volID", volID:           "", </del> <ins> testName:        "success: json file doesn't exist, old style pv based global path -> use PV", volID:           "project/zone/test-vol1", </ins> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", <ins> stageUnstageSet: true, </ins> createPV:        true, <del> shouldFail:      true, </del> }, { <del> testName:        "no json, no PV", </del> <ins> testName:        "success: invalid json file, old style pv based global path -> use PV", </ins> volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", <del> jsonFile:        "", createPV:        false, </del> <ins> jsonFile:        `{"driverName"}}`, </ins> stageUnstageSet: true, <del> shouldFail:      true, </del> <ins> createPV:        true, </ins> }, { <del> testName:        "stage_unstage not set no vars should not fail", </del> <ins> testName:        "stage_unstage not set, PV based path, unmount device is skipped", </ins> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, <ins> // Old style PV based path negative test cases { testName:        "fail: json file doesn't exist, old style pv based device path, missing PV", volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", stageUnstageSet: true, shouldFail:      true, }, { testName:        "fail: no json, no PV.volID, old style pv based global path", volID:           "", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", shouldFail:      true, }, </ins> } for _, tc := range testCases {

if tc.deviceMountPath != "" { tc.deviceMountPath = filepath.Join(tmpDir, tc.deviceMountPath) } // Add the volume to NodeStagedVolumes cdc := csiAttacher.csiClient.(*fakeCsiDriverClient) cdc.nodeClient.AddNodeStagedVolume(tc.volID, tc.deviceMountPath, nil) <del> // Make JSON for this object </del> <ins> // Make the device staged path </ins> if tc.deviceMountPath != "" { if err := os.MkdirAll(tc.deviceMountPath, 0755); err != nil { t.Fatalf("error creating directory %s: %s", tc.deviceMountPath, err) } } dir := filepath.Dir(tc.deviceMountPath) <ins> // Make JSON for this object </ins> if tc.jsonFile != "" { dataPath := filepath.Join(dir, volDataFileName) if err := ioutil.WriteFile(dataPath, []byte(tc.jsonFile), 0644); err != nil {

csiAttacher.watchTimeout = watchTimeout return csiAttacher } <ins> func generateSha(handle string) string { result := sha256.Sum256([]byte(fmt.Sprintf("%s", handle))) return fmt.Sprintf("%x", result) } </ins>

} wg.Wait() }) <ins> ginkgo.It("should mount multiple PV pointing to the same storage on the same node", func() { // csi-hostpath driver does not support this test case. In this test case, we have 2 PV containing the same underlying storage. // during the NodeStage call for the second volume, csi-hostpath fails the call, because it thinks the volume is already staged at a different path. // Note: This is not an issue with driver like PD CSI where the NodeStage is a no-op for block mode. if pattern.VolMode == v1.PersistentVolumeBlock { e2eskipper.Skipf("skipping multiple PV mount test for block mode") } init() defer cleanup() l.testCase.PvCheck = func(claim *v1.PersistentVolumeClaim) { MultiplePVMountSingleNodeCheck(l.cs, f.Timeouts, claim, l.config.ClientNodeSelection) } _, clearProvisionedStorageClass := SetupStorageClass(l.testCase.Client, l.testCase.Class) defer clearProvisionedStorageClass() l.testCase.TestDynamicProvisioning() }) </ins> } // SetupStorageClass ensures that a StorageClass from a spec exists, if the StorageClass already exists

return dataSourceRef, cleanupFunc } <ins> // MultiplePVMountSingleNodeCheck checks that multiple PV pointing to the same underlying storage can be mounted simultaneously on a single node. // // Steps: // - Start Pod1 using PVC1, PV1 (which points to a underlying volume v) on node N1. // - Create PVC2, PV2 and prebind them. PV2 points to the same underlying volume v. // - Start Pod2 using PVC2, PV2 (which points to a underlying volume v) on node N1. func MultiplePVMountSingleNodeCheck(client clientset.Interface, timeouts *framework.TimeoutContext, claim *v1.PersistentVolumeClaim, node e2epod.NodeSelection) { pod1Config := e2epod.Config{ NS:            claim.Namespace, NodeSelection: node, PVCs:          []*v1.PersistentVolumeClaim{claim}, } pod1, err := e2epod.CreateSecPodWithNodeSelection(client, &pod1Config, timeouts.PodStart) framework.ExpectNoError(err) defer func() { ginkgo.By(fmt.Sprintf("Deleting Pod %s/%s", pod1.Namespace, pod1.Name)) framework.ExpectNoError(e2epod.DeletePodWithWait(client, pod1)) }() ginkgo.By(fmt.Sprintf("Created Pod %s/%s on node %s", pod1.Namespace, pod1.Name, pod1.Spec.NodeName)) // Create new PV which points to the same underlying storage. Retain policy is used so that deletion of second PVC does not trigger the deletion of its bound PV and underlying storage. e2evolume, err := getBoundPV(client, claim) framework.ExpectNoError(err) pv2Config := e2epv.PersistentVolumeConfig{ NamePrefix:       fmt.Sprintf("%s-", "pv"), StorageClassName: *claim.Spec.StorageClassName, PVSource:         e2evolume.Spec.PersistentVolumeSource, AccessModes:      e2evolume.Spec.AccessModes, VolumeMode:       e2evolume.Spec.VolumeMode, ReclaimPolicy:    v1.PersistentVolumeReclaimRetain, } pvc2Config := e2epv.PersistentVolumeClaimConfig{ NamePrefix:       fmt.Sprintf("%s-", "pvc"), StorageClassName: &claim.Namespace, AccessModes:      e2evolume.Spec.AccessModes, VolumeMode:       e2evolume.Spec.VolumeMode, } pv2, pvc2, err := e2epv.CreatePVCPV(client, timeouts, pv2Config, pvc2Config, claim.Namespace, true) framework.ExpectNoError(err, "PVC, PV creation failed") framework.Logf("Created PVC %s/%s and PV %s in namespace %s", pvc2.Namespace, pvc2.Name, pv2.Name) pod2Config := e2epod.Config{ NS:            pvc2.Namespace, NodeSelection: e2epod.NodeSelection{Name: pod1.Spec.NodeName}, PVCs:          []*v1.PersistentVolumeClaim{pvc2}, } pod2, err := e2epod.CreateSecPodWithNodeSelection(client, &pod2Config, timeouts.PodStart) framework.ExpectNoError(err) ginkgo.By(fmt.Sprintf("Created Pod %s/%s on node %s", pod2.Namespace, pod2.Name, pod2.Spec.NodeName)) ginkgo.By(fmt.Sprintf("Deleting Pod %s/%s", pod2.Namespace, pod2.Name)) framework.ExpectNoError(e2epod.DeletePodWithWait(client, pod2)) err = e2epv.DeletePersistentVolumeClaim(client, pvc2.Name, pvc2.Namespace) framework.ExpectNoError(err, "Failed to delete PVC: %s/%s", pvc2.Namespace, pvc2.Name) err = e2epv.DeletePersistentVolume(client, pv2.Name) framework.ExpectNoError(err, "Failed to delete PV: %s", pv2.Name) } </ins>