Code change:
} } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

Related discussion:
text: A workaround in CSI volume plugin won't really help here, it will work only for drivers without .
I finally reproduced it on AWS with a long sleep in : Steps to reproduce (with the hack above): a pod with a volume the pod. Wait until is called. a new pod with the same volume. - checks that the volume is still attached and the device is still mounted and enqueues an operation to mount the volume. When finishes, starts and 1) uses empty and 2) calls on volume that's not DeviceMounted.
Fix: Any idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one.
I'm going to work on a targeted fix for CSI. We can continue to pursue fixing aws in-tree separately.
PR to fix csi is here:
what are the final set of PRs pending for this? , and ?
Only is needed to fix this particular test failure.
can you please confirm if we are going to fix this using or ?
I discussed with offline, he has some suggestions for that I am investigating. I think we will want both PRs eventually, although only one of them will be needed to fix the CSI test for now. Since it's his off hours, I will continue working on today.
Are we talking 1.13 for as well?
Reopening this since we are seeing consistent failures in latest runs. We also see one more test fail in the job with same error. Failing job https://k8s- Failing Tests [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: gcePD [Serial] should provision storage [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: hostPath [Serial] should provision storage Error This is now 1.13 release blocking.
/priority critical-urgent /kind failing-test
/remove-priority important-soon
I think this is a different issue and could be due to the fact that CSI is intentionally not backwards compatible between 1.12 and 1.13. We may need to enhance the test to figure out what version the cluster is on and launch appropriate versions of the CSI driver.
"intentionally not backwards compatible" -- is there a writeup for the release notes and release announcement for this change?