Code change:
} } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

Related discussion:
text: /priority important-soon
/assign for triage
We should find the root cause and fix for 1.13
/assign Patrick can you PTAL at what is causing flakiness in this test.
I will have a look. Right now the log output of the sidecar containers and the CSI driver are not captured, which makes debugging a bit hard. In PR I am adding code which includes those message in the test log. Let me finish that PR and once it is merged, debugging the next test flake might provide some better insight into what went wrong.
Hey, this issue is considered a blocker for 1.13beta. Is there anything we can do to get it resolved sooner, so that it doesn't delay beta next week?
is trying to instrument the e2e tests to get logs from the csi driver containers so that we can debug. He has tried running the test locally ~100 times and didn't see it flake.
here's one recent one you can check out:
Michelle Au <: I had a look, but something is odd about that one. The error for the second pod is there: But under there is no further information about node bootstrap-e2e-gce-mz-minion-group-3kn6. Shouldn't that be available?
"Patrick Ohly" Here's a better example where the kublet log with the error is available: That one also has the log of the external-attacher, but that doesn't tell me much. I'm now comparing a "bad" run with a "good" local run. One observation is that in the "good" run, operationExecutor.UnmountDevice is completed before the second usage, including the "Volume detached for volume" step in Then operationExecutor.VerifyControllerAttachedVolume is done again for the second pod, exactly as it was done for the first: In the "bad" run, operationExecutor.MountVolume is invoked before that "Volume detached for volume" and then runs with an empty device path, without going through VerifyControllerAttachedVolume again: So to me it looks like a race between two conflicting operations. Does that make sense? How to fix it is less obvious. The relevant code seems to be this in Any suggestions?
This seems inline with the observations that made initially. We should trace where DevicePath is getting set.