Change:
The test flakes very occasionally. Below are the findings of the circumstances of the flake. During two pods are created in quick succession. The test flakes when the second Pod fails to come up. The second pod fails to come up with error: Upon exploring the logs we discovered that there was only 1 Attach call made at the beginning, and one Detach call upon timeout of the test [See timestamps]: We tracked down the error to the likely culprit in func : based on the error message returned. This is likely caused by (string) being empty [uncomfirmed], which is what the Kubelet would get from the DevicePath field on the Node object. This DevicePath field is normally on Attach and removed on Detach of the volume. The DevicePath is clearly present on the Node when the first pod is being brought up. But it is possible that it is somehow missing or not able to be retrieved by the time the second Pod is starting up (and continues to be unavailable exhausting the 15m timeout on the test) even though no Detach operation was performed on the volume in the meantime (according to logs). We are currently unable to reproduce this flake on command and did all analysis based on logs from the two flakes we saw naturally occur in the 100s of runs performed of the test. This flake was originally seen in and the test logs can be found here: https://k8s- Environment: GCE Kubernetes version (use ):Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ): /kind flake /kind bug /sig storage /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale A check in triage shows that this failing test is still a problem, and in fact has become more of a problem since October 4th. Has there been any progress on a fix?
/priority important-soon
/assign for triage
We should find the root cause and fix for 1.13
/assign Patrick can you PTAL at what is causing flakiness in this test.
I will have a look. Right now the log output of the sidecar containers and the CSI driver are not captured, which makes debugging a bit hard. In PR I am adding code which includes those message in the test log. Let me finish that PR and once it is merged, debugging the next test flake might provide some better insight into what went wrong.
Hey, this issue is considered a blocker for 1.13beta. Is there anything we can do to get it resolved sooner, so that it doesn't delay beta next week?
is trying to instrument the e2e tests to get logs from the csi driver containers so that we can debug. He has tried running the test locally ~100 times and didn't see it flake.
here's one recent one you can check out:
Michelle Au <: I had a look, but something is odd about that one. The error for the second pod is there: But under there is no further information about node bootstrap-e2e-gce-mz-minion-group-3kn6. Shouldn't that be available?
"Patrick Ohly" Here's a better example where the kublet log with the error is available: That one also has the log of the external-attacher, but that doesn't tell me much. I'm now comparing a "bad" run with a "good" local run. One observation is that in the "good" run, operationExecutor.UnmountDevice is completed before the second usage, including the "Volume detached for volume" step in Then operationExecutor.VerifyControllerAttachedVolume is done again for the second pod, exactly as it was done for the first: In the "bad" run, operationExecutor.MountVolume is invoked before that "Volume detached for volume" and then runs with an empty device path, without going through VerifyControllerAttachedVolume again: So to me it looks like a race between two conflicting operations. Does that make sense? How to fix it is less obvious. The relevant code seems to be this in Any suggestions?
This seems inline with the observations that made initially. We should trace where DevicePath is getting set. Could there be a race with AD controller or external attacher if they try to detach the volume? is another race around this area. Unclear if it's the same yet.
Michelle Au <: No idea. Even if it isn't, it's at least related to the same piece of code. Probably worth fixing both together. Can we get someone who has worked in this area to fix these issues? Today is the first day that I've ever seen this code, so I'm a bit afraid that it'll take me a long time to get it right, and I'm worried about regressions.
I've asked to help investigate
It is not a blocker for beta or freeze. Can be addressed in stabilization. Let's keep the 1.13 tag on it. And based on progress by stabilization we can reassess if it is even a 1.13 blocker (at this point I don't believe it is).
The in the actual state of the world gets reset during unmounting:
That removes the device path looks wrong to me. I've submitted PR which changes that. Let's see how much that breaks...
/assign The conclusion in the CSI working group was that while my analysis and the resulting fix might be correct, further debugging and a second pair of eyes would be good. Let me reassign, because I have done what I could for this issue.
The RT considers this a blocker now, because flakes are happening often enough to interfere with getting clear signal on several jobs. Even if the underlying problem isn't that serious.
/assign I will help triage as well.
Ack Thanks
Instead of which may have negative side effects for non-CSI volumes, let's do this instead: Modify the CSI method in to ignore the passed in (which is populated from devicePath. Instead let's have that method generate the from scratch, e.g. attachID := getAttachmentName(volID, driverName, string(nodeName)) This will remove any CSI dependency on devicePath and not effect other volume plugins.
I am trying to twist my head around the code. IMO it affects also AWS that uses devicePath stored in and it should be reproducible there too. So far I was unsuccessful. In the logs it's clearly visible that UnmountDevice succeeded. So why kubelet continues with MountVolume instead of MountDevice? That's wrong. A workaround in CSI volume plugin won't really help here, it will work only for drivers without .
I finally reproduced it on AWS with a long sleep in : Steps to reproduce (with the hack above): a pod with a volume the pod. Wait until is called. a new pod with the same volume. - checks that the volume is still attached and the device is still mounted and enqueues an operation to mount the volume. When finishes, starts and 1) uses empty and 2) calls on volume that's not DeviceMounted.
Fix: Any idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one.
I'm going to work on a targeted fix for CSI. We can continue to pursue fixing aws in-tree separately.
PR to fix csi is here:
what are the final set of PRs pending for this? , and ?
Only is needed to fix this particular test failure.
can you please confirm if we are going to fix this using or ?
I discussed with offline, he has some suggestions for that I am investigating. I think we will want both PRs eventually, although only one of them will be needed to fix the CSI test for now. Since it's his off hours, I will continue working on today.
Are we talking 1.13 for as well?
Reopening this since we are seeing consistent failures in latest runs. We also see one more test fail in the job with same error. Failing job https://k8s- Failing Tests [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: gcePD [Serial] should provision storage [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: hostPath [Serial] should provision storage Error This is now 1.13 release blocking.
/priority critical-urgent /kind failing-test
/remove-priority important-soon
I think this is a different issue and could be due to the fact that CSI is intentionally not backwards compatible between 1.12 and 1.13. We may need to enhance the test to figure out what version the cluster is on and launch appropriate versions of the CSI driver.
"intentionally not backwards compatible" -- is there a writeup for the release notes and release announcement for this change?
Yes has the release note
thanks, I'll also call it to the Release Notes lead's attention
what does it mean from the user impact perspective for CSI to not be backward compatible with 1.12? can you please point me to any discussions on this behavior for 1.13?
Kubernetes v1.13 speaks CSI v1.0 and no longer understands CSI v0.x. Any CSI drivers deployed on previous versions of Kubernetes speaking CSI v0.x must removed, updated to be CSI 1.0 compatible, and redeployed on the on Kubernetes v1.13. If only the master is upgraded to 1.13 (and nodes are 1.12 or earlier) older drivers will continue to work. When nodes are upgraded to 1.13, then older drivers will stop working. I will capture this in detail in the "Action Required" section.
Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by and for non CSI /close
Closing this issue. <detailsIn response to : Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as FYI to call this out clearly in the Release notes.

Discussion:
klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

} } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {