Code change:
for mountDevice, mappingVolumeID := range deviceMappings { if volumeID == mappingVolumeID { if assign { <del> klog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID) </del> <ins> // DescribeInstances shows the volume as attached / detaching, while Kubernetes // cloud provider thinks it's detached. // This can happened when the volume has just been detached from the same node // and AWS API returns stale data in this DescribeInstances ("eventual consistency"). // Fail the attachment and let A/D controller retry in a while, hoping that // AWS API returns consistent result next time (i.e. the volume is detached). status := volumeStatus[mappingVolumeID] klog.Warningf("Got assignment call for already-assigned volume: %s@%s, volume status: %s", mountDevice, mappingVolumeID, status) return mountDevice, false, fmt.Errorf("volume is still being detached from the node") </ins> } return mountDevice, true, nil }

Related discussion:
text: (Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.