Code change:
We observed in an e2e that if attach is performed on a volume that was detached before from same node, sometimes the mount of the volume (for 2nd pod) could be stuck waiting on invalid devicePath in call. The problem is - when attach for second pod failed with this caused an dangling volume error from AWS cloudprovider(implemented by us). The dangling volume error is supposed to auto-correct any volumes which are attached to a node that it shouldn't be. But in this case - the dangling volume error was triggered by the fact that, volume was not actually attached to the node but because call for 2nd attachment returned stale data (because of AWS eventual consistency). So, dangling volume exception caused volume to be to actual state of world (so as it can be detached later), but this caused volume to be reported to node's Status field. And that caused second pod to assume volume as attached to invalid device path and hence 2nd pod was stuck for 10 minutes waiting for volume to attach on a device path (/dev/xvdbl) which does not exist.
/sig storage /sig aws cc
Here is k8s logs that confirm that, did return stale data for volume: See detach succeeds at 19.30.25: and yet: so second DescribeVolume API call is actually returning stale data.
There are couple of ways of fixing this: error is used by AWS and Openstack cloudproviders to fix incorrectly attached volumes. We could fix mechanism of dangling volumes, so as when it adds volumes to actual state of the world - it does so without reporting the volume to node's Status. can make a local fix inside AWS so that when we return error, before returning that error we also verify volume attachment by querying the instance. We already have instance information in that call. Verifying volume attachment by and should reduce likelyhood of returning incorrect errors. There is still a chance though that, both API calls would land on a AWS backend which does not yet have updated information.
expanding on fix - I am thinking when we add a volume to actual state of the world via DanglingVolume error, then we should add the volume as "uncertain". We will update the code to not report "uncertain" volumes in node's status and hence in this case volume state will be fixed correctly. If I understand correctly - did not intend to report "uncertain" volumes to node's status anyways.
This looks like another symptom of . I tried that, it does not always help. Both and may return the same stale data, see IMO, is the right way to go. Unless it break something else, all these AWS corner cases were complicated already and now anything can break.
/cc

Related discussion:
text: volumeToAttach.VolumeSpec, volumeToAttach.NodeName) if attachErr != nil { <ins> uncertainNode := volumeToAttach.NodeName </ins> if derr, ok := attachErr.(*volerr.DanglingAttachError); ok { <del> addErr := actualStateOfWorld.MarkVolumeAsAttached( v1.UniqueVolumeName(""), originalSpec, derr.CurrentNode, derr.DevicePath) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsAttached failed to fix dangling volume error for volume %q with %s", volumeToAttach.VolumeName, addErr) } } else { addErr := actualStateOfWorld.MarkVolumeAsUncertain( v1.UniqueVolumeName(""), originalSpec, volumeToAttach.NodeName) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsUncertain fail to add the volume %q to actual state with %s", volumeToAttach.VolumeName, addErr) } </del> <ins> uncertainNode = derr.CurrentNode } addErr := actualStateOfWorld.MarkVolumeAsUncertain( v1.UniqueVolumeName(""), originalSpec, uncertainNode) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsUncertain fail to add the volume %q to actual state with %s", volumeToAttach.VolumeName, addErr) </ins> } // On failure, return error. Caller will log and retry. return volumeToAttach.GenerateError("AttachVolume.Attach failed", attachErr) }