Change:
E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume. (Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case. So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached. What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.

Discussion:
"github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/klog" "k8s.io/apimachinery/pkg/api/resource"

"k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

assign bool) (assigned mountDevice, alreadyAttached bool, err error) { deviceMappings := map[mountDevice]EBSVolumeID{} <ins> volumeStatus := map[EBSVolumeID]string{} // for better logging of volume status </ins> for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") {

if len(name) < 1 || len(name) > 2 { klog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } <ins> if blockDevice.Ebs != nil && blockDevice.Ebs.VolumeId != nil { volumeStatus[EBSVolumeID(*blockDevice.Ebs.VolumeId)] = aws.StringValue(blockDevice.Ebs.Status) } </ins> deviceMappings[mountDevice(name)] = EBSVolumeID(aws.StringValue(blockDevice.Ebs.VolumeId)) }

for mountDevice, mappingVolumeID := range deviceMappings { if volumeID == mappingVolumeID { if assign { <del> klog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID) </del> <ins> // DescribeInstances shows the volume as attached / detaching, while Kubernetes // cloud provider thinks it's detached. // This can happened when the volume has just been detached from the same node // and AWS API returns stale data in this DescribeInstances ("eventual consistency"). // Fail the attachment and let A/D controller retry in a while, hoping that // AWS API returns consistent result next time (i.e. the volume is detached). status := volumeStatus[mappingVolumeID] klog.Warningf("Got assignment call for already-assigned volume: %s@%s, volume status: %s", mountDevice, mappingVolumeID, status) return mountDevice, false, fmt.Errorf("volume is still being detached from the node") </ins> } return mountDevice, true, nil }