Change:
} volumeName := KubernetesVolumeID("aws://" + aws.StringValue(response.AvailabilityZone) + "/" + string(awsID)) <del> // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if len(volumeOptions.KmsKeyID) > 0 { err := c.waitUntilVolumeAvailable(volumeName) if err != nil { if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") } return "", err </del> <ins> err = c.waitUntilVolumeAvailable(volumeName) if err != nil { // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") </ins> } <ins> return "", err </ins> } return volumeName, nil

Discussion:
What happened: Kubernetes tries to attach EBS volumes while their status is still : I believe this started to happen after , when volumes and their tags started to be created in the same request. Before that, tags would be set through another call (i.e., ), which retries if the volume isn't . What you expected to happen: Kubernetes should try to attach the volume only when it's available. In order to do that, should make sure the volume is when it returns. How to reproduce it (as minimally and precisely as possible): Just create a dynamically provisioned EBS volume. /kind bug /sig storage /sig aws
CC
What is the downside of this? IIRC - even when we tagged separately, we used to have few cases where by the time attachment was attempted, volume was not available. May be tagging while creating the volume makes it worse but someone has to wait for volume to become available - either attach call or create volume call itself.
In worse case this causes first attach to fail - but controller will retry anyway.
I think it re-introduces
No - looking at events attached to the pod, it looks like volume eventually got attached. I think is unrelated.
It is related. If you create volume with invalid keys, AWS returns you a volume and in this case Kubernetes will create a PV for it. But then the volume silently disappears. The only way how to detect that is to wait for the volume to either get Available or disappear.
we are still waiting for volume to become available if volume has kms key -
+1 on this issue. I have seen this quite often on my v1.12 cluster too. So this is an issue even before
we are still waiting for volume to become available if volume has kms key - Oh, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
EBS CSI driver is already regardless of whether kms key is used or not. Any objections to do the same for in-tree?
I like it.