Code change:
This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events. Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV. And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available". I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:

Related discussion:
text: return "", fmt.Errorf("error tagging volume %s: %q", volumeName, err) } <ins> // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if len(volumeOptions.KmsKeyId) > 0 { err := c.waitUntilVolumeAvailable(volumeName) if err != nil { if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") } return "", err } } </ins> return volumeName, nil } <del> // checkEncryptionKey tests that given encryption key exists. func (c *Cloud) checkEncryptionKey(keyId string) (missing bool, err error) { input := &kms.DescribeKeyInput{ KeyId: aws.String(keyId), } _, err = c.kms.DescribeKey(input) if err == nil { return false, nil </del> <ins> func (c *Cloud) waitUntilVolumeAvailable(volumeName KubernetesVolumeID) error { disk, err := newAWSDisk(c, volumeName) if err != nil { // Unreachable code return err </ins> } <del> if awsError, ok := err.(awserr.Error); ok { if awsError.Code() == "NotFoundException" { return true, fmt.Errorf("KMS key %s not found: %q", keyId, err) </del> <ins> err = wait.Poll(encryptedCheckInterval, encryptedCheckTimeout, func() (done bool, err error) { vol, err := disk.describeVolume() if err != nil { return true, err </ins> } <del> } return false, fmt.Errorf("Error checking KSM key %s: %q", keyId, err) </del> <ins> if vol.State != nil { switch *vol.State { case "available": // The volume is Available, it won't be deleted now. return true, nil case "creating": return false, nil default: return true, fmt.Errorf("unexpected State of newly created AWS EBS volume %s: %q", volumeName, *vol.State) } } return false, nil }) return err </ins> } // DeleteDisk implements Volumes.DeleteDisk