Change:
self.mutex.Lock() defer self.mutex.Unlock() <del> // We cache both for efficiency and correctness if self.deviceMappings == nil { info, err := self.describeInstance() if err != nil { return "", false, err </del> <ins> info, err := self.describeInstance() if err != nil { return "", false, err } deviceMappings := map[mountDevice]string{} for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") { name = name[7:] </ins> } <del> deviceMappings := map[mountDevice]string{} for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") { name = name[7:] } if strings.HasPrefix(name, "/dev/xvd") { name = name[8:] } if len(name) != 1 { glog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } deviceMappings[mountDevice(name)] = aws.StringValue(blockDevice.Ebs.VolumeId) </del> <ins> if strings.HasPrefix(name, "/dev/xvd") { name = name[8:] } if len(name) != 1 { glog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) </ins> } <del> self.deviceMappings = deviceMappings </del> <ins> deviceMappings[mountDevice(name)] = aws.StringValue(blockDevice.Ebs.VolumeId) } for mountDevice, volume := range self.attaching { deviceMappings[mountDevice] = volume </ins> } // Check to see if this volume is already assigned a device on this machine <del> for mountDevice, mappingVolumeID := range self.deviceMappings { </del> <ins> for mountDevice, mappingVolumeID := range deviceMappings { </ins> if volumeID == mappingVolumeID { if assign { glog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID)

Discussion:
Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-&lt;id(&lt;id)": timeout expired waiting for volumes to attach/mount for pod "default"/"&lt;id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"&lt;id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?