Change:
Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host. thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-&lt;id(&lt;id)": timeout expired waiting for volumes to attach/mount for pod "default"/"&lt;id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"&lt;id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?

Discussion:
mutex sync.Mutex <del> // We must cache because otherwise there is a race condition, // where we assign a device mapping and then get a second request before we attach the volume deviceMappings map[mountDevice]string </del> <ins> // We keep an active list of devices we have assigned but not yet // attached, to avoid a race condition where we assign a device mapping // and then get a second request before we attach the volume attaching map[mountDevice]string </ins> } // newAWSInstance creates a new awsInstance object

subnetID:         aws.StringValue(instance.SubnetId), } <del> // We lazy-init deviceMappings self.deviceMappings = nil </del> <ins> self.attaching = make(map[mountDevice]string) </ins> return self }

self.mutex.Lock() defer self.mutex.Unlock() <del> // We cache both for efficiency and correctness if self.deviceMappings == nil { info, err := self.describeInstance() if err != nil { return "", false, err </del> <ins> info, err := self.describeInstance() if err != nil { return "", false, err } deviceMappings := map[mountDevice]string{} for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") { name = name[7:] </ins> } <del> deviceMappings := map[mountDevice]string{} for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") { name = name[7:] } if strings.HasPrefix(name, "/dev/xvd") { name = name[8:] } if len(name) != 1 { glog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } deviceMappings[mountDevice(name)] = aws.StringValue(blockDevice.Ebs.VolumeId) </del> <ins> if strings.HasPrefix(name, "/dev/xvd") { name = name[8:] } if len(name) != 1 { glog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) </ins> } <del> self.deviceMappings = deviceMappings </del> <ins> deviceMappings[mountDevice(name)] = aws.StringValue(blockDevice.Ebs.VolumeId) } for mountDevice, volume := range self.attaching { deviceMappings[mountDevice] = volume </ins> } // Check to see if this volume is already assigned a device on this machine <del> for mountDevice, mappingVolumeID := range self.deviceMappings { </del> <ins> for mountDevice, mappingVolumeID := range deviceMappings { </ins> if volumeID == mappingVolumeID { if assign { glog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID)

valid := instanceType.getEBSMountDevices() chosen := mountDevice("") for _, mountDevice := range valid { <del> _, found := self.deviceMappings[mountDevice] </del> <ins> _, found := deviceMappings[mountDevice] </ins> if !found { chosen = mountDevice break

} if chosen == "" { <del> glog.Warningf("Could not assign a mount device (all in use?).  mappings=%v, valid=%v", self.deviceMappings, valid) </del> <ins> glog.Warningf("Could not assign a mount device (all in use?).  mappings=%v, valid=%v", deviceMappings, valid) </ins> return "", false, nil } <del> self.deviceMappings[chosen] = volumeID </del> <ins> self.attaching[chosen] = volumeID </ins> glog.V(2).Infof("Assigned mount device %s -> volume %s", chosen, volumeID) return chosen, false, nil } <del> func (self *awsInstance) releaseMountDevice(volumeID string, mountDevice mountDevice) { </del> <ins> func (self *awsInstance) endAttaching(volumeID string, mountDevice mountDevice) { </ins> self.mutex.Lock() defer self.mutex.Unlock() <del> existingVolumeID, found := self.deviceMappings[mountDevice] </del> <ins> existingVolumeID, found := self.attaching[mountDevice] </ins> if !found { <del> glog.Errorf("releaseMountDevice on non-allocated device") </del> <ins> glog.Errorf("endAttaching on non-allocated device") </ins> return } if volumeID != existingVolumeID { <del> glog.Errorf("releaseMountDevice on device assigned to different volume") </del> <ins> glog.Errorf("endAttaching on device assigned to different volume") </ins> return } glog.V(2).Infof("Releasing mount device mapping: %s -> volume %s", mountDevice, volumeID) <del> delete(self.deviceMappings, mountDevice) </del> <ins> delete(self.attaching, mountDevice) </ins> } type awsDisk struct {

return volumes[0], nil } <ins> // waitForAttachmentStatus polls until the attachment status is the expected value // TODO(justinsb): return (bool, error) </ins> func (self *awsDisk) waitForAttachmentStatus(status string) error { // TODO: There may be a faster way to get this when we're attaching locally attempt := 0

ec2Device = "/dev/sd" + string(mountDevice) } <del> attached := false </del> <ins> // attachEnded is set to true if the attach operation completed // (successfully or not) attachEnded := false </ins> defer func() { <del> if !attached { awsInstance.releaseMountDevice(disk.awsID, mountDevice) </del> <ins> if attachEnded { awsInstance.endAttaching(disk.awsID, mountDevice) </ins> } }()

attachResponse, err := c.ec2.AttachVolume(request) if err != nil { <ins> attachEnded = true </ins> // TODO: Check if the volume was concurrently attached? return "", fmt.Errorf("Error attaching EBS volume: %v", err) }

return "", err } <del> attached = true </del> <ins> attachEnded = true </ins> return hostDevice, nil }

return "", errors.New("no response from DetachVolume") } <del> // TODO: Fix this - just remove the cache? // If we don't have a cache; we don't have to wait any more (the driver does it for us) // Also, maybe we could get the locally connected drivers from the AWS metadata service? // At this point we are waiting for the volume being detached. This // releases the volume and invalidates the cache even when there is a timeout. // // TODO: A timeout leaves the cache in an inconsistent state. The volume is still // detaching though the cache shows it as ready to be attached again. Subsequent // attach operations will fail. The attach is being retried and eventually // works though. An option would be to completely flush the cache upon timeouts. // defer func() { // TODO: Not thread safe? for mountDevice, existingVolumeID := range awsInstance.deviceMappings { if existingVolumeID == disk.awsID { awsInstance.releaseMountDevice(disk.awsID, mountDevice) return } } }() </del> err = disk.waitForAttachmentStatus("detached") if err != nil { return "", err } <ins> if mountDevice != "" { awsInstance.endAttaching(disk.awsID, mountDevice) } </ins> hostDevicePath := "/dev/xvd" + string(mountDevice) return hostDevicePath, err }