Issue 1:
I'm using Kube 1.4.5, and AWS storage.

When I try to attach multiple volumes using PVC's, one of the volumes consistently gets stuck in the attaching state whilst the other is successful.

Below are the definitions that I used.

sonar-persistence.yml

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: sonarqube-data
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
volumeID: aws://eu-west-1a/vol-XXXXXXX
fsType: ext4

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: sonarqube-extensions
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
volumeID: aws://eu-west-1a/vol-XXXXXX
fsType: ext4

sonar-claim.yml

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: sonarqube-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: sonarqube-extensions
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi

sonar-deployment.yml

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sonar
spec:
replicas: 1
template:
metadata:
name: sonar
labels:
name: sonar
spec:
containers:
- image: sonarqube:lts
args:
- -Dsonar.web.context=/sonar
name: sonar
env:
- name: SONARQUBE_JDBC_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-pwd
key: password
- name: SONARQUBE_JDBC_URL
value: jdbc:postgresql://sonar:5432/sonar
ports:
- containerPort: 9000
name: sonar
volumeMounts:
- name: sonarqube-data
mountPath: /opt/sonarqube/data
- name: sonarqube-extensions
mountPath: /opt/sonarqube/extensions
volumes:
- name: sonarqube-data
persistentVolumeClaim:
claimName: sonarqube-data
- name: sonarqube-extensions
persistentVolumeClaim:
claimName: sonarqube-extensions

The data volume always appears to be successful, and maybe coincidentally is first in the list. I have tried this multiple times but the result is always the same.

The error message is as follows:

Unable to mount volumes for pod "sonar-3504269494-tnzwo_default(2cc5292c-a5d4-11e6-bd99-0a82a8a86ebf)": timeout expired waiting for volumes to attach/mount for pod "sonar-3504269494-tnzwo"/"default". list of unattached/unmounted volumes=[sonarqube-data sonarqube-extensions]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "sonar-3504269494-tnzwo"/"default". list of unattached/unmounted volumes=[sonarqube-data sonarqube-extensions]

Issue 2:
Kubernetes version:

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6+coreos.0", GitCommit:"f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd", GitTreeState:"clean", BuildDate:"2016-08-29T17:01:01Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: AWS

OS: CoreOS

Kernel: 4.6.3-coreos

Install tools: kube-aws

What happened:

Some EBS volumes remain in the attachment state "attaching". These EBS volumes are attached on a device name that was used for a previous attachment.

Looking on the worker node, the volume is not visible in the device list (eg by running lsblk). After a reboot of the node the volume is attached successfully. The attachment state then is "attached".

Expected behavior:

The EBS volumes are attached correctly.

How to reproduce it:

Spin up a worker node

Create 20 EBS volumes

Create 19 pods with volumes attached on this node

Remove the first pod

Create a 20th pod with the 20th volume attached

Now the 20th volume remains in the attachment state 'attaching'.

20 is not necessary the lower bound but the issue did not appear with 3 volumes.

Additional information:

It seems related to the reuse of device names. Only when the device name /dev/xvdXX was already used for a previous attachment, the volume remains in the attachment state "attaching".

When using a lower number of volumes, it is still possible to detach-attach. Only when attaching a considerable amount of volumes (about 20), the issue appears.

According to this AWS support article it is possible that the device name was not released by the block device driver after detaching.