Issue 1:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): GKE, GCE, volume timeout

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GKE

OS (e.g. from /etc/os-release): GCI

Kernel (e.g. uname -a): whatever GCI uses

Install tools:

Others:

What happened: Pods cannot start/restart because the volume will not mount. The volume cannot mount indefinitely (I have had pods in a ContainerCreating state for weeks in some cases).

21s 2m 2 camlistored-2824072653-3p1t7 Pod Warning FailedMount {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d} Unable to mount volumes for pod "camlistored-2824072653-3p1t7_prod(bd1cbbe3-b380-11e6-9d1b-42010af000ba)": timeout expired waiting for volumes to attach/mount for pod "camlistored-2824072653-3p1t7"/"prod". list of unattached/unmounted volumes=[camlistore-blobs]
21s 2m 2 camlistored-2824072653-3p1t7 Pod Warning FailedSync {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d} Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "camlistored-2824072653-3p1t7"/"prod". list of unattached/unmounted volumes=[camlistore-blobs]

I sometimes notice other errors when errors first start occurring. Here is one example:

15s 4m 11 camlistored-2824072653-3p1t7 Pod Warning FailedMount {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d} MountVolume.SetUp failed for volume "kubernetes.io/gce-pd/camlistore-blobs-prod" (spec.Name: "camlistore-blobs-prod") pod "bd1cbbe3-b380-11e6-9d1b-42010af000ba" (UID: "bd1cbbe3-b380-11e6-9d1b-42010af000ba") with: mount failed: exit status 32
Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod /var/lib/kubelet/pods/bd1cbbe3-b380-11e6-9d1b-42010af000ba/volumes/kubernetes.io~gce-pd/camlistore-blobs-prod [remount]
Output: mount: cannot remount /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod read-write, is write-protected

Deleting and recreating a deployment sometimes works to allow me to remount the volume but it's rare.

What you expected to happen: I expect that volumes will be able to be re-mounted to pods within a reasonable time period. e.g. a few minutes.

How to reproduce it (as minimally and precisely as possible):

I don't know the specific circumstances that this happens but I notice that this happens often when a container with the volume attached crashes. The container will crash and cannot be restarted because the volume will not mount.

Anything else do we need to know:

This happens very frequently for certain types of pods that use the disk often like camlistore or MySQL when the pod is stopped and restarted, via deleting the pod, cluster upgrades etc. It's extremely frustrating.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

Environment:

Cloud provider or hardware configuration:

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools:

Others:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know: