Issue 1:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

timeout expired waiting for volumes to attach/mount gcePersistentDisk readOnly

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Bug report

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"windows/amd64"}

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:38:31Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GKE

OS (e.g. from /etc/os-release):

PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"

Kernel (e.g. uname -a): Linux gke-fatih-small-pool-59881027-k909 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

Install tools:

Others:

What happened:

readOnly persistent disk won't mount multiple times

What you expected to happen:

Can be mounted multiple times

How to reproduce it (as minimally and precisely as possible):

Create 3 RCs (with 1 replica each) that mount the same persistent disk as read only.

The first pod always succeeds mounting.

kubectl get po
NAME READY STATUS RESTARTS AGE
mongo-express 1/1 Running 0 2h
mongo-rc0-rpah5 1/1 Running 0 30m
mongo-rc1-xkcyy 1/1 Running 0 17m
mongo-rc2-aktox 0/1 ContainerCreating 0 30m

kubectl describe po mongo-rc0-rpah5
Name: mongo-rc0-rpah5
Namespace: default
Node: gke-fatih-small-pool-59881027-k909/10.142.0.5
Start Time: Tue, 23 Aug 2016 05:01:48 +0700
Labels: instance=fatih0
name=mongo-node0
Status: Running
IP: 10.60.2.5
Controllers: ReplicationController/mongo-rc0
Containers:
mongo-node0:
Container ID: docker://2845af5d815ff62505284ee8ef22bb5be3fa7f276e00d9347c9ce0e726e45d2f
Image: mongo
Image ID: docker://sha256:af52553e1c34b3ec48a2e50cf73a1eed1fc6d2fd2b0d3d73d7397c8d6341551f
Port: 27017/TCP
Command:
mongod
--replSet
bippo
--storageEngine
wiredTiger
--keyFile
/etc/mongo-conf/mongo.keyfile
Requests:
cpu: 100m
State: Running
Started: Tue, 23 Aug 2016 05:02:06 +0700
Ready: True
Restart Count: 0
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
mongo-conf:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: mongo-conf
FSType: ext4
Partition: 0
ReadOnly: true
mongo-persistent-storage0:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: mongodb-disk0
FSType: ext4
Partition: 0
ReadOnly: false
default-token-i0lox:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-i0lox
QoS Tier: Burstable
Events:
FirstSeen LastSeen Count From SubobjectPath
Type Reason Message
--------- -------- ----- ---- -------------
-------- ------ -------
33m 33m 1 {default-scheduler } Normal
Scheduled Successfully assigned mongo-rc0-rpah5 to gke-fatih-small-pool-59881027-k909
33m 33m 1 {kubelet gke-fatih-small-pool-59881027-k909} spec.containers{mongo-node0}
Normal Pulling pulling image "mongo"
33m 33m 1 {kubelet gke-fatih-small-pool-59881027-k909} spec.containers{mongo-node0}
Normal Pulled Successfully pulled image "mongo"
33m 33m 1 {kubelet gke-fatih-small-pool-59881027-k909} spec.containers{mongo-node0}
Normal Created Created container with docker id 2845af5d815f
33m 33m 1 {kubelet gke-fatih-small-pool-59881027-k909} spec.containers{mongo-node0}
Normal Started Started container with docker id 2845af5d815f

The second pod is intermittent. The third pod (below) always fails.

kubectl describe po mongo-rc2-aktox
Name: mongo-rc2-aktox
Namespace: default
Node: gke-fatih-small-pool-59881027-h46g/10.142.0.6
Start Time: Tue, 23 Aug 2016 05:01:49 +0700
Labels: instance=fatih2
name=mongo-node2
Status: Pending
IP:
Controllers: ReplicationController/mongo-rc2
Containers:
mongo-node2:
Container ID:
Image: mongo
Image ID:
Port: 27017/TCP
Command:
mongod
--replSet
bippo
--storageEngine
wiredTiger
--keyFile
/etc/mongo-conf/mongo.keyfile
Requests:
cpu: 100m
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
mongo-conf:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: mongo-conf
FSType: ext4
Partition: 0
ReadOnly: true
mongo-persistent-storage2:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: mongodb-disk2
FSType: ext4
Partition: 0
ReadOnly: false
default-token-i0lox:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-i0lox
QoS Tier: Burstable
Events:
FirstSeen LastSeen Count From SubobjectPath Type
Reason Message
--------- -------- ----- ---- ------------- --------
------ -------
30m 30m 1 {default-scheduler } Normal
Scheduled Successfully assigned mongo-rc2-aktox to gke-fatih-small-pool-59881027-h46g
28m 1m 13 {kubelet gke-fatih-small-pool-59881027-h46g} Warning
FailedMount Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
28m 1m 13 {kubelet gke-fatih-small-pool-59881027-h46g} Warning
FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]

Anything else do we need to know:

Relevant /var/log/kubelet.log in the node:

ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n100 /var/log/kubelet.log | grep mongo-conf

E0822 22:30:30.157423 3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:30:30.157465 3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:31:05.117374 3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:31:05.120603 3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:33:05.120584643 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:32:45.156181 3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:32:45.156226 3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:33:05.156100 3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:33:05.159285 3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:35:05.159269631 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:34:58.157744 3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:34:58.161069 3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:35:05.188747 3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:35:05.191576 3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:37:05.191557765 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.

/var/log/kubelet.log for 50 last lines:

ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log
I0822 22:35:22.870417 3439 server.go:959] GET /healthz: (28.36µs) 200 [[curl/7.26.0] 127.0.0.1:47687]
I0822 22:35:32.239049 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:35:32.242781 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:35:32.883593 3439 server.go:959] GET /healthz: (42.657µs) 200 [[curl/7.26.0] 127.0.0.1:47694]
I0822 22:35:38.160222 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:35:38.163717 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:35:38.537711 3439 server.go:959] GET /healthz: (44.537µs) 200 [[Go-http-client/1.1] 127.0.0.1:47701]
I0822 22:35:39.660496 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:35:42.896284 3439 server.go:959] GET /healthz: (38.658µs) 200 [[curl/7.26.0] 127.0.0.1:47705]
I0822 22:35:52.910178 3439 server.go:959] GET /healthz: (25.065µs) 200 [[curl/7.26.0] 127.0.0.1:47712]
I0822 22:36:02.922869 3439 server.go:959] GET /healthz: (24.963µs) 200 [[curl/7.26.0] 127.0.0.1:47717]
I0822 22:36:05.013580 3439 server.go:959] GET /stats/summary/: (6.953162ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:36:12.935348 3439 server.go:959] GET /healthz: (27.682µs) 200 [[curl/7.26.0] 127.0.0.1:47724]
I0822 22:36:22.948215 3439 server.go:959] GET /healthz: (28.114µs) 200 [[curl/7.26.0] 127.0.0.1:47731]
I0822 22:36:32.961851 3439 server.go:959] GET /healthz: (44.817µs) 200 [[curl/7.26.0] 127.0.0.1:47738]
I0822 22:36:38.537762 3439 server.go:959] GET /healthz: (47.175µs) 200 [[Go-http-client/1.1] 127.0.0.1:47743]
I0822 22:36:39.176725 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:39.189075 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:36:39.663850 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:36:42.974584 3439 server.go:959] GET /healthz: (40.231µs) 200 [[curl/7.26.0] 127.0.0.1:47746]
I0822 22:36:43.184638 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:43.187674 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:36:52.988636 3439 server.go:959] GET /healthz: (40.953µs) 200 [[curl/7.26.0] 127.0.0.1:47754]
I0822 22:37:03.003414 3439 server.go:959] GET /healthz: (42.733µs) 200 [[curl/7.26.0] 127.0.0.1:47759]
I0822 22:37:05.034193 3439 server.go:959] GET /stats/summary/: (6.947947ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:37:05.233719 3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:37:05.236650 3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:39:05.23662061 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:37:09.156182 3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:37:09.156242 3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:37:13.015933 3439 server.go:959] GET /healthz: (54.514µs) 200 [[curl/7.26.0] 127.0.0.1:47768]
I0822 22:37:23.028881 3439 server.go:959] GET /healthz: (43.187µs) 200 [[curl/7.26.0] 127.0.0.1:47775]
I0822 22:37:23.185043 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:37:23.188665 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e").
I0822 22:37:33.042085 3439 server.go:959] GET /healthz: (42.351µs) 200 [[curl/7.26.0] 127.0.0.1:47784]
I0822 22:37:38.537583 3439 server.go:959] GET /healthz: (40.577µs) 200 [[Go-http-client/1.1] 127.0.0.1:47789]
I0822 22:37:39.667127 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:37:43.055076 3439 server.go:959] GET /healthz: (26.24µs) 200 [[curl/7.26.0] 127.0.0.1:47792]
I0822 22:37:53.067791 3439 server.go:959] GET /healthz: (26.387µs) 200 [[curl/7.26.0] 127.0.0.1:47800]
I0822 22:38:03.089970 3439 server.go:959] GET /healthz: (38.445µs) 200 [[curl/7.26.0] 127.0.0.1:47805]
I0822 22:38:05.013454 3439 server.go:959] GET /stats/summary/: (7.716171ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:38:06.174617 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:06.177284 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:38:11.184431 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:11.187573 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:38:13.101813 3439 server.go:959] GET /healthz: (35.796µs) 200 [[curl/7.26.0] 127.0.0.1:47814]
I0822 22:38:23.122472 3439 server.go:959] GET /healthz: (36.311µs) 200 [[curl/7.26.0] 127.0.0.1:47821]
I0822 22:38:33.136366 3439 server.go:959] GET /healthz: (40.312µs) 200 [[curl/7.26.0] 127.0.0.1:47828]
I0822 22:38:38.537838 3439 server.go:959] GET /healthz: (44.226µs) 200 [[Go-http-client/1.1] 127.0.0.1:47833]
I0822 22:38:39.670329 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:38:43.157960 3439 server.go:959] GET /healthz: (34.945µs) 200 [[curl/7.26.0] 127.0.0.1:47836]
ceefour@gke-fatih-small-pool-59881027-h46g:~$
ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log
I0822 22:35:38.537711 3439 server.go:959] GET /healthz: (44.537µs) 200 [[Go-http-client/1.1] 127.0.0.1:47701]
I0822 22:35:39.660496 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:35:42.896284 3439 server.go:959] GET /healthz: (38.658µs) 200 [[curl/7.26.0] 127.0.0.1:47705]
I0822 22:35:52.910178 3439 server.go:959] GET /healthz: (25.065µs) 200 [[curl/7.26.0] 127.0.0.1:47712]
I0822 22:36:02.922869 3439 server.go:959] GET /healthz: (24.963µs) 200 [[curl/7.26.0] 127.0.0.1:47717]
I0822 22:36:05.013580 3439 server.go:959] GET /stats/summary/: (6.953162ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:36:12.935348 3439 server.go:959] GET /healthz: (27.682µs) 200 [[curl/7.26.0] 127.0.0.1:47724]
I0822 22:36:22.948215 3439 server.go:959] GET /healthz: (28.114µs) 200 [[curl/7.26.0] 127.0.0.1:47731]
I0822 22:36:32.961851 3439 server.go:959] GET /healthz: (44.817µs) 200 [[curl/7.26.0] 127.0.0.1:47738]
I0822 22:36:38.537762 3439 server.go:959] GET /healthz: (47.175µs) 200 [[Go-http-client/1.1] 127.0.0.1:47743]
I0822 22:36:39.176725 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:39.189075 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:36:39.663850 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:36:42.974584 3439 server.go:959] GET /healthz: (40.231µs) 200 [[curl/7.26.0] 127.0.0.1:47746]
I0822 22:36:43.184638 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:43.187674 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:36:52.988636 3439 server.go:959] GET /healthz: (40.953µs) 200 [[curl/7.26.0] 127.0.0.1:47754]
I0822 22:37:03.003414 3439 server.go:959] GET /healthz: (42.733µs) 200 [[curl/7.26.0] 127.0.0.1:47759]
I0822 22:37:05.034193 3439 server.go:959] GET /stats/summary/: (6.947947ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:37:05.233719 3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:37:05.236650 3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:39:05.23662061 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:37:09.156182 3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:37:09.156242 3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:37:13.015933 3439 server.go:959] GET /healthz: (54.514µs) 200 [[curl/7.26.0] 127.0.0.1:47768]
I0822 22:37:23.028881 3439 server.go:959] GET /healthz: (43.187µs) 200 [[curl/7.26.0] 127.0.0.1:47775]
I0822 22:37:23.185043 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:37:23.188665 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e").
I0822 22:37:33.042085 3439 server.go:959] GET /healthz: (42.351µs) 200 [[curl/7.26.0] 127.0.0.1:47784]
I0822 22:37:38.537583 3439 server.go:959] GET /healthz: (40.577µs) 200 [[Go-http-client/1.1] 127.0.0.1:47789]
I0822 22:37:39.667127 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:37:43.055076 3439 server.go:959] GET /healthz: (26.24µs) 200 [[curl/7.26.0] 127.0.0.1:47792]
I0822 22:37:53.067791 3439 server.go:959] GET /healthz: (26.387µs) 200 [[curl/7.26.0] 127.0.0.1:47800]
I0822 22:38:03.089970 3439 server.go:959] GET /healthz: (38.445µs) 200 [[curl/7.26.0] 127.0.0.1:47805]
I0822 22:38:05.013454 3439 server.go:959] GET /stats/summary/: (7.716171ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:38:06.174617 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:06.177284 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:38:11.184431 3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:11.187573 3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:38:13.101813 3439 server.go:959] GET /healthz: (35.796µs) 200 [[curl/7.26.0] 127.0.0.1:47814]
I0822 22:38:23.122472 3439 server.go:959] GET /healthz: (36.311µs) 200 [[curl/7.26.0] 127.0.0.1:47821]
I0822 22:38:33.136366 3439 server.go:959] GET /healthz: (40.312µs) 200 [[curl/7.26.0] 127.0.0.1:47828]
I0822 22:38:38.537838 3439 server.go:959] GET /healthz: (44.226µs) 200 [[Go-http-client/1.1] 127.0.0.1:47833]
I0822 22:38:39.670329 3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:38:43.157960 3439 server.go:959] GET /healthz: (34.945µs) 200 [[curl/7.26.0] 127.0.0.1:47836]
I0822 22:38:53.169820 3439 server.go:959] GET /healthz: (25µs) 200 [[curl/7.26.0] 127.0.0.1:47843]
I0822 22:39:03.181619 3439 server.go:959] GET /healthz: (29.069µs) 200 [[curl/7.26.0] 127.0.0.1:47849]
I0822 22:39:05.018297 3439 server.go:959] GET /stats/summary/: (6.900965ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:39:05.318114 3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:39:05.321046 3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:41:05.321027505 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
I0822 22:39:13.194669 3439 server.go:959] GET /healthz: (26.614µs) 200 [[curl/7.26.0] 127.0.0.1:47856]

Issue 2:
Url

https://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes/v1.0.1/docs/api-reference/definitions.html#_v1_pod

Documentation Page

http://kubernetes.io/v1.0/docs/user-guide/pods.html

Section API Object