Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): no

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): api namespace path slash missing

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.1", GitCommit:"8c6525e891be1c44cbdd6fcf53097a3adb11d68c", GitTreeState:"clean", BuildDate:"2016-11-21T05:16:10Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.1", GitCommit:"8c6525e891be1c44cbdd6fcf53097a3adb11d68c", GitTreeState:"clean", BuildDate:"2016-11-21T05:16:10Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: Vagrant environment on Linux x86_64

OS (e.g. from /etc/os-release): RHEL clone

Kernel (e.g. uname -a): Linux csos-builder.csos.io 4.8.8-1.el7.csos.x86_64 #1 SMP Wed Nov 16 08:51:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Install tools: Installed from RPMs generated with a build script

Others:

What happened:

The selfLink namespace path doesn't have a slash between the end of the namespace and the resource itself.

This was tracked down as a side effect of debugging why Heapster was no longer processing pod metrics. Here curl output (compacted with jq) from a fresh machine:

# curl -s -k https://172.27.1.10:6443/api/v1/namespaces
{"kind":"NamespaceList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/","resourceVersion":"1807"},"items":[{"metadata":{"name":"default","selfLink":"/api/v1/namespacesdefault","uid":"55a50c17-b068-11e6-a4d3-080027a649ab","resourceVersion":"6","creationTimestamp":"2016-11-22T04:01:24Z"},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}},{"metadata":{"name":"kube-system","selfLink":"/api/v1/namespaceskube-system","uid":"56333705-b068-11e6-a4d3-080027a649ab","resourceVersion":"26","creationTimestamp":"2016-11-22T04:01:25Z"},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}]}

In addition this also affects the watch verb on a resource (output trimmed):

# curl -s -k https://172.27.1.10:6443/api/v1/watch/nodes
{"type":"ADDED","object":{"kind":"Node","apiVersion":"v1","metadata":{"name":"builder","selfLink":"/api/v1/nodesbuilder","uid":"55b6a17a-b068-11e6-a4d3-080027a649ab","resourceVersion":"4209","creationTimestamp":"2016-11-22T04:01:24Z","labels":

Running kubectl returns correct output:

# kubectl get node
NAME STATUS AGE
builder Ready 59m

What you expected to happen:

Expected the REST interface to return the correct selfLink path

How to reproduce it (as minimally and precisely as possible):

Build v1.5.0-beta.1 from git and run the above curl commands.

Anything else do we need to know:

Issue 2:
Is this a request for help?: No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): timeout, pd, persistent disk

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GCE

OS (e.g. from /etc/os-release):

BUILD_ID=8872.26.0
NAME="Container-VM Image"
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=55
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Google Container-VM Image"
VERSION=55
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=gci

Kernel (e.g. uname -a): Linux gke-favega-pool-1-01f44ce6-xr0y 4.4.21+ #1 SMP Wed Oct 26 13:00:52 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux

Install tools: I don't know what this is

Others: -

What happened: Early this morning I was woken up because all of our containers with GCE Persistent Disk volume mounts were down. I tried restarting the nodes in the cluster, which worked for all pods but one.

The pod in question:

Name:	postgresql-200534733-39m2y
Namespace:	addressbooks
Node:	gke-favega-pool-1-01f44ce6-xr0y/10.132.0.3
Start Time:	Wed, 02 Nov 2016 14:34:35 +0100
Labels:	app=addressbooks
pod-template-hash=200534733
role=db
Status:	Pending
IP:
Controllers:	ReplicaSet/postgresql-200534733
Containers:
postgres:
Container ID:
Image:	postgres:9.5
Image ID:
Port:	5432/TCP
State:	Waiting
Reason:	ContainerCreating
Ready:	False
Restart Count:	0
Environment Variables:
POSTGRES_USER:	addressbooks
POSTGRES_PASSWORD:	addressbooks
PGDATA:	/var/lib/postgresql/data/pgdata
Conditions:
Type	Status
Initialized	True
Ready	False
PodScheduled	True
Volumes:
postgres-volume:
Type:	GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName:	addressbooks-postgres-volume
FSType:	ext4
Partition:	0
ReadOnly:	false
default-token-mp5nw:
Type:	Secret (a volume populated by a Secret)
SecretName:	default-token-mp5nw
QoS Tier:	BestEffort
Events:
FirstSeen	LastSeen	Count	From	SubobjectPath	Type	Reason	Message
---------	--------	-----	----	-------------	--------	------	-------
1h	1m	29	{kubelet gke-favega-pool-1-01f44ce6-xr0y}	Warning	FailedMount	Unable to mount volumes for pod "postgresql-200534733-39m2y_addressbooks(183566c0-a101-11e6-bd04-42010a84016f)": timeout expired waiting for volumes to attach/mount for pod "postgresql-200534733-39m2y"/"addressbooks". list of unattached/unmounted volumes=[postgres-volume]
1h	1m	29	{kubelet gke-favega-pool-1-01f44ce6-xr0y}	Warning	FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "postgresql-200534733-39m2y"/"addressbooks". list of unattached/unmounted volumes=[postgres-volume]

The deployment for the pod:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgresql
namespace: addressbooks
spec:
replicas: 1
template:
metadata:
labels:
app: addressbooks
role: db
spec:
containers:
- name: postgres
image: postgres:9.5
ports:
- containerPort: 5432
env:
- name: POSTGRES_USER
value: addressbooks
- name: POSTGRES_PASSWORD
value: addressbooks
- name: PGDATA
value: "/var/lib/postgresql/data/pgdata"
volumeMounts:
- mountPath: "/var/lib/postgresql/data/"
name: postgres-volume
volumes:
- name: postgres-volume
gcePersistentDisk:
pdName: addressbooks-postgres-volume
fsType: ext4

Note I can manually mount the PD on the node k8s is trying to schedule the pod on just fine with gcloud.

Fishy journalctl:

Nov 02 13:55:37 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:37.185645 1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Nov 02 13:55:37 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:37.186721 1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Nov 02 13:55:41 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:41.521733 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/df702b83-a0f8-11e6-8de3-42010a84016f-default-token-5yom9" (spec.Name: "default-token-5yom9") pod "df702b83-a0f8-11e6-8de3-42010a84016f" (UID: "df702b83-a0f8-11e6-8de3-42010a84016f").

Nov 02 13:55:41 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:41.523158 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/dfef300a-a0f8-11e6-8de3-42010a84016f-default-token-ck675" (spec.Name: "default-token-ck675") pod "dfef300a-a0f8-11e6-8de3-42010a84016f" (UID: "dfef300a-a0f8-11e6-8de3-42010a84016f").

Nov 02 13:55:46 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:46.479252 1251 server.go:967] GET /healthz: (40.214µs) 200 [[curl/7.50.3] 127.0.0.1:51956]

Nov 02 13:55:46 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:46.551729 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/b88fbb51-a0fc-11e6-bd04-42010a84016f-default-token-9qv5p" (spec.Name: "default-token-9qv5p") pod "b88fbb51-a0fc-11e6-bd04-42010a84016f" (UID: "b88fbb51-a0fc-11e6-bd04-42010a84016f").

Nov 02 13:55:47 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:47.209007 1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Nov 02 13:55:47 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:47.209613 1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Nov 02 13:55:52 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:52.527417 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/df63b213-a0f8-11e6-8de3-42010a84016f-default-token-slpdg" (spec.Name: "default-token-slpdg") pod "df63b213-a0f8-11e6-8de3-42010a84016f" (UID: "df63b213-a0f8-11e6-8de3-42010a84016f").

Nov 02 13:55:53 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:53.518538 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/df8a26b2-a0f8-11e6-8de3-42010a84016f-default-token-5yom9" (spec.Name: "default-token-5yom9") pod "df8a26b2-a0f8-11e6-8de3-42010a84016f" (UID: "df8a26b2-a0f8-11e6-8de3-42010a84016f").

Nov 02 13:55:56 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:56.501685 1251 server.go:967] GET /healthz: (32.53µs) 200 [[curl/7.50.3] 127.0.0.1:52048]

Nov 02 13:55:57 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:57.272752 1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Nov 02 13:55:57 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:57.273347 1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Nov 02 13:55:58 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:58.593168 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/dfa7ae50-a0f8-11e6-8de3-42010a84016f-default-token-eobju" (spec.Name: "default-token-eobju") pod "dfa7ae50-a0f8-11e6-8de3-42010a84016f" (UID: "dfa7ae50-a0f8-11e6-8de3-42010a84016f").

Nov 02 13:56:00 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:56:00.601068 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/e00fd33c-a0f8-11e6-8de3-42010a84016f-default-token-hwo5r" (spec.Name: "default-token-hwo5r") pod "e00fd33c-a0f8-11e6-8de3-42010a84016f" (UID: "e00fd33c-a0f8-11e6-8de3-42010a84016f").

Nov 02 13:56:00 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:56:00.602974 1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/e01041a6-a0f8-11e6-8de3-42010a84016f-default-token-hwo5r" (spec.Name: "default-token-hwo5r") pod "e01041a6-a0f8-11e6-8de3-42010a84016f" (UID: "e01041a6-a0f8-11e6-8de3-42010a84016f").

What you expected to happen: I expected the pod to start just fine, like every other pod in my cluster.

How to reproduce it (as minimally and precisely as possible): It only fails for this single pod, so I don't know how to reproduce it.

Anything else do we need to know: I don't think so.