Issue 1:
I'm running k8s on AWS. kube-up.sh works, and I can run a simple Rails app (controller and service). Everything is great, but when I go back the next day to access the app or load a new version the nodes have degraded and are all NotReady

$ kubectl get nodes
NAME LABELS STATUS
ip-172-20-0-196.us-west-2.compute.internal kubernetes.io/hostname=ip-172-20-0-196.us-west-2.compute.internal NotReady
ip-172-20-0-197.us-west-2.compute.internal kubernetes.io/hostname=ip-172-20-0-197.us-west-2.compute.internal NotReady
ip-172-20-0-198.us-west-2.compute.internal kubernetes.io/hostname=ip-172-20-0-198.us-west-2.compute.internal NotReady
ip-172-20-0-199.us-west-2.compute.internal kubernetes.io/hostname=ip-172-20-0-199.us-west-2.compute.internal NotReady

The pods are Pending

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
chronic-controller-16uic 0/1 Pending 0 6h
chronic-controller-6x025 0/1 Pending 0 1h
chronic-controller-x7owf 0/1 Pending 0 6h
private-registry-pbvdi 0/1 Pending 0 6h
tripplanner-controller-3q6km 0/1 Pending 0 6h
tripplanner-controller-bhwwy 0/1 Pending 0 1h
tripplanner-controller-fdep8 0/1 Pending 0 6h

Here are the recent events:

$ kubectl get events
FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT REASON SOURCE MESSAGE
Sat, 12 Sep 2015 22:44:10 -0500 Tue, 15 Sep 2015 15:04:08 -0500 44 ip-172-20-0-199.us-west-2.compute.internal Node NodeNotReady {kubelet ip-172-20-0-199.us-west-2.compute.internal} Node ip-172-20-0-199.us-west-2.compute.internal status is now: NodeNotReady
Mon, 14 Sep 2015 09:24:35 -0500 Tue, 15 Sep 2015 15:22:41 -0500 7531 chronic-controller-ljzn6 Pod failedScheduling {scheduler } no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500 Tue, 15 Sep 2015 15:22:41 -0500 7757 private-registry-ysjfj Pod failedScheduling {scheduler } no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500 Tue, 15 Sep 2015 15:22:42 -0500 7284 tripplanner-controller-7b0yr Pod failedScheduling {scheduler } no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500 Tue, 15 Sep 2015 15:22:42 -0500 7413 tripplanner-controller-s86wz Pod failedScheduling {scheduler } no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500 Tue, 15 Sep 2015 15:22:42 -0500 7374 chronic-controller-f2nrx Pod failedScheduling {scheduler } no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500 Tue, 15 Sep 2015 15:22:42 -0500 7447 chronic-controller-j4538 Pod failedScheduling {scheduler } no nodes available to schedule pods
Mon, 14 Sep 2015 01:42:38 -0500 Tue, 15 Sep 2015 15:22:42 -0500 9303 tripplanner-controller-04ztj Pod failedScheduling {scheduler } no nodes available to schedule pods

I logged into the master and one of the minions to look at /var/log/kubelet.log but that file doesn't exist on those hosts. Trying to ssh into one of the AWS instances is also spotty for me, but the EC2 instances all claim to be up and healthy.

I have killed everything with kube-down.sh and built it all again. Everything worked for the afternoon and then was down again the next morning. I don't know what else to look at or do. I'm happy to look at things. I'm @barrettclark in the slack channel.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.4.6

Environment:

Cloud provider or hardware configuration: GCE

OS (e.g. from /etc/os-release): Google, Container-Optimized OS, beta 55-8872.47.0, Kernel: ChromiumOS-4.4.4 Kubernetes: 1.4.5 Docker: 1.11.2

Kernel (e.g. uname -a): Linux gke-warpdrive-default-pool-eae1f3b0-spgr 4.4.21+ #1 SMP Thu Nov 10 21:43:53 PST 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

Install tools:

Others:

What happened:

A a persistent disk was created via:

gcloud compute disks create warpdrive-jenkins-home --type pd-standard

This persistent disk is referenced in a K8s Deployment (yaml). The Pod failed on startup, due to FailedMount followed by FailedSync.

$ kubectl describe pods --namespace=jenkins | tail
Events:
FirstSeen	LastSeen	Count	From	SubobjectPath	Type	Reason	Message
---------	--------	-----	----	-------------	--------	------	-------
5m	5m	1	{default-scheduler }	Normal	Scheduled	Successfully assigned jenkins-master-3888267560-x2rpc to gke-warpdrive-default-pool-af4bf63e-qjrx
3m	1m	2	{kubelet gke-warpdrive-default-pool-af4bf63e-qjrx}	Warning	FailedMount	Unable to mount volumes for pod "jenkins-master-3888267560-x2rpc_jenkins(5f752595-b01a-11e6-972b-42010a8e0029)": timeout expired waiting for volumes to attach/mount for pod "jenkins-master-3888267560-x2rpc"/"jenkins". list of unattached/unmounted volumes=[jenkins-home]
3m	1m	2	{kubelet gke-warpdrive-default-pool-af4bf63e-qjrx}	Warning	FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "jenkins-master-3888267560-x2rpc"/"jenkins". list of unattached/unmounted volumes=[jenkins-home]

Interestingly, the persistent disk was actually attached:

$ gcloud compute disks describe warpdrive-jenkins-home | tail
lastAttachTimestamp: '2016-11-21T10:43:21.433-08:00'
lastDetachTimestamp: '2016-11-21T10:37:20.688-08:00'
name: warpdrive-jenkins-home
selfLink: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/disks/warpdrive-jenkins-home
sizeGb: '1000'
status: READY
type: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/diskTypes/pd-standard
users:
- https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/instances/gke-warpdrive-default-pool-af4bf63e-qjrx
zone: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b

I attempted to recreate , but the issue persisted across all attempts.

Worth noting is that the issue is avoided if the persistent disk is defined by:

gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v2.tar.gz
gcloud compute disks create jenkins-home --image jenkins-home-image

Also worth noting is that I was able to ssh into a node and manually mount, write some files, unmount, then have K8s auto detach/attach via deletion/creation of Deployments, and then manually mount and verify that written files were there.

What you expected to happen:

I expected the persistent disk to attach and mount with no issue.

How to reproduce it (as minimally and precisely as possible):

gcloud container clusters create warpdrive --machine-type n1-standard-4 --num-nodes 4 --subnetwork sym-infra-dev-guse1 --enable-autoscaling --min-nodes 4 --max-nodes 20 --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"

gcloud compute disks create warpdrive-jenkins-home --type pd-standard

kubectl create ns jenkins

create Deployment with:

volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
volumes:
- name: jenkins-home
gcePersistentDisk:
pdName: warpdrive-jenkins-home
fsType: ext4
partition: 1

watch in horror as the pod fails

Anything else do we need to know:

We're desperate