Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

node, timeout, restart

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: https://cloud.google.com/container-engine/

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools:

Others:

What happened:

Requests started to timeout every so often, we found this from runscope monitoring. When I reproduced the timeout it was only intermittent. All the k8 deployments and pods were online.

I looked in kubectl get events and saw:

44m 44m 1 gke-backend-default-pool-a671890f-17dj Node Normal NodeNotReady {controllermanager } Node gke-backend-default-pool-a671890f-17dj status is now: NodeNotReady

When I ran kubectl get nodes it was in Ready status:

$ kubectl get nodes
NAME STATUS AGE
gke-backend-default-pool-a671890f-17dj Ready 7d
gke-backend-default-pool-a671890f-berw Ready 7d
gke-backend-default-pool-a671890f-h0li Ready 7d
gke-backend-default-pool-a671890f-oqjm Ready 7d
gke-backend-default-pool-a671890f-pu28 Ready 7d
gke-backend-default-pool-a671890f-r14r Ready 7d
gke-backend-default-pool-a671890f-sga0 Ready 7d
gke-backend-default-pool-a671890f-uxby Ready 7d

I then manually deleted the node:

kubectl delete node gke-backend-default-pool-a671890f-17dj

And the timeouts went away. We had the same problem last week but at that time I just removed all the nodes because I didn't have time to dig into the issue.

What you expected to happen:

When the node goes into NodeNotReady it should remove itself from the pool and have another one take it's place. Also kubectl get events says the node went into NodeNotReady but kubectl get nodes showed it as Ready, maybe it went back to Ready status but for some reason wasn't accepting http request to the exposed service IP.

How to reproduce it (as minimally and precisely as possible):

We weren't touching the cluster when the node stopped accepting requests, it was working for a week and then stopped at a seemingly random time.

Anything else do we need to know:

Not that I can think of, but let me know if there's any other information I can provide, unfortunatly the problem node has been deleted already but if it happens again I could run some commands to help diagnose the issue.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No,

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

"already being used"

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:35:03Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1228+2212c421f6e10e", GitCommit:"2212c421f6e10e3bb7ff1e106f14dcd049990235", GitTreeState:"clean", BuildDate:"2016-12-02T17:27:41Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

Environment

Cloud provider or hardware configuration: GCE

OS (e.g. from /etc/os-release): Debian GNU/Linux 7 (wheezy)

Kernel (e.g. uname -a):

Install tools: cluster/gce/upgrade.sh

Others:

What happened:

A Pod with a PVC got deleted during a rolling Node upgrade. A controller recreated it on a different Node. The new Pod was unable to attach the PD because it was still marked as being in use by the previous Node:

Failed to attach volume "pvc-61045ddc-b8c7-11e6-bee7-42010a800002" on node "kubernetes-minion-group-yl0d" with: googleapi: Error 400: The disk resource 'kubernetes-dynamic-pvc-61045ddc-b8c7-11e6-bee7-42010a800002' is already being used by 'kubernetes-minion-group-c4x6'

The issue resolved itself after 7min. I was able to reproduce this based on #37585 (comment). Although it happened to a different pod this time, the delay was about the same (7-8min).

What you expected to happen:

The PD should mount to the new Node that the Pod has scheduled to, without such a long delay.

How to reproduce it (as minimally and precisely as possible):

Start a v1.4.6 cluster on GCE.

Create the following example PetSet and wait for all Pods to be Running.

# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
# *.nginx.default.svc.cluster.local
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1alpha1
kind: PetSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 5
template:
metadata:
labels:
app: nginx
annotations:
pod.alpha.kubernetes.io/initialized: "true"
spec:
containers:
- name: nginx
image: gcr.io/google_containers/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi

Delete PetSet without cascading (keep the Pods).

kubectl delete petset web --cascade=false

Upgrade master to v1.6.0-alpha.0.1228+2212c421f6e10e with cluster/gce/upgrade.sh.

git checkout release-1.5
git pull upstream release-1.5
cluster/gce/upgrade.sh -M v1.6.0-alpha.0.1228+2212c421f6e10e

Create the following StatefulSet to adopt the Pods.

# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
# *.nginx.default.svc.cluster.local
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 5
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: gcr.io/google_containers/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi

Upgrade nodes to v1.6.0-alpha.0.1228+2212c421f6e10e with cluster/gce/upgrade.sh.

cluster/gce/upgrade.sh -N v1.6.0-alpha.0.1228+2212c421f6e10e

Check for Pods stuck in ContainerCreating. Look in the event log for The disk resource ... is already being used by ...

NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 16m
web-1 1/1 Running 0 15m
web-2 0/1 ContainerCreating 0 4m
web-3 1/1 Running 0 15m
web-4 1/1 Running 0 15m

Anything else do we need to know:

kubectl describe pod on the stuck pod

kube-controller-manager.log

kubernetes-minion-group-c4x6 kubelet.log

kubernetes-minion-group-yl0d kubelet.log