Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):Bug

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: AWS, this particular node is a c3.4xlarge with 60 GB gp2 root volume.

OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)

Kernel (e.g. uname -a): Linux 4.4.26-k8s #1 SMP Fri Oct 21 05:21:13 UTC 2016 x86_64 GNU/Linux

Install tools: Kops

Others:

What happened:

The kubelet image garbage collection is triggered because disk space falls below the minimum threshold. Kubelet starts image deletions to recover disk space but by the end of the cleanup the node has been severely impacted with pods becoming unresponsive, restarted, and often rescheduled onto other nodes. In this particular case you can see that CPU becomes almost zero because everything has been moved off this node.

Timing of this particular failure:

16:56:38.000 GC started with image_gc_manager.go:238] [imageGCManager]: Disk usage on "/dev/xvda1" (/) is at 90% which is over the high threshold (90%). Trying to free 5766938624 bytes message in logs

16:56:38 See the first image_gc_manager.go:303] [imageGCManager]: Removing image message in the logs

17:01:38 See the final Removing Image message in the logs which is the 8th image to be deleted

17:02 By this time graphs show just about everything has been moved off this node and there is nothing left running on it.

The following graphs seem to indicate that as soon as the GC has finished kubelet "wakes up" and realizes everything has been removed from this node. In the dashboard I saw this node Ready state as Unknown instead of True during this period. It recovers and becomes functional after this hiccup and works fine when new pods get scheduled back onto the node.

As expected the IO is higher during the cleanup but other metrics look fine and don't indicate a overloaded instance.

What you expected to happen:

No impact to running services on node.

How to reproduce it (as minimally and precisely as possible):

This issue has happened on 3 different nodes all configured the same way. I don't believe a GC has occurred without causing this impact.

Anything else do we need to know:

Discussed briefly with @justinsb

Just a guess but is there any chance im.imageRecordsLock.Lock() at the top of image_gc_manager.go->freeSpace causes kubelet to stop processing other requests while GC is occuring?

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

initContainers imagePullPolicy

initContainers required value

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.3", GitCommit:"35fd1363821b8432ccde55632cf9ead4c79eddbb", GitTreeState:"clean", BuildDate:"2016-12-08T21:58:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.3", GitCommit:"35fd1363821b8432ccde55632cf9ead4c79eddbb", GitTreeState:"clean", BuildDate:"2016-12-08T21:54:19Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GCE

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools: cluster/kube-up.sh

Others:

What happened:

When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying imagePullPolicy is a required value.

The ReplicaSet "nginx" is invalid: spec.template.spec.initContainers[0].imagePullPolicy: Required value

The ReplicationController "nginx" is invalid: spec.template.spec.initContainers[0].imagePullPolicy: Required value

What you expected to happen:

The create should succeed and the Init Container should receive a default imagePullPolicy.

How to reproduce it (as minimally and precisely as possible):

Try to create either of the following:

ReplicaSet

apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: nginx
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "busybox",
"command": ["wget", "-O", "/work-dir/index.html", "http://kubernetes.io/index.html"],
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
}
]
}
]'
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}

Replication Controller

apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 3
selector:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "busybox",
"command": ["wget", "-O", "/work-dir/index.html", "http://kubernetes.io/index.html"],
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
}
]
}
]'
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}

Anything else do we need to know:

If I add "imagePullPolicy": "IfNotPresent", to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds.

When I create an Init Container without imagePullPolicy in a StatefulSet Pod template, it succeeds.

When I create an Init Container without imagePullPolicy in a bare Pod, it succeeds:

apiVersion: v1
kind: Pod
metadata:
name: nginx
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "busybox",
"command": ["wget", "-O", "/work-dir/index.html", "http://kubernetes.io/index.html"],
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
}
]
}
]'
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}