Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): cAdvisor inotify_init: too many open files

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Environment:

Cloud provider or hardware configuration: hardware

OS (e.g. from /etc/os-release): SUSE 12

Kernel (e.g. uname -a):

Install tools: manul

Others:

What happened: monit monitor kubelet and flanneld service, and kubelet service depends on flanneld service, i make flanneld not running, so kubelet will restart by monit, after many times restart, i fix flanneld service, but kubelet launch with error

What you expected to happen:

kubelet running success

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

E1123 14:47:21.372435 24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1123 14:47:21.372469 24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
I1123 14:47:25.806786 24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
I1123 14:47:30.806954 24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
E1123 14:47:31.397830 24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1123 14:47:31.397861 24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
I1123 14:47:35.807141 24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
I1123 14:47:40.807326 24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
E1123 14:47:41.415673 24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1123 14:47:41.415704 24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
I1123 14:47:45.807487 24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

docker hang

docker ps hangs

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GKE v1.4.5

OS (e.g. from /etc/os-release):

BUILD_ID=8872.26.0

NAME="Container-VM Image"

GOOGLE_CRASH_ID=Lakitu

VERSION_ID=55

VERSION=55

ID=gci

Kernel (e.g. uname -a): Linux gke-api-cluster-1-pool-v1-0f4fa4ec-p95f 4.4.21+ #1 SMP Wed Oct 26 13:00:52 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

Install tools:

Others:

What happened: docker ps hangs for more than 10 seconds in kube-docker-monitor.service and the docker daemon is killed.

What you expected to happen: docker ps to complete quickly and docker not to be killed.

How to reproduce it (as minimally and precisely as possible): I have not had success reproducing this at will. It's a daily occurrence though.

Anything else do we need to know:

It feels weird to file this here as it's rather GKE-specific, but we had GCP support direct us here.

I've collected several stack dumps here (taken after the 10s hang):

https://gist.github.com/gdvalle/b58b5d793ebd229f9c40438a4c5ae160

At least a couple of these seem to match up with this bug:

moby/moby#27607

Looks like those patches should ship with the next 1.12 release...but no date for that yet. However, 1.13 did just ship, and that looks like it has the patches. So maybe we wait for k8s validation on that.

So far it looks like we have a workaround by just allowing more time for the docker ps to succeed. Existing containers on the node were still healthy so it seemed reasonable to bump it.

kubernetes/cluster/gce/gci/health-monitor.sh

Line 29
in
8fd4145

if ! timeout 10 docker ps > /dev/null; then

We've upped the timeout to 50s, but over the last couple days we haven't seen it need more than about 35s in total with most being just a few seconds longer than 10.