Issue 1:
I find it curious that this doesn't already exist. I assume there is a reason for that but I couldn't find any references for it. What is the reason?

If there is hope of adding custom gke scopes, I'd be happy to submit a PR, but I'd like some feedback first. I noticed notes for @mbforbes like "Refactor setting scope flags." What refactor was in mind? Currently scopes are set in the MINION_SCOPES variable which is a bash array. Because it's an array it needs a join_csv function. But as far as I can tell, the variable is never "pushed" or "popped" so I see no benefit from using the array. Although if we eventually want to push e.g. monitoring, logging scopes onto the array, we should keep it. Alternatively it might be easy to just concat comma-separated strings.

I don't mean to bike shed here, I'd just like to do the simplest PR that will be accepted and enable custom scopes in gke.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

unable to delete filesystem

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT.

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:15:30Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:08:43Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GKE

OS (e.g. from /etc/os-release):

PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"

NAME="Debian GNU/Linux"

VERSION_ID="7"

VERSION="7 (wheezy)"

ID=debian

ANSI_COLOR="1;31"

HOME_URL="http://www.debian.org/"

SUPPORT_URL="http://www.debian.org/support/"

BUG_REPORT_URL="http://bugs.debian.org/"

Kernel (e.g. uname -a): Linux gke-xxx-default-pool-xxx-xxx 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

Install tools: None.

Others: None.

What happened:

Garbage collector fails to remove old containers. Eventually nodes run out of disc space and docker daemon and/or other things start to crash. Instances get stuck with 100% CPU usage.

I've since replaced the stuck instances (but this is not the first time). I'm still seeing following error in Google Logs:

kubelet: "Failed to remove dead container "/k8s_POD.xxxx": Error response from daemon: Unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy"

docker: Handler for DELETE /containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e returned error: Unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy"

Before I saw more interesting errors:

docker: "collecting stats for 2356f6b7eb7f993d6ea1e9286e5edc0da11dda4e9d2e4d4beb05c35e71878067: invalid id:

kubelet: "Failed to create summary reader for "/docker/1a2010b84bf49f396e5acd8a4d0d96458043ff10eeb5dc4126ad3213ef213155": none of the resources are being tracked."

A bit later:

22:06:11.000
eviction manager: unexpected err: failed GetContainerInfoV2: unknown container: "/"
22:06:11.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:12.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:13.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:13.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:14.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:14.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:14.000
Failed to create summary reader for "/docker/9dd56e5a42bb17701e322476aeeb3767295c1705afc3bb37bb0c2c83dcdc9750": none of the resources are being tracked.
22:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:16.000
while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
22:06:16.000
while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
22:06:16.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:17.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:17.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:18.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:18.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:18.000
Failed to create summary reader for "/docker/82ddd29cbe84685adf1322ad77016b1c4721a5e228fe23381b320c480cbb904f": none of the resources are being tracked.
22:06:19.000
Error getting image list: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:19.000
Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
22:06:19.000
Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /

and

15:22:03.000
Failed to delete container: operation timeout: context deadline exceeded; Skipping pod "ec5aa9ac-7e6e-11e6-b32a-42010a800070"
15:22:04.000
Failed to create pod infra container: RunContainerError; Skipping pod "xxx(323984f3-7fd4-11e6-b32a-42010a800070)": runContainer: operation timeout: context deadline exceeded
15:22:19.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:29.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:33.000
GenericPLEG: Unable to retrieve pods: operation timeout: context deadline exceeded
15:22:34.000
Error listing containers: dockertools.operationTimeout{err:(*errors.errorString)(0xc82000f310)}
15:22:34.000
Failed cleaning pods: operation timeout: context deadline exceeded
15:22:39.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:40.000
Container garbage collection failed: operation timeout: context deadline exceeded
15:22:49.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:59.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:23:05.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:23:09.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq

Eventually leading to this:

03:06:14.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:14.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
Error getting image list: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
03:06:15.000
Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
03:06:15.000
Failed to create summary reader for "/docker/93f4b7062198072d70f8ec855639464926747ba8ddb902265c0155623c5f1c63": none of the resources are being tracked.
03:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
eviction manager: unexpected err: failed ImageStats: failed to list docker images - Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:17.000

What you expected to happen:

Garbage collector doesn't let nodes run out of disc space.

How to reproduce it (as minimally and precisely as possible):

I'm not exactly sure. We have multiple deployments in multiple namespaces. The only out of ordinary thing is that we use big container images ~1gb. This has happened couple times. We were running weave-scope before and it seemed that maybe it might be related because when I inspected failing nodes using kubectl describe they showed event "deleting all containers" and the only container left running was the scope. It was stuck in "terminating" state and no way to remove it (kubectl delete pod) didn't work.

I have since removed weave-scope, but I still see the Failed to remove dead container errors.

Anything else do we need to know:

I've tracked down a docker issue which seems to be related - moby/moby#21704.

Also seems related to #26000, but we're using standard GKE image using AUFS.

Docker issue mentions that upgrading kernel to >=3.19 might help. Any chances we could have newer kernel in GKE image?