Issue 1:
Every node on GCE by default has cAdvisor, fluentd-es running as pod. We are going to put etcd server into a pod soon, and other master components into pods later. Those pods are pre-provisioned, and not through apiserver, and scheduler, I called them "static" pods.

Today, kubelet does generate events related to those static pods, and push them to etcd. But there is no way to query those pods config from clients, and static pods in crashloop cannot be detected easily because apiserver doesn't aware of them.

Also those pods do consume resources, and without awareness of static pods on a node, scheduler might overcommit a node easily.

We should have a mechanism defined for kubelet to sync back to master with those static Pods.

cc/ @bgrant0607 @thockin @brendandburns

Issue 2:
https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/25701/node-pull-build-e2e-test/7060/build-log.txt

------------------------------
â€¢ Failure [20.010 seconds]
Container runtime Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:314
container runtime conformance blackbox test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:313
when running a container with invalid image
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:306
it should run with the expected status [Conformance] [It]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:305

Expected error:
<*errors.errorString | 0xc82039fb20>: {
s: "Failed to get container status",
}
Failed to get container status
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:287
------------------------------
STEP: Creating a host exec pod
â€¢ Failure [120.071 seconds]
PrivilegedPod
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:83
should test privileged pod [It]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:82

Failed waiting for pod "privileged-pods/hostexec" to become running: Time out while waiting for pod privileged-pods/hostexec to become running; current status: {Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[]}
Expected error:
<*errors.errorString | 0xc8204dd330>: {
s: "Time out while waiting for pod privileged-pods/hostexec to become running; current status: {Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[]}",
}
Time out while waiting for pod privileged-pods/hostexec to become running; current status: {Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[]}
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:210
------------------------------
W0520 22:12:02.102156 932 request.go:347] Field selector: v1 - pods - metadata.name - busybox-scheduling-b70d3785-1ed6-11e6-bc67-42010af00031: need to check if this is versioned correctly.
STEP: Building a namespace api object
May 20 22:12:02.088: INFO: Skipping waiting for service account
May 20 22:17:02.103: INFO: Unexpected error occurred: timed out waiting for the condition
STEP: Collecting resource usage data
May 20 22:17:02.111: INFO: Closed stop channel. Waiting for 1 workers
May 20 22:17:02.111: INFO: Closing worker for tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
May 20 22:17:02.111: INFO: Waitgroup finished.
May 20 22:17:02.111: INFO: Unknown output type: . Skipping.
May 20 22:17:02.111: INFO: Waiting up to 1m0s for all nodes to be ready
May 20 22:17:02.114: INFO: Found DeleteNamespace=false, skipping namespace deletion!
â€¢ Failure [300.029 seconds]
[k8s.io] Kubelet
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508
when scheduling a busybox command in a pod
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:81
it should print the output to logs [It]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:80

Expected error:
<*errors.errorString | 0xc82000fea0>: {
s: "timed out waiting for the condition",
}
timed out waiting for the condition
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:68
------------------------------
STEP: Building a namespace api object
May 20 22:17:02.117: INFO: Skipping waiting for service account
STEP: Collecting resource usage data
May 20 22:18:02.128: INFO: Closed stop channel. Waiting for 1 workers
May 20 22:18:02.128: INFO: Closing worker for tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
May 20 22:18:02.128: INFO: Waitgroup finished.
May 20 22:18:02.128: INFO: Unknown output type: . Skipping.
May 20 22:18:02.128: INFO: Waiting up to 1m0s for all nodes to be ready
May 20 22:18:02.132: INFO: Found DeleteNamespace=false, skipping namespace deletion!
â€¢ Failure [60.018 seconds]
[k8s.io] Kubelet
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508
when scheduling a read only busybox container
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:124
it should not write to root filesystem [It]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:123

Timed out after 60.000s.
Expected
<string>:
to equal
<string>: sh: can't create /file: Read-only file system

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:122
------------------------------
W0520 22:18:02.140060 932 request.go:347] Field selector: v1 - pods - metadata.name - stats-busybox-b77a134c-1ed8-11e6-bc67-42010af000310: need to check if this is versioned correctly.
STEP: Building a namespace api object
May 20 22:18:02.134: INFO: Skipping waiting for service account
May 20 22:23:02.140: INFO: Unexpected error occurred: timed out waiting for the condition
STEP: Collecting resource usage data
May 20 22:23:02.141: INFO: Closed stop channel. Waiting for 1 workers
May 20 22:23:02.141: INFO: Closing worker for tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
May 20 22:23:02.141: INFO: Waitgroup finished.
May 20 22:23:02.141: INFO: Unknown output type: . Skipping.
May 20 22:23:02.141: INFO: Waiting up to 1m0s for all nodes to be ready
May 20 22:23:02.143: INFO: Found DeleteNamespace=false, skipping namespace deletion!
â€¢ Failure [300.011 seconds]
[k8s.io] Kubelet
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508
metrics api
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:165
when querying /stats/summary
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:164
it should report resource usage through the stats api [It]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:163

Expected error:
<*errors.errorString | 0xc82000fea0>: {
s: "timed out waiting for the condition",
}
timed out waiting for the condition
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:216
------------------------------
E0520 22:25:02.144225 932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:25:02.144281 932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:25:02.144414 932 provider.go:115] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0520 22:25:02.144545 932 docker.go:184] Pulling image gcr.io/google_containers/busybox:1.24 without credentials
â€¢ Failure in Spec Setup (BeforeEach) [240.002 seconds]
Image Container Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:138
image conformance blackbox test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:137
when testing images that exist
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:84
It should present successfully [Conformance] [BeforeEach]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:66

Timed out after 120.001s.
Expected error:
<*errors.errorString | 0xc8202de490>: {
s: "image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request. details: (operation timeout: context deadline exceeded)",
}
image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request. details: (operation timeout: context deadline exceeded)
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55
------------------------------
E0520 22:29:02.146143 932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:29:02.146170 932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:29:02.146236 932 docker.go:184] Pulling image gcr.io/google_containers/busybox:1.24 without credentials
â€¢ Failure in Spec Setup (BeforeEach) [240.002 seconds]
Image Container Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:138
image conformance blackbox test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:137
when testing images that exist
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:84
should list pulled images [Conformance] [BeforeEach]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:75

Timed out after 120.000s.
Expected error:
<*errors.errorString | 0xc8202de4b0>: {
s: "image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request. details: (operation timeout: context deadline exceeded)",
}
image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request. details: (operation timeout: context deadline exceeded)
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55
------------------------------
E0520 22:33:02.147760 932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:33:02.147787 932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:33:02.147874 932 provider.go:115] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0520 22:33:02.147973 932 docker.go:184] Pulling image foo.com/foo/fooimage:latest without credentials
E0520 22:37:02.148580 932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:37:02.148619 932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:37:02.148949 932 docker.go:184] Pulling image gcr.io/google_containers/not_exist:latest without credentials
â€¢ Failure [840.004 seconds]
Image Container Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:138
image conformance blackbox test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:137
when testing image that does not exist
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:136
it should not present images [Conformance] [It]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:111

Expected error:
<dockertools.operationTimeout>: {
err: {
s: "context deadline exceeded",
},
}
operation timeout: context deadline exceeded
not to have occurred

/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:108
------------------------------
E0520 22:47:01.562282 932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:47:01.562307 932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:47:01.562377 932 provider.go:115] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0520 22:47:01.562486 932 docker.go:184] Pulling image foo.com/foo/fooimage:latest without credentials

Summarizing 8 Failures:

[Fail] Container runtime Conformance Test container runtime conformance blackbox test when running a container with invalid image [It] it should run with the expected status [Conformance]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:287

[Fail] PrivilegedPod [It] should test privileged pod
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:210

[Fail]
---------------------------------------------------------
Received interrupt. Running AfterSuite...
^C again to terminate immediately
I0520 22:48:41.805703 932 e2e_node_suite_test.go:84] Tests Finished
[k8s.io] Kubelet when scheduling a busybox command in a pod [It] it should print the output to logs
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:68

[Fail] [k8s.io] Kubelet when scheduling a read only busybox container [It] it should not write to root filesystem
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:122

[Fail] [k8s.io] Kubelet metrics api when querying /stats/summary [It] it should report resource usage through the stats api
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:216

[Fail] Image Container Conformance Test image conformance blackbox test when testing images that exist [BeforeEach] It should present successfully [Conformance]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55

[Fail] Image Container Conformance Test image conformance blackbox test when testing images that exist [BeforeEach] should list pulled images [Conformance]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55

[Fail] Image Container Conformance Test image conformance blackbox test when testing image that does not exist [It] it should not present images [Conformance]
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:108

Ran 18 of 18 Specs in 2699.365 seconds
FAIL! -- 4 Passed | 8 Failed | 0 Pending | 0 Skipped I0520 22:48:41.809756 932 e2e_node_suite_test.go:118] Process Log For Failed Suite On tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
-------------------------------------------------------------