Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Features Request

Kubernetes version (use kubectl version):

[root@localhost ~]# kubectl version

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1877+3f4a66f3d6892b-dirty", GitCommit:"3f4a66f3d6892b8d8831a8a60b91fd1afbefee4d", GitTreeState:"dirty", BuildDate:"2016-10-31T20:47:46Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

[root@localhost ~]# kubeadm version

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:x89_64

OS (e.g. from /etc/os-release):Ubuntu 16.04

Kernel (e.g. uname -a): Linux kubenode01 4.4.0-38-generic #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Install tools:kubeadm

Others:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

Issue 2:
While running performance tests (start 30 pods per node in 50 nodes cluster) I faced the following problem.

Create replication controller and resize it to 1500. I saw the following logs:

I0327 14:45:09.036772 14025 density.go:109] Controller my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b: Found 0 pods out of 1500

I0327 14:45:16.918259 14025 density.go:109] Controller my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b: Found 835 pods out of 1500

I0327 14:45:24.465116 14025 density.go:129] Controller my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b: Found 1500 pods out of 1500

However, looking into scheduler logs, the first time when it attempts to schedule any of those pods is related to the log:

I0327 13:45:55.474005 4551 factory.go:166] About to try and schedule pod my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39bfjo6e

This is roughly ~45 seconds after the first pod was created. The question is what the scheduler was doing in the meantime - the only logs from that time are:

I0327 13:40:52.230407 4551 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF

E0327 13:40:52.231778 4551 reflector.go:149] watch of *api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [4053/16]) [

5052]

I0327 13:45:53.272954 4551 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF

I0327 13:45:53.273448 4551 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF

E0327 13:45:53.277590 4551 reflector.go:149] watch of *api.Pod ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [12989/5082]) [1

3988]

E0327 13:45:53.277823 4551 reflector.go:149] watch of *api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [12989/5082]

) [13988]

I0327 13:45:55.474005 4551 factory.go:166] About to try and schedule pod my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39bfjo6e

A very similar thing happened in Kubelet. During the same test, the first pod was bind to minion "*0jxm" at:

I0327 13:46:00.291905 4551 factory.go:303] Attempting to bind my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b7aukl to e2e-test-wojtekt-minion-0jxm.c.groovy-sentry-504.internal

However, Kubelet reacted to it 50 seconds later:

I0327 13:46:50.560684 4712 kubelet.go:1099] No Infra Container for "my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b7aukl_default" found. All containers will be restarted.

Again, it seems that nothing was really happening during that 50 seconds. The Kubelet logs from that time are:

I0327 13:44:55.038542 4712 server.go:654] GET /healthz: (886.441µs) 0 [[monit/5.4] 127.0.0.1:37322]

E0327 13:46:43.817523 4712 reflector.go:149] watch of *api.Node ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [17177/7110]) [18176

]

I0327 13:46:46.492099 4712 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF

I0327 13:46:46.492181 4712 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF

E0327 13:46:47.256804 4712 reflector.go:149] watch of *api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [17354/6384]) [18

353]

E0327 13:46:47.256950 4712 reflector.go:149] watch of *api.Pod ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [17352/8087]) [18351]

I0327 13:46:50.558471 4712 kubelet.go:1099] No Infra Container for "my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b8hwgv_default" found. All containers will be restarted.

...

[and it works since then].

Investigate what is happening during that 45 seconds periods of "nothing".

cc @fgrzadkowski