Issue 1:
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28599/kubernetes-pull-build-test-e2e-gce/58426/

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:88
Expected error:
<*errors.errorString | 0xc820c80740>: {
s: "error while stopping RC: rc-light-ctrl: timed out waiting for \"rc-light-ctrl\" to be synced",
}
error while stopping RC: rc-light-ctrl: timed out waiting for "rc-light-ctrl" to be synced
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:307

Happened on a presubmit run in #28599.

Previous issues for this test: #27443 #27835 #28900

Issue 2:
Actually there are 2 results in density test that seems conflict but NOT.

1.throughput case

when run the density test, this is the 1st test that shows count of different pod status. This pic indicate the throughput of apiserver+controller/schduler/kubelet. As the waiting curve drop down slowly and the pending curve is always 0. So that scheduler is the main bottleneck in the system when create a lot of pod at one time.

2. one pod latency case

Nov 27 19:10:28.552: INFO: 10% worst schedule latencies: [{density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-13 33.1.1.212 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-66 33.1.1.232 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-214 33.1.1.228 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-251 33.1.2.26 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-276 33.1.1.193 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-174 33.1.1.89 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-134 33.1.1.214 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-90 33.1.2.20 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-105 33.1.1.159 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-140 33.1.2.71 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-63 33.1.1.192 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-125 33.1.1.96 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-144 33.1.1.104 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-54 33.1.1.172 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-118 33.1.2.37 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-131 33.1.1.169 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-23 33.1.1.201 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-75 33.1.2.78 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-99 33.1.1.98 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-253 33.1.1.244 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-137 33.1.2.56 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-18 33.1.2.26 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-254 33.1.2.32 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-168 33.1.2.29 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-146 33.1.2.51 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-142 33.1.2.40 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-143 33.1.1.179 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-145 33.1.2.37 2s}]
Nov 27 19:10:28.553: INFO: perc50: 0, perc90: 1s, perc99: 2s
Nov 27 19:10:28.553: INFO: 10% worst run-after-schedule latencies: [{density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-20 33.1.2.65 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-269 33.1.2.39 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-250 33.1.1.126 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-206 33.1.1.85 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-66 33.1.1.232 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-151 33.1.1.120 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-217 33.1.1.187 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-46 33.1.1.227 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-143 33.1.1.179 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-11 33.1.1.165 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-182 33.1.1.120 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-157 33.1.1.133 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-5 33.1.1.177 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-3 33.1.1.143 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-31 33.1.1.149 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-95 33.1.1.124 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-69 33.1.1.189 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-131 33.1.1.169 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-220 33.1.1.142 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-184 33.1.1.168 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-74 33.1.1.112 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-1 33.1.1.154 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-247 33.1.1.165 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-178 33.1.1.139 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-93 33.1.1.159 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-127 33.1.1.160 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-164 33.1.1.182 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-2 33.1.1.196 5s}]
Nov 27 19:10:28.553: INFO: perc50: 2s, perc90: 3s, perc99: 5s

Again in the density test. this case create one pod a time and observe its latency, which shows that scheduler latency is 2s and run-after-scheduler is 5s. Is that means scheduler is faster than other component is the system?

3. explanation

the conflict between two case is not a real conflict. scheduler is sequential across pod, but kubelet is naturally parallel.

for scheduler,

latency * throughput = 1

as scheduler has just one loop to process all pods

but for kubelet

latency * throughput = n(count of nodes)

maybe the latency of kubelet is near or slow than scheduler, but there are many kubelets run in system. If latency(scheduler) == latency(kubelet), and there are 500 nodes, the throughput(kubelet) == 500 times of throughput(scheduler). this explains the different result of these 2 tests.

When cluster scale, the throughput gap between scheduler and kubelet/apiserver will be bigger.

So it is necessary to make scheduler parallel across pod.

environment

20 phycial machine 40 core, 380G, with openstack

k8s-master(1VM) 8 core, 16 G

k8s-node(500VM) 2 core, 4G

etcd(5VM) 4 core, 8G

ubuntu 14.04

Go 1.4.3

Etcd 2.2.2

K8s master head of 12.7