Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): validates resource limits of pods that are allowed to run Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1 kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-01T06:22:44Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-01T06:22:44Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Local Setup OS (e.g. from /etc/os-release): VMware Photon 1.0" Kernel (e.g. ): Linux k8s-master-0--0a28-4fd8-b416- 4.4.41-1.ph1-esx -photon SMP Tue Jan 10 23:46:44 UTC 2017 x8664 GNU/Linux Install tools: Others: What happened: This e2e test (validates resource limits of pods that are allowed to run) failed. What you expected to happen: Should always pass. How to reproduce it (as minimally and precisely as possible): Run this test in a setup which node's cpu capacity is different than cpu allocatable. Anything else we need to know: I will submit a fix for this issue. Details about capacity and allocatable: In the local setup, capacity = 2000m and allocatable = 1900m. root [ ~ ]# kubectl describe node k8s-node-0--0a28-4fd8-b416- Name: k8s-node-0--0a28-4fd8-b416- Role: Labels: failure- failure- node- Annotations: Taints: <noneCreationTimestamp: Tue, 13 Jun 2017 10:46:07 +0000 Phase: Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- OutOfDisk False Thu, 15 Jun 2017 06:51:47 +0000 Tue, 13 Jun 2017 10:46:07 +0000 KubeletHasSufficientDisk kubelet has su MemoryPressure False Thu, 15 Jun 2017 06:51:47 +0000 Tue, 13 Jun 2017 10:46:07 +0000 KubeletHasSufficientMemory kubelet has su DiskPressure False Thu, 15 Jun 2017 06:51:47 +0000 Tue, 13 Jun 2017 10:46:07 +0000 KubeletHasNoDiskPressure kubelet has no Ready True Thu, 15 Jun 2017 06:51:47 +0000 Tue, 13 Jun 2017 10:46:07 +0000 KubeletReady kubelet is pos Addresses: 192.168.0.4,k8s-node-0--0a28-4fd8-b416- Capacity: cpu: 2 memory: 4048232Ki pods: 110 Allocatable: cpu: 1900m memory: 3445832Ki pods: 110 System Info: Machine ID: System UUID: 4217DB4D-954B-7D1A-014B-CB31E02B0679 Boot ID: -5c82-453a-9e03- Kernel Version: 4.4.41-1.ph1-esx OS Image: Debian GNU/Linux 8 (jessie) Operating System: linux Architecture: amd64 Container Runtime Version: docker://1.12.1 Kubelet Version: v1.6.1 Kube-Proxy Version: v1.6.1 ExternalID: -d6ea-4d62-958a- Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory --------- ---- ------------ ---------- --------------- ------ kube-system dnsmasq-z77g3 70m (3%) 100m (5%) 70Mi (2%) 170Mi kube-system flannel-k8s-node-0--0a28-4fd8-b416- 150m (7%) 300m (15%) 64M (1%) 500M ( kube-system kube-proxy-k8s-node-0--0a28-4fd8-b416- 150m (7%) 500m (26%) 64M (1%) 2G (56 kube-system kubedns-autoscaler--btwl6 20m (1%) 0 (0%) 10Mi (0%) 0 (0%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 390m (20%) 900m (47%) 206920Ki (6%) (75%) Events: <noneFull output of the test case: 04:31:52 [It] validates resource limits of pods that are allowed to run [Conformance] 04:31:52 04:31:52 Jun 13 11:37:49.256: INFO: Pod dnsmasq--sk20q requesting resource cpu=40m on Node k8s-node-1--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod dnsmasq-autoscaler--45s0t requesting resource cpu=20m on Node k8s-node-0--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod flannel-k8s-node-0--0a28-4fd8-b416- requesting resource cpu=150m on Node k8s-node-0--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod flannel-k8s-node-1--0a28-4fd8-b416- requesting resource cpu=150m on Node k8s-node-1--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod flannel-k8s-node-2--0a28-4fd8-b416- requesting resource cpu=150m on Node k8s-node-2--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod kube-proxy-k8s-node-0--0a28-4fd8-b416- requesting resource cpu=150m on Node k8s-node-0--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod kube-proxy-k8s-node-1--0a28-4fd8-b416- requesting resource cpu=150m on Node k8s-node-1--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod kube-proxy-k8s-node-2--0a28-4fd8-b416- requesting resource cpu=150m on Node k8s-node-2--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod kubedns--6d6k7 requesting resource cpu=150m on Node k8s-node-1--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Pod kubedns-autoscaler--btwl6 requesting resource cpu=20m on Node k8s-node-0--0a28-4fd8-b416- 04:31:52 Jun 13 11:37:49.256: INFO: Using pod capacity: 500m 04:31:52 Jun 13 11:37:49.256: INFO: Node: k8s-node-0--0a28-4fd8-b416- has cpu capacity: 1660m 04:31:52 Jun 13 11:37:49.256: INFO: Node: k8s-node-1--0a28-4fd8-b416- has cpu capacity: 1510m 04:31:52 Jun 13 11:37:49.256: INFO: Node: k8s-node-2--0a28-4fd8-b416- has cpu capacity: 1700m 04:31:52 STEP: Starting additional 9 Pods to fully saturate the cluster CPU and trying to start another one 04:31:53 Jun 13 11:37:50.582: INFO: Waiting for running... 04:41:53 Jun 13 11:47:50.611: INFO: Unexpected error occurred: Error waiting for 9 pods to be running - probably a timeout: Timeout while waiting for pods with labels "startPodsID=-502c-11e7-b75f-" to be running 04:41:53 [AfterEach] [] SchedulerPredicates [Serial] 04:41:53 04:41:53 STEP: Collecting events from namespace "e2e-tests-sched-pred-jhgng". 04:41:54 STEP: Found 35 events. 04:41:54 Jun 13 11:47:50.792: INFO: At 2017-06-13 11:37:59. +0000 UTC - event for overcommit-0: {default-scheduler } Scheduled: Successfully assigned overcommit-0 to k8s-node-2--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.793: INFO: At 2017-06-13 11:37:59. +0000 UTC - event for overcommit-1: {default-scheduler } Scheduled: Successfully assigned overcommit-1 to k8s-node-0--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.793: INFO: At 2017-06-13 11:37:59. +0000 UTC - event for overcommit-2: {default-scheduler } Scheduled: Successfully assigned overcommit-2 to k8s-node-1--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.793: INFO: At 2017-06-13 11:38:00 +0000 UTC - event for overcommit-8: {default-scheduler } FailedScheduling: No nodes are available that match all of the following predicates:: Insufficient cpu (3). 04:41:54 Jun 13 11:47:50.793: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-0: {kubelet k8s-node-2--0a28-4fd8-b416-} Pulled: Container image "" already present on machine 04:41:54 Jun 13 11:47:50.793: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-3: {default-scheduler } Scheduled: Successfully assigned overcommit-3 to k8s-node-0--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.793: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-0: {kubelet k8s-node-2--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-0: {kubelet k8s-node-2--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-4: {default-scheduler } Scheduled: Successfully assigned overcommit-4 to k8s-node-2--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-1: {kubelet k8s-node-0--0a28-4fd8-b416-} Pulled: Container image "" already present on machine 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-1: {kubelet k8s-node-0--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-5: {default-scheduler } Scheduled: Successfully assigned overcommit-5 to k8s-node-1--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-1: {kubelet k8s-node-0--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.794: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-6: {default-scheduler } Scheduled: Successfully assigned overcommit-6 to k8s-node-2--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-3: {kubelet k8s-node-0--0a28-4fd8-b416-} Pulled: Container image "" already present on machine 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-3: {kubelet k8s-node-0--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-7: {default-scheduler } Scheduled: Successfully assigned overcommit-7 to k8s-node-0--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-4: {kubelet k8s-node-2--0a28-4fd8-b416-} Pulled: Container image "" already present on machine 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-4: {kubelet k8s-node-2--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-3: {kubelet k8s-node-0--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-4: {kubelet k8s-node-2--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.795: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-6: {kubelet k8s-node-2--0a28-4fd8-b416-} Pulled: Container image "" already present on machine 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-2: {kubelet k8s-node-1--0a28-4fd8-b416-} Pulling: pulling image "" 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-6: {kubelet k8s-node-2--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-7: {kubelet k8s-node-0--0a28-4fd8-b416-} Pulled: Container image "" already present on machine 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-5: {kubelet k8s-node-1--0a28-4fd8-b416-} Pulling: pulling image "" 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:00. +0000 UTC - event for overcommit-7: {kubelet k8s-node-0--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:01. +0000 UTC - event for overcommit-6: {kubelet k8s-node-2--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:01. +0000 UTC - event for overcommit-7: {kubelet k8s-node-0--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.796: INFO: At 2017-06-13 11:38:06. +0000 UTC - event for overcommit-2: {kubelet k8s-node-1--0a28-4fd8-b416-} Pulled: Successfully pulled image "" 04:41:54 Jun 13 11:47:50.797: INFO: At 2017-06-13 11:38:06. +0000 UTC - event for overcommit-2: {kubelet k8s-node-1--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.797: INFO: At 2017-06-13 11:38:06. +0000 UTC - event for overcommit-2: {kubelet k8s-node-1--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:50.797: INFO: At 2017-06-13 11:38:09. +0000 UTC - event for overcommit-5: {kubelet k8s-node-1--0a28-4fd8-b416-} Pulled: Successfully pulled image "" 04:41:54 Jun 13 11:47:50.797: INFO: At 2017-06-13 11:38:09. +0000 UTC - event for overcommit-5: {kubelet k8s-node-1--0a28-4fd8-b416-} Created: Created container with docker id ; Security:[seccomp=unconfined] 04:41:54 Jun 13 11:47:50.797: INFO: At 2017-06-13 11:38:09. +0000 UTC - event for overcommit-5: {kubelet k8s-node-1--0a28-4fd8-b416-} Started: Started container with docker id 04:41:54 Jun 13 11:47:51.055: INFO: POD NODE PHASE GRACE CONDITIONS 04:41:54 Jun 13 11:47:51.055: INFO: overcommit-0 k8s-node-2--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC }] 04:41:54 Jun 13 11:47:51.055: INFO: overcommit-1 k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC }] 04:41:54 Jun 13 11:47:51.055: INFO: overcommit-2 k8s-node-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:07 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: overcommit-3 k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: overcommit-4 k8s-node-2--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:37:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: overcommit-5 k8s-node-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:10 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: overcommit-6 k8s-node-2--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: overcommit-7 k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: overcommit-8 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2017-06-13 11:38:00. +0000 UTC Unschedulable No nodes are available that match all of the following predicates:: Insufficient cpu (3).}] 04:41:54 Jun 13 11:47:51.056: INFO: dnsmasq--sk20q k8s-node-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:36 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:38 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:36 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: dnsmasq-autoscaler--45s0t k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:39 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:39 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: flannel-k8s-master-0--0a28-4fd8-b416- k8s-master-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:39 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:42 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:37 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: flannel-k8s-master-1--0a28-4fd8-b416- k8s-master-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:40 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:42 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:37 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: flannel-k8s-node-0--0a28-4fd8-b416- k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:43 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:38 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: flannel-k8s-node-1--0a28-4fd8-b416- k8s-node-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:39 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:42 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:37 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: flannel-k8s-node-2--0a28-4fd8-b416- k8s-node-2--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:55:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:56:12 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:56:10 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: kube-apiserver-k8s-master-0--0a28-4fd8-b416- k8s-master-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:57 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:56 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: kube-apiserver-k8s-master-1--0a28-4fd8-b416- k8s-master-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:58 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:56 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: kube-controller-manager-k8s-master-0--0a28-4fd8-b416- k8s-master-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:11 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:13 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:11 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: kube-controller-manager-k8s-master-1--0a28-4fd8-b416- k8s-master-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:13 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:13 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:11 +0000 UTC }] 04:41:54 Jun 13 11:47:51.056: INFO: kube-proxy-k8s-master-0--0a28-4fd8-b416- k8s-master-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:45 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:40 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kube-proxy-k8s-master-1--0a28-4fd8-b416- k8s-master-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:42 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:45 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:41 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kube-proxy-k8s-node-0--0a28-4fd8-b416- k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:45 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:40 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kube-proxy-k8s-node-1--0a28-4fd8-b416- k8s-node-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:43 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:45:39 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kube-proxy-k8s-node-2--0a28-4fd8-b416- k8s-node-2--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:54:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:56:12 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:56:10 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kube-scheduler-k8s-master-0--0a28-4fd8-b416- k8s-master-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:14 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:12 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kube-scheduler-k8s-master-1--0a28-4fd8-b416- k8s-master-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:13 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:46:12 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kubedns--6d6k7 k8s-node-1--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:08 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:48 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:08 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: kubedns-autoscaler--btwl6 k8s-node-0--0a28-4fd8-b416- Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:10 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:11 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-06-13 10:47:10 +0000 UTC }] 04:41:54 Jun 13 11:47:51.057: INFO: 04:41:54 Jun 13 11:47:51.198: INFO: 04:41:54 Logging node info for node k8s-master-0--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:51.320: INFO: Node Info: &Node{ObjectMeta:k8sioapimachinerypkgapismetav1.ObjectMeta{Name:k8s-master-0--0a28-4fd8-b416-,GenerateName:,Namespace:,SelfLink:/api/v1/nodesk8s-master-0--0a28-4fd8-b416-,UID:-5025-11e7-9536-,ResourceVersion:8623,Generation:0,CreationTimestamp:2017-06-13 10:46:02. +0000 UTC,DeletionTimestamp:<nil,DeletionGracePeriodSeconds:nil,Labels:map[string]string{ amd64, linux,failure- SDDC,failure- nova, k8s-master-0--0a28-4fd8-b416-,node- true,},Annotations:map[string]string{ 0, true,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:NodeSpec{PodCIDR:,ExternalID:-7d19-4ef8-b2ab-,ProviderID:openstack:///-7d19-4ef8-b2ab-,Unschedulable:true,Taints:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil} 2 DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{1900 -3} {<nil} 1900m DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Phase:,Conditions:[{OutOfDisk False 2017-06-13 11:47:58 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2017-06-13 11:47:58 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2017-06-13 11:47:58 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {Ready True 2017-06-13 11:47:58 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletReady kubelet is posting ready status}],Addresses:[{InternalIP 192.168.0.6} {ExternalIP 10.111.89.136} {Hostname k8s-master-0--0a28-4fd8-b416-}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:4217B741-B992-F08D-976D-EA97EF8D988D,BootID:-6927-407a-9b79-,KernelVersion:4.4.41-1.ph1-esx,OSImage:Debian GNU/Linux 8 (jessie),ContainerRuntimeVersion:docker://1.12.1,KubeletVersion:v1.6.1,KubeProxyVersion:v1.6.1,OperatingSystem:linux,Architecture:amd64,},Images:[{[10.111.89.130:/coreos/hyperkube 10.111.89.130:/coreos/hyperkube:v1.6.1coreos.0] } {[10.111.89.130:/nginx 10.111.89.130:/nginx:1.11.4] } {[10.111.89.130:/googlecontainers/kubedns-amd64 10.111.89.130:/googlecontainers/kubedns-amd64:1.7] } {[10.111.89.130:/coreos/flannel 10.111.89.130:/coreos/flannel:v0.6.2] } {[10.111.89.130:/googlecontainers/exechealthz-amd64 10.111.89.130:/googlecontainers/exechealthz-amd64:1.1] } {[10.111.89.130:/googlecontainers/kube-dnsmasq-amd64 10.111.89.130:/googlecontainers/kube-dnsmasq-amd64:1.3] } {[10.111.89.130:/busybox 10.111.89.130:/busybox:latest] } {[10.111.89.130:/googlecontainers/pause-amd64 10.111.89.130:/googlecontainers/pause-amd64:3.0] }],VolumesInUse:[],VolumesAttached:[],},} 04:41:54 Jun 13 11:47:51.321: INFO: 04:41:54 Logging kubelet events for node k8s-master-0--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:51.442: INFO: 04:41:54 Logging pods the kubelet thinks is on node k8s-master-0--0a28-4fd8-b416- 04:41:54 Jun 13 11:47:51.738: INFO: flannel-k8s-master-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:54 Jun 13 11:47:51.739: INFO: kube-proxy-k8s-master-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:54 Jun 13 11:47:51.739: INFO: kube-apiserver-k8s-master-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:54 Jun 13 11:47:51.739: INFO: kube-controller-manager-k8s-master-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:54 Jun 13 11:47:51.739: INFO: kube-scheduler-k8s-master-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:55 W0613 11:47:51. 275 ] Master node is not registered. Grabbing metrics from Scheduler and ControllerManager is disabled. 04:41:55 Jun 13 11:47:52.279: INFO: 04:41:55 Latency metrics for node k8s-master-0--0a28-4fd8-b416- 04:41:55 Jun 13 11:47:52.280: INFO: 04:41:55 Logging node info for node k8s-master-1--0a28-4fd8-b416- 04:41:55 Jun 13 11:47:52.390: INFO: Node Info: &Node{ObjectMeta:k8sioapimachinerypkgapismetav1.ObjectMeta{Name:k8s-master-1--0a28-4fd8-b416-,GenerateName:,Namespace:,SelfLink:/api/v1/nodesk8s-master-1--0a28-4fd8-b416-,UID:-5025-11e7-b8fd-,ResourceVersion:8620,Generation:0,CreationTimestamp:2017-06-13 10:46:02. +0000 UTC,DeletionTimestamp:<nil,DeletionGracePeriodSeconds:nil,Labels:map[string]string{ amd64, linux,failure- SDDC,failure- nova, k8s-master-1--0a28-4fd8-b416-,node- true,},Annotations:map[string]string{ 0, true,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:NodeSpec{PodCIDR:,ExternalID:-c340-4be7-893c-,ProviderID:openstack:///-c340-4be7-893c-,Unschedulable:true,Taints:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil} 2 DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{1900 -3} {<nil} 1900m DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Phase:,Conditions:[{OutOfDisk False 2017-06-13 11:47:56 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2017-06-13 11:47:56 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2017-06-13 11:47:56 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {Ready True 2017-06-13 11:47:56 +0000 UTC 2017-06-13 10:46:02 +0000 UTC KubeletReady kubelet is posting ready status}],Addresses:[{InternalIP 192.168.0.5} {Hostname k8s-master-1--0a28-4fd8-b416-}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:42179DBD-7BE0-07B4-49EF-DFFE428E61C9,BootID:-afe5-4d6b-8de4-,KernelVersion:4.4.41-1.ph1-esx,OSImage:Debian GNU/Linux 8 (jessie),ContainerRuntimeVersion:docker://1.12.1,KubeletVersion:v1.6.1,KubeProxyVersion:v1.6.1,OperatingSystem:linux,Architecture:amd64,},Images:[{[10.111.89.130:/coreos/hyperkube 10.111.89.130:/coreos/hyperkube:v1.6.1coreos.0] } {[10.111.89.130:/nginx 10.111.89.130:/nginx:1.11.4] } {[10.111.89.130:/googlecontainers/kubedns-amd64 10.111.89.130:/googlecontainers/kubedns-amd64:1.7] } {[10.111.89.130:/coreos/flannel 10.111.89.130:/coreos/flannel:v0.6.2] } {[10.111.89.130:/googlecontainers/exechealthz-amd64 10.111.89.130:/googlecontainers/exechealthz-amd64:1.1] } {[10.111.89.130:/googlecontainers/kube-dnsmasq-amd64 10.111.89.130:/googlecontainers/kube-dnsmasq-amd64:1.3] } {[10.111.89.130:/busybox 10.111.89.130:/busybox:latest] } {[10.111.89.130:/googlecontainers/pause-amd64 10.111.89.130:/googlecontainers/pause-amd64:3.0] }],VolumesInUse:[],VolumesAttached:[],},} 04:41:55 Jun 13 11:47:52.391: INFO: 04:41:55 Logging kubelet events for node k8s-master-1--0a28-4fd8-b416- 04:41:55 Jun 13 11:47:52.504: INFO: 04:41:55 Logging pods the kubelet thinks is on node k8s-master-1--0a28-4fd8-b416- 04:41:56 Jun 13 11:47:52.766: INFO: kube-scheduler-k8s-master-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:56 Jun 13 11:47:52.766: INFO: flannel-k8s-master-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:56 Jun 13 11:47:52.766: INFO: kube-proxy-k8s-master-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:56 Jun 13 11:47:52.766: INFO: kube-apiserver-k8s-master-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:56 Jun 13 11:47:52.766: INFO: kube-controller-manager-k8s-master-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:56 W0613 11:47:52. 275 ] Master node is not registered. Grabbing metrics from Scheduler and ControllerManager is disabled. 04:41:56 Jun 13 11:47:53.251: INFO: 04:41:56 Latency metrics for node k8s-master-1--0a28-4fd8-b416- 04:41:56 Jun 13 11:47:53.251: INFO: 04:41:56 Logging node info for node k8s-node-0--0a28-4fd8-b416- 04:41:56 Jun 13 11:47:53.367: INFO: Node Info: &Node{ObjectMeta:k8sioapimachinerypkgapismetav1.ObjectMeta{Name:k8s-node-0--0a28-4fd8-b416-,GenerateName:,Namespace:,SelfLink:/api/v1/nodesk8s-node-0--0a28-4fd8-b416-,UID:-5025-11e7-b8fd-,ResourceVersion:8615,Generation:0,CreationTimestamp:2017-06-13 10:46:07. +0000 UTC,DeletionTimestamp:<nil,DeletionGracePeriodSeconds:nil,Labels:map[string]string{ amd64, linux,failure- SDDC,failure- nova, k8s-node-0--0a28-4fd8-b416-,node- true,},Annotations:map[string]string{ 0, true,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:NodeSpec{PodCIDR:,ExternalID:-d6ea-4d62-958a-,ProviderID:openstack:///-d6ea-4d62-958a-,Unschedulable:false,Taints:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil} 2 DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{1900 -3} {<nil} 1900m DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Phase:,Conditions:[{OutOfDisk False 2017-06-13 11:47:53 +0000 UTC 2017-06-13 10:46:07 +0000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2017-06-13 11:47:53 +0000 UTC 2017-06-13 10:46:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2017-06-13 11:47:53 +0000 UTC 2017-06-13 10:46:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {Ready True 2017-06-13 11:47:53 +0000 UTC 2017-06-13 10:46:07 +0000 UTC KubeletReady kubelet is posting ready status}],Addresses:[{InternalIP 192.168.0.4} {Hostname k8s-node-0--0a28-4fd8-b416-}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:4217DB4D-954B-7D1A-014B-CB31E02B0679,BootID:-5c82-453a-9e03-,KernelVersion:4.4.41-1.ph1-esx,OSImage:Debian GNU/Linux 8 (jessie),ContainerRuntimeVersion:docker://1.12.1,KubeletVersion:v1.6.1,KubeProxyVersion:v1.6.1,OperatingSystem:linux,Architecture:amd64,},Images:[{[10.111.89.130:/coreos/hyperkube 10.111.89.130:/coreos/hyperkube:v1.6.1coreos.0] } {[ ] } {[10.111.89.130:/nginx 10.111.89.130:/nginx:1.11.4] } {[ ] } {[10.111.89.130:/googlecontainers/kubedns-amd64 10.111.89.130:/googlecontainers/kubedns-amd64:1.7] } {[ ] } {[10.111.89.130:/coreos/flannel 10.111.89.130:/coreos/flannel:v0.6.2] } {[10.111.89.130:/googlecontainers/exechealthz-amd64 10.111.89.130:/googlecontainers/exechealthz-amd64:1.1] } {[ ] } {[ ] } {[10.111.89.130:/googlecontainers/kube-dnsmasq-amd64 10.111.89.130:/googlecontainers/kube-dnsmasq-amd64:1.3] } {[ ] } {[ ] } {[ ] } {[10.111.89.130:/busybox 10.111.89.130:/busybox:latest] } {[10.111.89.130:/googlecontainers/pause-amd64 10.111.89.130:/googlecontainers/pause-amd64:3.0 ] }],VolumesInUse:[],VolumesAttached:[],},} 04:41:56 Jun 13 11:47:53.368: INFO: 04:41:56 Logging kubelet events for node k8s-node-0--0a28-4fd8-b416- 04:41:56 Jun 13 11:47:53.489: INFO: 04:41:56 Logging pods the kubelet thinks is on node k8s-node-0--0a28-4fd8-b416- 04:41:57 Jun 13 11:47:53.776: INFO: flannel-k8s-node-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:57 Jun 13 11:47:53.776: INFO: kube-proxy-k8s-node-0--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:57 Jun 13 11:47:53.777: INFO: dnsmasq-autoscaler--45s0t started at 2017-06-13 10:46:39 +0000 UTC (0+1 container statuses recorded) 04:41:57 Jun 13 11:47:53.777: INFO: Container autoscaler ready: true, restart count 0 04:41:57 Jun 13 11:47:53.777: INFO: kubedns-autoscaler--btwl6 started at 2017-06-13 10:47:10 +0000 UTC (0+1 container statuses recorded) 04:41:57 Jun 13 11:47:53.777: INFO: Container autoscaler ready: true, restart count 0 04:41:57 Jun 13 11:47:53.777: INFO: overcommit-1 started at 2017-06-13 11:37:59 +0000 UTC (0+1 container statuses recorded) 04:41:57 Jun 13 11:47:53.777: INFO: Container overcommit-1 ready: true, restart count 0 04:41:57 Jun 13 11:47:53.778: INFO: overcommit-3 started at 2017-06-13 11:37:59 +0000 UTC (0+1 container statuses recorded) 04:41:57 Jun 13 11:47:53.778: INFO: Container overcommit-3 ready: true, restart count 0 04:41:57 Jun 13 11:47:53.778: INFO: overcommit-7 started at 2017-06-13 11:38:00 +0000 UTC (0+1 container statuses recorded) 04:41:57 Jun 13 11:47:53.778: INFO: Container overcommit-7 ready: true, restart count 0 04:41:57 W0613 11:47:53. 275 ] Master node is not registered. Grabbing metrics from Scheduler and ControllerManager is disabled. 04:41:57 Jun 13 11:47:54.336: INFO: 04:41:57 Latency metrics for node k8s-node-0--0a28-4fd8-b416- 04:41:57 Jun 13 11:47:54.336: INFO: 04:41:57 Logging node info for node k8s-node-1--0a28-4fd8-b416- 04:41:57 Jun 13 11:47:54.454: INFO: Node Info: &Node{ObjectMeta:k8sioapimachinerypkgapismetav1.ObjectMeta{Name:k8s-node-1--0a28-4fd8-b416-,GenerateName:,Namespace:,SelfLink:/api/v1/nodesk8s-node-1--0a28-4fd8-b416-,UID:-5025-11e7-9536-,ResourceVersion:8626,Generation:0,CreationTimestamp:2017-06-13 10:46:01. +0000 UTC,DeletionTimestamp:<nil,DeletionGracePeriodSeconds:nil,Labels:map[string]string{ amd64, linux,failure- SDDC,failure- nova, k8s-node-1--0a28-4fd8-b416-,node- true,},Annotations:map[string]string{ 0, true,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:NodeSpec{PodCIDR:,ExternalID:-f999-4cee-8c49-,ProviderID:openstack:///-f999-4cee-8c49-,Unschedulable:false,Taints:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil} 2 DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{1900 -3} {<nil} 1900m DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Phase:,Conditions:[{OutOfDisk False 2017-06-13 11:48:01 +0000 UTC 2017-06-13 10:46:01 +0000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2017-06-13 11:48:01 +0000 UTC 2017-06-13 10:46:01 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2017-06-13 11:48:01 +0000 UTC 2017-06-13 10:46:01 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {Ready True 2017-06-13 11:48:01 +0000 UTC 2017-06-13 10:46:01 +0000 UTC KubeletReady kubelet is posting ready status}],Addresses:[{InternalIP 192.168.0.3} {Hostname k8s-node-1--0a28-4fd8-b416-}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:4217527B-BE59-4714-EEC8-8FCFC4392E6F,BootID:-458f-4e70-ab5d-,KernelVersion:4.4.41-1.ph1-esx,OSImage:Debian GNU/Linux 8 (jessie),ContainerRuntimeVersion:docker://1.12.1,KubeletVersion:v1.6.1,KubeProxyVersion:v1.6.1,OperatingSystem:linux,Architecture:amd64,},Images:[{[10.111.89.130:/coreos/hyperkube 10.111.89.130:/coreos/hyperkube:v1.6.1coreos.0] } {[ ] } {[10.111.89.130:/nginx 10.111.89.130:/nginx:1.11.4] } {[10.111.89.130:/googlecontainers/kubedns-amd64 10.111.89.130:/googlecontainers/kubedns-amd64:1.7] } {[10.111.89.130:/googlecontainers/kubedns-amd64 10.111.89.130:/googlecontainers/kubedns-amd64:1.9] } {[10.111.89.130:/coreos/flannel 10.111.89.130:/coreos/flannel:v0.6.2] } {[10.111.89.130:/googlecontainers/exechealthz-amd64 10.111.89.130:/googlecontainers/exechealthz-amd64:1.1] } {[ ] } {[10.111.89.130:/andyshinn/dnsmasq 10.111.89.130:/andyshinn/dnsmasq:2.72] } {[10.111.89.130:/googlecontainers/kube-dnsmasq-amd64 10.111.89.130:/googlecontainers/kube-dnsmasq-amd64:1.3] } {[ ] } {[ ] } {[ ] } {[10.111.89.130:/busybox 10.111.89.130:/busybox:latest] } {[10.111.89.130:/googlecontainers/pause-amd64 10.111.89.130:/googlecontainers/pause-amd64:3.0 ] }],VolumesInUse:[],VolumesAttached:[],},} 04:41:57 Jun 13 11:47:54.454: INFO: 04:41:57 Logging kubelet events for node k8s-node-1--0a28-4fd8-b416- 04:41:57 Jun 13 11:47:54.568: INFO: 04:41:57 Logging pods the kubelet thinks is on node k8s-node-1--0a28-4fd8-b416- 04:41:58 Jun 13 11:47:54.812: INFO: overcommit-5 started at 2017-06-13 11:38:00 +0000 UTC (0+1 container statuses recorded) 04:41:58 Jun 13 11:47:54.812: INFO: Container overcommit-5 ready: true, restart count 0 04:41:58 Jun 13 11:47:54.813: INFO: flannel-k8s-node-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:58 Jun 13 11:47:54.813: INFO: kube-proxy-k8s-node-1--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:58 Jun 13 11:47:54.813: INFO: dnsmasq--sk20q started at 2017-06-13 10:46:36 +0000 UTC (0+1 container statuses recorded) 04:41:58 Jun 13 11:47:54.813: INFO: Container dnsmasq ready: true, restart count 0 04:41:58 Jun 13 11:47:54.813: INFO: kubedns--6d6k7 started at 2017-06-13 10:47:08 +0000 UTC (0+3 container statuses recorded) 04:41:58 Jun 13 11:47:54.813: INFO: Container dnsmasq ready: true, restart count 0 04:41:58 Jun 13 11:47:54.813: INFO: Container healthz ready: true, restart count 0 04:41:58 Jun 13 11:47:54.813: INFO: Container kubedns ready: true, restart count 0 04:41:58 Jun 13 11:47:54.813: INFO: overcommit-2 started at 2017-06-13 11:37:59 +0000 UTC (0+1 container statuses recorded) 04:41:58 Jun 13 11:47:54.813: INFO: Container overcommit-2 ready: true, restart count 0 04:41:58 W0613 11:47:54. 275 ] Master node is not registered. Grabbing metrics from Scheduler and ControllerManager is disabled. 04:41:58 Jun 13 11:47:55.330: INFO: 04:41:58 Latency metrics for node k8s-node-1--0a28-4fd8-b416- 04:41:58 Jun 13 11:47:55.331: INFO: 04:41:58 Logging node info for node k8s-node-2--0a28-4fd8-b416- 04:41:58 Jun 13 11:47:55.455: INFO: Node Info: &Node{ObjectMeta:k8sioapimachinerypkgapismetav1.ObjectMeta{Name:k8s-node-2--0a28-4fd8-b416-,GenerateName:,Namespace:,SelfLink:/api/v1/nodesk8s-node-2--0a28-4fd8-b416-,UID:-5026-11e7-b8fd-,ResourceVersion:8632,Generation:0,CreationTimestamp:2017-06-13 10:54:08. +0000 UTC,DeletionTimestamp:<nil,DeletionGracePeriodSeconds:nil,Labels:map[string]string{ amd64, linux,failure- SDDC,failure- nova, k8s-node-2--0a28-4fd8-b416-,node- true,},Annotations:map[string]string{ 0, true,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:NodeSpec{PodCIDR:,ExternalID:-7a1f-4ed2-bc75-,ProviderID:openstack:///-7a1f-4ed2-bc75-,Unschedulable:false,Taints:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil} 2 DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{1900 -3} {<nil} 1900m DecimalSI},memory: {{ 0} {<nil} BinarySI},pods: {{110 0} {<nil} 110 DecimalSI},},Phase:,Conditions:[{OutOfDisk False 2017-06-13 11:48:05 +0000 UTC 2017-06-13 10:54:08 +0000 UTC KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2017-06-13 11:48:05 +0000 UTC 2017-06-13 10:54:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2017-06-13 11:48:05 +0000 UTC 2017-06-13 10:54:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {Ready True 2017-06-13 11:48:05 +0000 UTC 2017-06-13 10:54:08 +0000 UTC KubeletReady kubelet is posting ready status}],Addresses:[{InternalIP 192.168.0.9} {Hostname k8s-node-2--0a28-4fd8-b416-}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:42178CEB-F2F4-C246-9DC8-EB18C5318F31,BootID:-fbd2-432c-b2d4-,KernelVersion:4.4.41-1.ph1-esx,OSImage:Debian GNU/Linux 8 (jessie),ContainerRuntimeVersion:docker://1.12.1,KubeletVersion:v1.6.1,KubeProxyVersion:v1.6.1,OperatingSystem:linux,Architecture:amd64,},Images:[{[10.111.89.130:/coreos/hyperkube 10.111.89.130:/coreos/hyperkube:v1.6.1coreos.0] } {[ ] } {[ ] } {[10.111.89.130:/nginx 10.111.89.130:/nginx:1.11.4] } {[ ] } {[ ] } {[10.111.89.130:/googlecontainers/kubedns-amd64 10.111.89.130:/googlecontainers/kubedns-amd64:1.7] } {[10.111.89.130:/coreos/flannel 10.111.89.130:/coreos/flannel:v0.6.2] } {[ ] } {[10.111.89.130:/googlecontainers/exechealthz-amd64 10.111.89.130:/googlecontainers/exechealthz-amd64:1.1] } {[ ] } {[ ] } {[10.111.89.130:/googlecontainers/kube-dnsmasq-amd64 10.111.89.130:/googlecontainers/kube-dnsmasq-amd64:1.3] } {[ ] } {[ ] } {[ ] } {[ ] } {[ ] } {[ ] } {[ ] } {[ ] } {[ ] } {[10.111.89.130:/busybox 10.111.89.130:/busybox:latest] } {[10.111.89.130:/googlecontainers/pause-amd64 10.111.89.130:/googlecontainers/pause-amd64:3.0 ] }],VolumesInUse:[],VolumesAttached:[],},} 04:41:58 Jun 13 11:47:55.455: INFO: 04:41:58 Logging kubelet events for node k8s-node-2--0a28-4fd8-b416- 04:41:58 Jun 13 11:47:55.569: INFO: 04:41:58 Logging pods the kubelet thinks is on node k8s-node-2--0a28-4fd8-b416- 04:41:59 Jun 13 11:47:55.819: INFO: kube-proxy-k8s-node-2--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:59 Jun 13 11:47:55.819: INFO: overcommit-0 started at 2017-06-13 11:37:59 +0000 UTC (0+1 container statuses recorded) 04:41:59 Jun 13 11:47:55.819: INFO: Container overcommit-0 ready: true, restart count 0 04:41:59 Jun 13 11:47:55.819: INFO: overcommit-4 started at 2017-06-13 11:37:59 +0000 UTC (0+1 container statuses recorded) 04:41:59 Jun 13 11:47:55.819: INFO: Container overcommit-4 ready: true, restart count 0 04:41:59 Jun 13 11:47:55.820: INFO: overcommit-6 started at 2017-06-13 11:38:00 +0000 UTC (0+1 container statuses recorded) 04:41:59 Jun 13 11:47:55.820: INFO: Container overcommit-6 ready: true, restart count 0 04:41:59 Jun 13 11:47:55.820: INFO: flannel-k8s-node-2--0a28-4fd8-b416- started at <nil(0+0 container statuses recorded) 04:41:59 W0613 11:47:55. 275 ] Master node is not registered. Grabbing metrics from Scheduler and ControllerManager is disabled. 04:41:59 Jun 13 11:47:56.375: INFO: 04:41:59 Latency metrics for node k8s-node-2--0a28-4fd8-b416- 04:41:59 STEP: Dumping a list of prepulled images on each node 04:41:59 Jun 13 11:47:56.496: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready 04:41:59 STEP: Destroying namespace "e2e-tests-sched-pred-jhgng" for this suite. 04:42:29 Jun 13 11:48:25.881: INFO: namespace: e2e-tests-sched-pred-jhgng, resource: bindings, ignored listing per whitelist 04:42:29 [AfterEach] [] SchedulerPredicates [Serial] 04:42:29 04:42:29 04:42:29 • Failure [700.011 seconds] 04:42:29 [] SchedulerPredicates [Serial] 04:42:29 04:42:29 validates resource limits of pods that are allowed to run [Conformance] [It] 04:42:29 04:42:29 04:42:29 Expected error: 04:42:29 <*errors.errorString | 0xc421280120: { 04:42:29 s: "Error waiting for 9 pods to be running - probably a timeout: Timeout while waiting for pods with labels "startPodsID=-502c-11e7-b75f-" to be running", 04:42:29 } 04:42:29 Error waiting for 9 pods to be running - probably a timeout: Timeout while waiting for pods with labels "startPodsID=-502c-11e7-b75f-" to be running 04:42:29 not to have occurred 04:42:29 04:42:29 04:42:29 ------------------------------ 04:42:29 S
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig scheduling
/sig testing

Discussion:
// It assumes that cluster add-on pods stay stable and cannot be run in parallel with any other test that touches Nodes or Pods. // It is so because we need to have precise control on what's running in the cluster. It("validates resource limits of pods that are allowed to run [Conformance]", func() { <del> nodeMaxCapacity := int64(0) </del> <ins> nodeMaxAllocatable := int64(0) </ins> <del> nodeToCapacityMap := make(map[string]int64) </del> <ins> nodeToAllocatableMap := make(map[string]int64) </ins> for _, node := range nodeList.Items { <del> capacity, found := node.Status.Capacity["cpu"] </del> <ins> allocatable, found := node.Status.Allocatable["cpu"] </ins> Expect(found).To(Equal(true)) <del> nodeToCapacityMap[node.Name] = capacity.MilliValue() if nodeMaxCapacity < capacity.MilliValue() { nodeMaxCapacity = capacity.MilliValue() </del> <ins> nodeToAllocatableMap[node.Name] = allocatable.MilliValue() if nodeMaxAllocatable < allocatable.MilliValue() { nodeMaxAllocatable = allocatable.MilliValue() </ins> } } framework.WaitForStableCluster(cs, masterNodes)

pods, err := cs.Core().Pods(metav1.NamespaceAll).List(metav1.ListOptions{}) framework.ExpectNoError(err) for _, pod := range pods.Items { <del> _, found := nodeToCapacityMap[pod.Spec.NodeName] </del> <ins> _, found := nodeToAllocatableMap[pod.Spec.NodeName] </ins> if found && pod.Status.Phase != v1.PodSucceeded && pod.Status.Phase != v1.PodFailed { framework.Logf("Pod %v requesting resource cpu=%vm on Node %v", pod.Name, getRequestedCPU(pod), pod.Spec.NodeName) <del> nodeToCapacityMap[pod.Spec.NodeName] -= getRequestedCPU(pod) </del> <ins> nodeToAllocatableMap[pod.Spec.NodeName] -= getRequestedCPU(pod) </ins> } } var podsNeededForSaturation int <del> milliCpuPerPod := nodeMaxCapacity / maxNumberOfPods </del> <ins> milliCpuPerPod := nodeMaxAllocatable / maxNumberOfPods </ins> if milliCpuPerPod < minPodCPURequest { milliCpuPerPod = minPodCPURequest } framework.Logf("Using pod capacity: %vm", milliCpuPerPod) <del> for name, leftCapacity := range nodeToCapacityMap { framework.Logf("Node: %v has cpu capacity: %vm", name, leftCapacity) podsNeededForSaturation += (int)(leftCapacity / milliCpuPerPod) </del> <ins> for name, leftAllocatable := range nodeToAllocatableMap { framework.Logf("Node: %v has cpu allocatable: %vm", name, leftAllocatable) podsNeededForSaturation += (int)(leftAllocatable / milliCpuPerPod) </ins> } By(fmt.Sprintf("Starting additional %v Pods to fully saturate the cluster CPU and trying to start another one", podsNeededForSaturation))