Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): kinda

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): OutOfCPU

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.2", GitCommit:"9bafa3400a77c14ee50782bb05f9efc5c91b3185", GitTreeState:"clean", BuildDate:"2016-07-17T18:30:39Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}

(cluster is now torn down, but is also 1.3.2 with the same commit)

Environment:

Cloud provider or hardware configuration: AWS - 80 c4.4xlarge nodes

OS (e.g. from /etc/os-release): CoreOS 1068.8.0 (stable)

What happened:

Scheduling roughly 10k Pods on 80 nodes with 16 cores each, lots of pods end up having issues with being scheduled and end up with an OutOfCPU error. In the case below, we were experimenting with increasing limits to beyond the node capacity, but we found the behaviour on the Guaranteed (limits and requests set to 14) and Burstable classes. To me, it appears like the scheduler is having problems keeping track of the resources in use and is trying to over-provision the node.

Here's the describe pods Note that the extra 100m CPU reported as being used is from system containers (fluentd, etc.)

$ kubectl describe pods worker-7f1df241-5c3f-45cd-8df3-6caa2c972f68
Name: worker-7f1df241-5c3f-45cd-8df3-6caa2c972f68
Namespace: default
Node: <REDACTED>
Start Time: Tue, 16 Aug 2016 13:02:38 +0800
Labels: sample=<REDACTED>,submitter=<REDACTED>
Status: Failed
Reason: OutOfCPU
Message: Pod Node didn't have enough resource: CPU, requested: 14000, used: 14100, capacity: 16000
IP:
Controllers: <none>
Containers:
pod-template:
Image: <REDACTED>
Port:
Command:
<REDACTED>
QoS Tier:
cpu: Burstable
memory: BestEffort
Limits:
cpu: 32
Requests:
cpu: 14
Volumes:
workdir:
Type: HostPath (bare host directory volume)
Path: /mnt
default-token-u3fnm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-u3fnm
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
31m 31m 1 {kubelet <REDACTED>} Warning OutOfCPU Node didn't have enough resource: CPU, requested: 14000, used: 14100, capacity: 16000

What you expected to happen:

Pod to stay in a Pending state whilst resources are not available for scheduling.

How to reproduce it (as minimally and precisely as possible):

Submit 10k jobs requiring 14 CPUs on a cluster containing 16 cores each. Note that in this case, we are submitting all 10k jobs within a 1 minute time window.

Anything else do we need to know:

Issue 2:
Is this a request for help?: No

What keywords did you search in Kubernetes issues before filing this one? OutOfCpu

Issue #29846 may be related; however, #29846 is not related to cluster autoscaling and is limited to an OutOfCpu condition.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.4.0

Environment: Google GKE

Cloud provider or hardware configuration: Google

OS (e.g. from /etc/os-release): Google Container-VM Image V55

Kernel (e.g. uname -a): Linux gke-omega-cluster-default-pool-d07fc610-clps 4.4.14+ #1 SMP Tue Sep 20 10:32:07 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

Install tools:

Others:

What happened:

Using cluster of n1-standard-1 nodes, 1 initial node and enabled cluster autoscaling, I started 34 pods requesting 100m CPU each. Since each node has only 1 CPU and 200m CPU is required for Kubernetes services on each node (300m CPU on the first node), each node can run 8 pods (7 for the first node). All 34 pods would, therefore, required 5 nodes. The sequence of events:

Around 15:06:30: All pods are created. As expected, most pods remain in Pending state.

15:06:42: Kubernetes correctly triggers scale-up; however, the scheduler only requests a single node.

15:07:34: Two pods ("out-of-cpurf679" and "out-of-cpuwi5ge") are killed with "Deleted by rescheduler in order to schedule critical pod kube-system_heapster-v1.2.0-3455740371-j973x".

15:09:33: Scale-up is triggered again, scaling from 2 to 3 nodes.

15:11:48: Two other pods ("out-of-cpu3gsro" and "out-of-cpun30p1") fail with OutOfCpu when 10 pods are assigned to the newly added 3rd node.

15:12:31: Next scale-up, going from 3 to 4 nodes.

Eventually, 30 pods are running, 2 pods are in OutOfCpu state, and 2 pods have disappeared.

15:15:16: Another scale-up is triggered from 4 to 5 nodes. The 5th node remains idle as there are no more pods to schedule.

Please refer to this zip file for details:

out-of-cpu-bug.zip

"out-of-cpu-bug.yaml" is the YAML that was used to run the experiment through the following commands:

1> gcloud container clusters create omega-cluster --machine-type=n1-standard-1 --zone us-central1-c --num-nodes 1 --enable-autoscaling --min-nodes=1 --max-nodes=8

2> bash -c 'for i in {1..34}; do kubectl create -f out-of-cpu-bug.yaml; done'

"out-of-cpu-bug-events.txt" is the capture of events from 'kubectl get events -w' up to the final scale-up. "out-of-cpu-bug-pods1.txt" shows the output of 'kubectl get nodes,pods -a' after the OutOfCpu condition of two pods. Note that only 32 pods show up, because 2 pods have been killed earlier and have "disappeared". "out-of-cpu-bug.pods2.txt" shows the nodes and pods after the final scale-up. The new node "gke-omega-cluster-default-pool-d07fc610-w79v" is unused.

What you expected to happen:

Scale-up should have been triggered to bring the cluster from 1 to 5 nodes in a single step.

Pods "out-of-cpurf679" and "out-of-cpuwi5ge" should not have been killed and disappeared.

Pods "out-of-cpu3gsro" and "out-of-cpun30p1" should not have been scheduled onto a full node and should not have received an OutOfCpu condition.

Scale-up should not have produced an idle node.

How to reproduce it (as minimally and precisely as possible):

See above. Create a cluster on Google with

gcloud container clusters create omega-cluster --machine-type=n1-standard-1 --zone us-central1-c --num-nodes 1 --enable-autoscaling --min-nodes=1 --max-nodes=8

Schedule pods with the supplied YAML:

bash -c 'for i in {1..34}; do kubectl create -f out-of-cpu-bug.yaml; done'

The exact numbers don't matter.

Anything else do we need to know:

The bug(s) appear to include some race condition as the error does not always (although frequently) happen. The more pods that are being created, the more likely the situation is observed. The YAML has a commented-out section to request pod affinity. This seems to increase the likelyhood of OutOfCpu failures. Nonetheless, the above commands will show the failure with a 1 in 2 chance. Increase the number of pods to 60 or more and the problem occurs almost every time. I consider this problem a showstopper for any serious cluster autoscaling application.