Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

NO, it is not a request of help, this is a BUG REPORT

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

HPA not down-scaling deployment

HPA not able to get load info

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Tested to repeat on 1.3.5 and on 1.3.6

Environment:

Cloud provider or hardware configuration: GCE/GKE managed

OS (e.g. from /etc/os-release): PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"

Kernel (e.g. uname -a): Linux gke-k8s-test1-default-pool-72fe9503-8tvm 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

Install tools: gcloud

Others: N/A

What happened:

We have development cluster with one of node pools of --min-nodes=2 and --max-nodes=3.

Normally cluster operates on 2 nodes in this pool.

Deployment under question specifically has a nodeSelector set to allocate nodes only to this pool.

Deployment under question has an HPA configured with maximum pod count of 20.

We observed a spike in the load, where HPA decided to allocate all 20 pods.

Due to CPU resource constraint, it triggered an additional node allocation: everything was as-expected at this time.

After additional node allocation, it was STILL NOT ENOUGH CPU to start all pods, but now limit on number of nodes was reached and pods were stuck in "Pending" state: as-expected.

What went wrong: after load was removed, HPA did not want to scale back to 1-2 pods we see normally running.

We found that HPA can not get CPU consumption info for some pods:

4m 4m 2 {horizontal-pod-autoscaler } Warning FailedGetMetrics failed to get CPU consumption and request: metrics obtained for 20/22 of pods
4m 4m 2 {horizontal-pod-autoscaler } Warning FailedComputeReplicas failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 20/22 of pods

It was suspicious it reports more pods than we see with "kubectl get pod -l ...", so we found the missing ones to be "stuck" and waiting to be reaped:

kubectl get pod -a -l run=cm-worker
NAME READY STATUS RESTARTS AGE
cm-worker-2850697223-1wtmo 1/1 Running 0 11m
cm-worker-2850697223-9evqk 1/1 Running 0 11m
cm-worker-2850697223-dfhiu 0/1 OutOfCPU 0 5h
cm-worker-2850697223-f6h9o 1/1 Running 0 11m
[...]
cm-worker-2850697223-wogui 1/1 Running 0 11m
cm-worker-2850697223-zdxjg 0/1 OutOfCPU 0 4h

At this point, we upgraded master from 1.3.5 to 1.3.6, but it did not resolve the situation.

Upon manual kill on these pods, cluster and HPA started to operate as appropriate: HPA scaled down and cluster scaled down node it did not need.

What you expected to happen:

It was expected that Kubernetes would properly discover and reap failed pods.

It was expected that HPA would not consider dead pods for query of current load.

How to reproduce it (as minimally and precisely as possible):

We are not sure we can reproduce, so we investigated to the root cause right in place.

Anything else do we need to know:

N/A

Issue 2:
Environment:

Kubernetes version: 1.3.5

Cloud provider or hardware configuration: Google Container Engine

Kernel: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

What happened:

We're using a HorizontalPodAutoscaler in our cluster to scale pods based on CPU load. While this is working for some deployments, one deployment seems to be in a broken state where the HPA tries to get cpu consumption from heapster for pods that do not exist.

The HPA was working for said deployment but started to misbehave after a scaling event.

Result from kubectl get pods

…
gosu-main-api-v1-3735438065-5n8t7 2/2 Running 1 2h
gosu-main-api-v1-3735438065-j027w 2/2 Running 1 2h
gosu-main-api-v1-3735438065-rydr6 2/2 Running 1 2h
gosu-main-api-v1-3735438065-vf59h 2/2 Running 1 2h
gosu-main-api-v1-3735438065-w4hzm 2/2 Running 1 2h
…

Result from kubectl get hpa

NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
…
gosu-main-api-v1 Deployment/gosu-main-api-v1 75% <waiting> 3 20 28m
…

Result from kubectl describe hpa gosu-main-api-v1

Name: gosu-main-api-v1
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sat, 27 Aug 2016 17:17:31 -0700
Reference: Deployment/gosu-main-api-v1
Target CPU utilization: 75%
Current CPU utilization: <unset>
Min replicas: 3
Max replicas: 20
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
18m 16m 4 {horizontal-pod-autoscaler } Warning FailedGetMetrics failed to get CPU consumption and request: metrics obtained for 0/12 of pods
18m 16m 4 {horizontal-pod-autoscaler } Warning FailedComputeReplicas failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/12 of pods
30m 30s 58 {horizontal-pod-autoscaler } Warning FailedGetMetrics failed to get CPU consumption and request: metrics obtained for 5/12 of pods
30m 30s 58 {horizontal-pod-autoscaler } Warning FailedComputeReplicas failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 5/12 of pods

Excerpt from heapster logs

I0828 00:47:34.525902 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:47:34.526057 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:47:34.526107 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:47:34.526165 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:47:34.526249 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:47:34.526304 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:47:34.526387 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:47:34.526444 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:47:34.526493 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:47:34.526548 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:47:34.526591 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:47:34.526671 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:47:34.526754 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:47:34.526822 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584791 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:48:04.584816 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:48:04.584837 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:48:04.584866 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:48:04.584893 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584961 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584986 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:48:04.585005 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:48:04.585048 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:48:04.585066 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:48:04.585097 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:48:04.585186 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:48:04.585201 1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:48:04.585206 1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-qscb9

What you expected to happen:

The deployment is currently set to 5 replicas, so I would expect for the HPA to get the resource consumption for the 5 pods that are scheduled.

How to reproduce it:

Cannot reliably bring other deployments into this broken state. Had the same issue with another deployment, but it started to work again without outer influence (maybe after we updated the deployment)