Change:
What happened: HPA always has a target of and events that say: There is a single container in the pods and it has resource requests and limits set. The metrics-server is running All pods have metrics show in All pods have metrics in Here's the HPA in YAML: What you expected to happen: No in HPA target How to reproduce it (as minimally and precisely as possible): I can't be sure. It's only a single HPA in our cluster. 10 other HPAs are working OK. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.6 Cloud provider or hardware configuration: EKS
/sig autoscaling
could you please share the yaml content of the pod? It seems the pod's cpu request is not set.
Sure. Here's from the deployment (): This shows there's only a single container in these pods. Here's a list of pods: And resources from all pods (): That just repeats 14 times, once for each pod. And then the 3 completed pods show as:
Ahhh, I deleted those pods and suddenly the HPA is back in action:
But these pods are not from the deployment that is specified in the HPA, they are created from a . Sure they don't have set but they should be ignored by the HPA, right? Here's the pod JSON from one: I will test creating more pods WITHOUT set and see if the issue returns. And then test creating more pods WITH and see if it's OK.
It is a little weird. How could I reproduce it?
HPA use only selector to filter pods without checking ownership, I think this is a horrible mistake!
It might be the cause of this issue. Would you like to send a PR for it? If not, I could try to fix it. Could you share the yaml of and , especially the labels part? cc
I am not working on this. :)
Here the job: And here's the deployment: I deleted all the pods and the HPA started working again. And then when new pods are present, I see these events:
Maybe worth noting that even though I got these events and the HPA is saying it still didn't transition to after 10 minutes. But if I create the HPA while there is these pods then it stays in state.
Any update
... I am not working on this issue.... are you working on this? Hope you dit not get an impression that I am working on this.....
Ah sorry, my mistake!
Ah I missed that :( I'll try to fix it.
Experiencing this same issue
I checked the code, but not sure whether it is intended to get pods list just by labels and not check the owner reference. Could you please help confirm it? Thanks! If it is not intended, I could send a PR to fix it.
I get the same issue: Here is my HPA spec: Wonder why this only happens to ONE StatefulSet but not the others?
Hi, Unfortunately, I am also facing the same: Warning FailedGetResourceMetric 3m59s (x35656 over 6d5h) horizontal-pod-autoscaler missing request for cpu. Any idea how to fix this?
I submitted a tentative PR
Configured HPA as and Deployment CPU resources as Cluster details
Experiencing the same issue: apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: dontdeletedcshpa namespace: docu spec: maxReplicas: 5 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: StatefulSet name: dontdeletedcs-pg targetCPUUtilizationPercentage: 80 kubectl describe hpa dontdeletedcshpa -n docu Name: dontdeletedcshpa Namespace: docu Labels: <noneAnnotations: <noneCreationTimestamp: Tue, 07 Jan 2020 12:14:55 +0530 Reference: StatefulSet/dontdeletedcs-pg Metrics: ( current / target ) resource cpu on pods (as a percentage of request): <unknown/ 80% Min replicas: 1 Max replicas: 5 StatefulSet pods: 3 current / 0 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedGetResourceMetric 8s horizontal-pod-autoscaler missing request for cpu Warning FailedComputeMetricsReplicas 8s horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu
I ran into this as well and this fixed it for me: I am running pods with more than one container. In my case, the other container is a linkerd sidecar. I was setting the resource requests and limits for my deployment but did not set resources for linkerd proxy. You must set resources for all containers within a pod otherwise you will get the error "failed to get cpu utilization". Maybe this error message could be updated? Hope this helps!
This worked for me. I had to add resource requests/limits to a sidecar container and remove "Completed" jobs from the namespace.
I had the same case with sidecars, setting request/limits for all containers fixed the problem. Thanks!
That's great but it's still a bug for people who only have a single container in their pods. I've asked for reviews on
Encountered the same issue and was able to resolve it by making sure that the cronjob pod should be different from the deployment . In case deployment pod matchlabels cronjob pod labels
It worked. Thanks!
Using the new beta API seems to solve it
Experiencing the same issue with one container. any suggestion? didn't help
I have the same issue: Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: did not receive metrics for any ready pods I use aks with windows nodes. aks 1.15.5 only one id three pod is active. warnings: Warning FailedGetResourceMetric horizontalpodautoscaler/kcglob1-hpa did not receive metrics for any ready pods 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kcglob1-hpa Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: did not receive metrics for any ready pods
Running AKS 1.15.7 HPA on stable/nginx-ingress is unable to get cpu or mem metrics. This deployment has only one container and all resources are defined I tried with both and had the same result and Example HPA:
I removed the cluster, and rebuild it from scratch. The problem doesn't appear anymore.
Interesting. I was able to create an HPA for a simple sample deployment but Nginx is still unable to retrieve metrics. Looks like this is related to how HPA uses labels
same issue:the HPA was unable to compute the replica count: missing request for cpu. single container in a pod..
work form me !! thanks
I had a proxy container injected into the pods with no nor defined. Once I defined them and reinitiated the deployments hpa was happily working.
Thank you!! work form me! (I tried in a single container)
I have this same problem for while it works perfectly for my , i.e: As it turns out target is unknown for and exact reason for this according to is: *Environment: Kubernetes version (use kubectl version): 1.18.1 Cloud provider or hardware configuration: AKS
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
for the you can either set the same limits on the defaultBackend or probably use 's idea and set probably and/or specifically if the requests/limits are much higher on the controller pods and it would be a waste of resources
I suspect the reason most of us are here is our beloved nginx-ingress-controller fails to autoscale leading to many tears and much frustration. It seems the chart is giving same labels to the defaultbackend that overlap with the controller deployment causing this issue. Set this in your values yaml to get around it: Feel free to change the labels to whatever you want if you want to be more creative. :D Other option is to set: This is cleaner option, but you cannot set it to this without a maintenance if you are currently deployed because you need to redeploy the controller in this instance because the label is immutable.
see comments above, two methods seems useful, setting request/limits and delete completed status pod but i try both of them, not working i meet this issue for istio in the process of installing kubeflow [root ~]# kubectl get hpa -n istio-system NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE cluster-local-gateway Deployment/cluster-local-gateway unknown/80% 1 5 1 35m istio-ingressgateway Deployment/istio-ingressgateway unknown/80% 1 5 1 35m istio-pilot Deployment/istio-pilot unknown/80% 1 5 1 35m istio-policy Deployment/istio-policy unknown/80% 1 5 1 35m istio-telemetry Deployment/istio-telemetry unknown/80% 1 5 1 35m really need help
Setting explicit resource requests for each container in the pod worked for me.
Hi I want to do same can you help me how i can set resources for the linkerd proxy containers
I did the injection on the Namespaces level: Alternatively you could apply on the Deployment level:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
I found the same issue and in my case the reason because the pod or pods are failing with the metrics is because the POD is not 100% ready... Check the healtchecks , security groups, etc. Here more info:
In my case, I had to add the request resource to the deployment to fix the issue
It works for me thanks!!
I`ve got this issue when all my pods were under one selector an I had to explicity fill resoures blocks for each pod affected this selector
Had same issue with a deployment that could not scale because of the error that the HPA of the deployment was showing. Finally got it fixed now. Here the reasons & background: My deployment consist of a Job that run at the beginning regular POD with three containers - two "sidecar" containers and one with the main app The "main app" container had "resources" set. Both "sidecar" containers had not. So first problem were the missing "resources" specs on both sidecar containers. Such behavior with multiple containers in the POD is described in The second problem was that the "Job" that ran before the actual app deployment has ALSO to have "resources" defined. And THAT was really unexpected. That is something where also stumbled upon & what i've tested then. - thanks for the hint :beer: So, TIL: enable "resources" on ALL containers/jobs in the POD
Also run into this. Had some old pods without requests which were in state (because of preemptible nodes). Deleting these pods got the HPA working.
Thank you so much <3 in minikube case just run "minikube start"
Encountered the same issue when attempting HPA based on CPU utilisation. Fixed the issue by setting CPU resource request at the Deployment level.
it worked for me. The cause is I used Helm to deploy the app along with post-install-job without resource limit. I confirm the Pod's containers must be set relevant resource request
Faced this same issue. In my case my pod container had limits set, but my Dapr injected sidecar container didn't.
I think (substitutes , proposed by approach that checks the GroupVersionKind of pod metrics is suboptimal. E.g. one may have two deployments that use the same selector and then HPA would get metrics for pods of both deployments, checking the kind would not work to tell them apart. See also - basically HPA, although it targets deployment by name, uses deployment selector labels to get pod's metrics.
I talked with and the underlying issue is that you can only query metrics-server by label. So our workaround is the way to go. ref:
My application is running on a pod with a as sidecar. I have solved this adding resource limits to the sidecar definition on the deployment but now I'm confused about how I should read the metrics. Should I read the value as an average of my containers? Also I have trying set this on my HPA definition but I have two issues: Documentation is not clear about how to refer the container in the pod It seems like there is a bug on this How can I set the metric based on my main container ignoring the sidecar?
Wondering if this issue should remain open, it seems what another user pointed out above could be the cause, and was true in my case:
I'm getting same issue, I have enable metric-server in minikube, when I create hpa its always says my deployment able to get scale, but not scale down even after hours, --------Edited------ I have tried my same deployment with kind cluster and its working fine, there is some issue with minikube
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten we keep seeing this with PSS on AKS, EKS. I suspect there is something wrong here. I didn't seen any evidence here tht this is being fixed/worked on? I might be wrong
Reading the discussion it seems to me the error message can have multiple causes, which adds to the confusion. IMO a good action item would be to make the message more detailed, e.g. pointing to which pod and container didn't have the requests set.
/assign

Discussion:
if containerRequest, ok := c.Resources.Requests[resource]; ok { podSum += containerRequest.MilliValue() } else { <del> return nil, fmt.Errorf("missing request for %s", resource) </del> <ins> return nil, fmt.Errorf("missing request for %s in container %s of Pod %s", resource, c.Name, pod.ObjectMeta.Name) </ins> } } }