Kubernetes: running metrics-server in AWS EKS for a Kubernetes Pod AutoScaler
By: Arseny Zinchenko, Setevoy
Tags: Aws Eks, Kubernetes, Monitoring

Assuming, we already have an AWS EKS cluster with worker nodes.

In this post — we will connect to a newly created cluster, will create a test deployment with an HPA — Kubernetes Horizontal Pod AutoScaler and will try to get information about resources usage using kubectl top .

Kubernetes cluster

Create a test cluster using eksctl :

$ eksctl create cluster — profile arseniy — region us-east-2 — name eks-dev-1

…

[ℹ] node “ip-192–168–54–141.us-east-2.compute.internal” is ready

[ℹ] node “ip-192–168–85–24.us-east-2.compute.internal” is ready

[ℹ] kubectl command should work with “/home/setevoy/.kube/config”, try ‘kubectl get nodes’

[✔] EKS cluster “eks-dev-1” in “us-east-2” region is ready

And switch to it.

Kubernetes cluster context

Configure your kubectl :

$ aws eks — profile arseniy — region us-east-2 update-kubeconfig — name eks-dev-1

Added new context arn:aws:eks:us-east-2:534***385:cluster/eks-dev-1 to /home/setevoy/.kube/config

Check EKS clusters available in your AWS account in the us-east-2 region:

$ aws eks — profile arseniy — region us-east-2 list-clusters — output text

CLUSTERS eksctl-bttrm-eks-production-1

CLUSTERS mobilebackend-dev-eks-0-cluster

CLUSTERS eks-dev-1

And check the current profile used:

$ kubectl config current-context

arn:aws:eks:us-east-2:534***385:cluster/eks-dev-1

aws eks already configured the last one cluster for us for the kubectl .

If need — you can always get all profiles available using get-contexts :

$ kubectl config get-contexts

And switch to a necessary one:

$ kubectl config use-context arn:aws:eks:us-east-2:534***385:cluster/eks-dev-1

Switched to context “arn:aws:eks:us-east-2:534***385:cluster/eks-dev-1”.

Deployment

For the testing purpose let’s create a deployment with an AutoScaler.

HPA will use metrics collected from the mertics-server to get data about resource usage on nodes and pods to know when it has to scale up or down a particular deployment's pods.

Create the HPA and deployment:

---

apiVersion: autoscaling/v2beta2

kind: HorizontalPodAutoscaler

metadata:

name: hello-hpa

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

name: hello

minReplicas: 1

maxReplicas: 2

metrics:

- type: Resource

resource:

name: cpu

target:

type: Utilization

averageUtilization: 80

---

apiVersion: apps/v1

kind: Deployment

metadata:

name: hello

spec:

replicas: 1

selector:

matchLabels:

app: hello

template:

metadata:

labels:

app: hello

spec:

containers:

- name: hello

image: gcr.io/google-samples/node-hello:1.0

resources:

limits:

cpu: "0.1"

requests:

cpu: "0.1"

Apply them:

$ kubectl apply -f example-deployment.yml

horizontalpodautoscaler.autoscaling/hello-hpa created

deployment.apps/hello created

Check:

$ kubectl get hpa

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

hello-hpa Deployment/hello <unknown>/80% 1 2 1 26s

HPA — TARGETS <TARGETS> and unable to get metrics

If you’ll check the HPA right now — you’ll see, that it is unable to collect data about its targets (nodes and pods):

$ kubectl get hpa

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

hello-hpa Deployment/hello <unknown>/80% 1 2 1 26s

In the Conditions and Events you can find more details about the issue:

$ kubectl describe hpa hello-hpa

…

Conditions:

Type Status Reason Message

— — — — — — — — — — — - AbleToScale True SucceededGetScale the HPA controller was able to get the target’s current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

…

Events: Type Reason Age From Message

— — — — — — — — — — — — -

Warning FailedGetResourceMetric 12s (x3 over 43s) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io) Warning FailedComputeMetricsReplicas 12s (x3 over 43s) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Actually, here is our cause:

failed to get cpu utilization: unable to get metrics for resource cpu

Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

Also, if try to use top for nodes and pods now - Kubernetes will throw another error:

$ kubectl top node

Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) $ kubectl top pod

Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

To make top able to display resources usage it trying to connect to the Heapster service, see the source code:

...

o.Client = metricsutil.NewHeapsterMetricsClient(clientset.CoreV1(), o.HeapsterOptions.Namespace, o.HeapsterOptions.Scheme, o.HeapsterOptions.Service, o.HeapsterOptions.Port)

...

But Heaspter is deprecated service used earlier to collect the metrics.

Nowadays for the CPU and Memory metrics the metrics-server service is used.

Running metrics-server

Clone the repository:



$ cd metrics-server/ $ git clone https://github.com/kubernetes-sigs/metrics-server.git $ cd metrics-server/

metrics-server configuration for AWS EKS

To make metrics-server able to find all resources in an AWS Elastic Kubernetes Service cluster — edit its deployment file deploy/kubernetes/metrics-server-deployment.yaml , and add the command with four arguments:

...

command:

- /metrics-server

- --logtostderr

- --kubelet-insecure-tls=true

- --kubelet-preferred-address-types=InternalIP

- --v=2

...

kubelet-insecure-tls - do not check kubelet-clients CA certificate on nodes

- do not check kubelet-clients CA certificate on nodes kubelet-preferred-address-types - how to find resources in the Kubernetes space - by using Hostname, InternalDNS, InternalIP, ExternalDNS or ExternalIP, for the EKS set it to the InternalIP value

- how to find resources in the Kubernetes space - by using Hostname, InternalDNS, InternalIP, ExternalDNS or ExternalIP, for the EKS set it to the InternalIP value v=2 - logs detalization level

Save and deploy it:

$ kubectl apply -f deploy/kubernetes/

clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created

clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created

rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created

apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

serviceaccount/metrics-server created

deployment.apps/metrics-server created

service/metrics-server created

clusterrole.rbac.authorization.k8s.io/system:metrics-server created

clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

Check:

$ kubectl -n kube-system get pod

NAME READY STATUS RESTARTS AGE

aws-node-mt9pq 1/1 Running 0 2m5s

aws-node-rl7t2 1/1 Running 0 2m2s

coredns-74dd858ddc-xmrhj 1/1 Running 0 7m33s

coredns-74dd858ddc-xpcwx 1/1 Running 0 7m33s

kube-proxy-b85rv 1/1 Running 0 2m5s

kube-proxy-n647l 1/1 Running 0 2m2s

metrics-server-546565fdc9–56xwl 1/1 Running 0 6s

Or by using:

$ kubectl get apiservices | grep metr

v1beta1.metrics.k8s.io kube-system/metrics-server True 91s

The service’ logs:

kubectl -n kube-system logs -f metrics-server-546565fdc9-qswck

I0215 11:59:05.896014 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)

I0215 11:59:06.439725 1 manager.go:95] Scraping metrics from 0 sources

I0215 11:59:06.439749 1 manager.go:148] ScrapeMetrics: time: 2.728µs, nodes: 0, pods: 0

I0215 11:59:06.450735 1 secure_serving.go:116] Serving securely on [::]:4443

E0215 11:59:10.096632 1 reststorage.go:160] unable to fetch pod metrics for pod default/hello-7d6c85c755-r88xn: no metrics known for pod

E0215 11:59:25.109059 1 reststorage.go:160] unable to fetch pod metrics for pod default/hello-7d6c85c755-r88xn: no metrics known for pod

Try top for nodes:

$ kubectl top node

error: metrics not available yet

And for pods:

$ kubectl top pod

W0215 13:59:58.319317 4014051 top_pod.go:259] Metrics not available for pod default/hello-7d6c85c755-r88xn, age: 4m51.319306547s

error: Metrics not available for pod default/hello-7d6c85c755-r88xn, age: 4m51.319306547s

After 1–2 minutes check metrics-server logs again:

I0215 12:00:06.439839 1 manager.go:95] Scraping metrics from 2 sources

I0215 12:00:06.447003 1 manager.go:120] Querying source: kubelet_summary:ip-192–168–54–141.us-east-2.compute.internal

I0215 12:00:06.450994 1 manager.go:120] Querying source: kubelet_summary:ip-192–168–85–24.us-east-2.compute.internal

I0215 12:00:06.480781 1 manager.go:148] ScrapeMetrics: time: 40.886465ms, nodes: 2, pods: 8

I0215 12:01:06.439817 1 manager.go:95] Scraping metrics from 2 sources

And try top again:

$ kubectl top node

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%

ip-192–168–54–141.us-east-2.compute.internal 25m 1% 406Mi 5%

ip-192–168–85–24.us-east-2.compute.internal 26m 1% 358Mi 4%

Pods:

$ kubectl top pod

NAME CPU(cores) MEMORY(bytes)

hello-7d6c85c755-r88xn 0m 8Mi

And the HPA service:

$ kubectl describe hpa hello-hpa

…

Conditions:

Type Status Reason Message

— — — — — — — — — — — - AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)

ScalingLimited True TooFewReplicas the desired replica count is more than the maximum replica count Events:

Type Reason Age From Message

— — — — — — — — — — — — -

Warning FailedComputeMetricsReplicas 4m23s (x12 over 7m10s) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io) Warning FailedGetResourceMetric 4m8s (x13 over 7m10s) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Pay attention here on the following messages:

the HPA was able to successfully calculate a replica count from cpu resource utilization — HPA was able now to collect metrics

unable to get metrics for resource cpu — check the Age, — its counter must stop growing (keeps [13 number)

Actually, that’s all needed to run a metrics-server to use it with an HPA.

Below — one more common error when using HPA and metrics-server .

HPA was unable to compute the replica count: missing request for cpu

Sometimes HPA can report about the following errors:

$ kubectl describe hpa hello-hpa

…

Conditions:

Type Status Reason Message

— — — — — — — — — — — -

AbleToScale True SucceededGetScale the HPA controller was able to get the target’s current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu

… Warning FailedGetResourceMetric 2s (x2 over 17s) horizontal-pod-autoscaler missing request for cpu

This may happen if a pod’s template in a Deployment has no resources and requests defined.

In this current case I commented out those lines:

...

---

apiVersion: apps/v1

kind: Deployment

metadata:

name: hello

spec:

replicas: 1

selector:

matchLabels:

app: hello

template:

metadata:

labels:

app: hello

spec:

containers:

- name: hello

image: gcr.io/google-samples/node-hello:1.0

# resources:

# limits:

# cpu: "0.1"

# requests:

# cpu: "0.1"

And deployed the pod without requests which resulted to the "missing request for cpu" error message.

Set the requests back - and it must be working now.

Done.