Writing custom metrics exporter for Kubernetes HPA
By: Shubhomoy Biswas
Tags: Docker, Monitoring, Autoscaling, Kubernetes, DevOps

The above idea is quite simple,

We create a Node web server (this can be any web server of your choice) that exposes the required custom.metrics.k8s.io API paths (in HTTPS). Here we do all the complicated metric calculations and return the metrics in the desired response format. Register the custom.metrics.k8s.io API using the Kubernetes APIService. Here, the version is chosen as v1beta1, but it can be anything as you desire. By performing the above two steps, we have just extended the Kubernetes API to be used internally by our HPA! The HPA can then start calling this new metric’s API in order to scale the main application pods as required.

Now let’s break down the things one by one

APIService

apiVersion: apiregistration.k8s.io/v1beta1

kind: APIService

metadata:

name: v1beta1.custom.metrics.k8s.io

spec:

insecureSkipTLSVerify: true

group: custom.metrics.k8s.io

groupPriorityMinimum: 1000

versionPriority: 5

service:

name: my-metrics-exporter

namespace: custom-metrics

version: v1beta1

APIService will register the custom.metrics.k8s.io API. We provide the version, and to which service it should forward the requests. In this case our Node webserver’s service (my-metrics-exporter).

NodeJS Webserver

We write the backend code needed to provide the registered APIs’ responses. The complete paths are:

/apis/custom.metrics.k8s.io/<VERSION>

We replace <VERSION> with the corresponding version we registered with in the APIService. In this case v1beta1. This path will provide a liveliness check for our web server, and thus return a “healthy” status. /apis/custom.metrics.k8s.io/<VERSION>/namespaces/<NAMESPACE>/services/<SERVICE>/<METRIC>

We replace the <NAMESPACE> and <SERVICE> with the corresponding namespace and the service name of the deployed NodeJS application respectively. <METRIC> will be the metric name we want to expose. Now the main question is, what should be the desired response format of this API in order for the HPA to consume this metric value correctly? And here is the JSON output template.

{

kind: "MetricValueList",

apiVersion: "custom.metrics.k8s.io/v1beta1",

metadata: {

selfLink: "/apis/custom.metrics.k8s.io/v1beta1/"

},

items: [{

describedObject: {

kind: "Service",

namespace: "<NAMESPACE>",

name: "<SERVICE>",

apiVersion: "/v1beta1"

},

metricName: "<METRIC>",

timestamp: new Date(),

value: <METRIC_VALUE>

}]

}

This is how we can expose the metric along with its current value for the HPA to consider.

Do note that Kubernetes API works with HTTPS, so make sure the Node web server exposes the API paths in HTTPS.

HPA

And here is a simple HPA as an example

apiVersion: autoscaling/v2beta2

kind: HorizontalPodAutoscaler

metadata:

name: <HPA_NAME>

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

name: <MAIN_APPLICATION>

minReplicas: 1

maxReplicas: 10

behavior:

scaleDown:

stabilizationWindowSeconds: 60

policies:

- type: Pods

value: 1

periodSeconds: 20

metrics:

- type: Object

object:

metric:

name: instance

describedObject:

apiVersion: /v1beta1

kind: Service

name: <SERVICE>

target:

type: Value

value: <METRIC_VALUE>

The HPA will now use the newly exposed APIs from the NodeJS web server and check whether it reaches the target <METRIC_VALUE> and scale the pods of the <MAIN_APPLICATION> accordingly. I also added a scaleDown behaviour for testing purpose.

Here is a small GIF to demonstrate the end result. I added one another API route in the NodeJS application in order to manually update the custom metric value for testing. At the bottom terminal window, as I update that metric value to 6 using that API, the HPA updates and and scales up the aggregator deployment which is my main test application.

Here is the Github link containing the custom metric exporter if you want to try it out.

Hope this read is useful to you. Some references worth considering…