Guide to Kubernetes Metrics. Recently I was asked to design a…
By: Cong Liu
Tags: Metrics, Kubernetes

Background

Metric Types

According to Kubernetes monitoring architecture, metrics in a Kubernetes cluster can be divided into system (resource) metrics and service metrics. System metrics are generally available from every component running in the cluster such as container CPU usage. Service metrics are application specific metrics defined in application code such as request count to API server.

Kubernetes Metrics API

Kubernetes defines two sets of metrics API: resource metrics API and custom metrics API. Kubernetes provides two implementations for resource metrics API: Heapster and metrics-server while the custom metrics API is up to the vendor to implement. Details about each API will be described below.

Metrics Source

Resource Metrics API

What metrics does this API expose? This API exposes core system metrics, i.e., resource (CPU & Memory) usage metrics for node and containers. Note that this is sometimes referred to as master metrics API. The API definition can be found here.

Which components are involved? The metrics-server and heapster. Note: Heapster is being deprecated. Please consider using metrics-server if possible. Currently in GKE, heapster is responsible for sending the core system metrics to Stackdriver.

How to access this API? This API is made available under the /apis/metrics.k8s.io/ endpoint. For example,

Use $ kubectl proxy --port=8080 and $ curl localhost:8080/apis/metrics.k8s.io/v1beta1/nodes for node metrics.

and for node metrics. Or use $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq for node metrics.

Custom Metrics API

What metrics does this API expose? Since this is a custom metrics API, it’s up to the metrics pipeline vendors to implement whatever metrics they want.

Which component exposes these metrics? Kubernetes defines this API but does not provide any official implementation. There are a few implementations such as the Prometheus Adapter and Google Stackdriver Adapter.

Prometheus Adapter. It queries Prometheus server to get metrics and then exposes them as custom Metrics API via Kubernetes API Aggregation. What metrics are exposed can be configured.

How to access this API? This API is made available under the /apis/custom.metrics.k8s.io/ endpoint. For example,

Use $ kubectl proxy --port=8080 and $ curl localhost:8080/apis/metrics.k8s.io/v1beta1/ to check what resources are available.

and to check what resources are available. Or use $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/" | jq to check what resources are available.

Other Metrics

The resource metrics and custom metrics API are defined by Kubernetes and can be used natively by Kubernetes components such as the Horizontal Auto Scaler (HPA). In addition to these native Kubernetes metrics sources, there are many other sources of metrics such as the Prometheus metrics endpoints. Although these metrics are not considered to be “native” or “first-class” metrics in Kubernetes, understanding them helps you to build your own metrics pipeline for your own custom metrics.

Prometheus metrics endpoint

This is probably the most popular way of exposing metrics in Kubernetes. An application can use Prometheus client libraries to define and expose service metrics via an HTTP endpoint, typically under /metrics .

Many Kubernetes components exposes their metrics via the /metrics endpoint, including API server, etcd and many other add-ons such as Goolge Cloud plugin for Fluentd.

You can configure your Prometheus server to scrape metrics from the metrics endpoints and do further analysis. If you use GKE, you can use a prometheus-to-sd sidecar to push these metrics to Google Stackdriver Monitoring.

Misc

kube-state-metrics is a component that exposes metrics about object state, such as pod_start_time . These metrics are in Prometheus format and are exposed at /metrics endpoint.

It’s one of many officially supported exporters by Prometheus. This is an exporter that exposes metrics about Kubernetes nodes.

References