Project: Kata Containers (Runtime / Container Runtime)
File: how-to-set-prometheus-in-k8s.md

This document describes how to run `kata-monitor` in a Kubernetes cluster using Prometheus's service discovery to scrape metrics from `kata-agent`. Warning: This how-to is only for evaluation purpose, you SHOULD NOT running it in production using this configurations. If you are running Kata containers in a Kubernetes cluster, the best way to run `kata-monitor` is using Kubernetes native `DaemonSet`, `kata-monitor` will run on desired Kubernetes nodes without other operations when new nodes joined the cluster. Prometheus also support a Kubernetes service discovery that can find scrape targets dynamically without explicitly setting `kata-monitor`'s metric endpoints. You must have a running Kubernetes cluster first. If not, first. Also you should ensure that `kubectl` working correctly. Note: More information about Kubernetes integrations: - Start Prometheus by utilizing our sample manifest: ``` $ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/prometheus.yml ``` This will create a new namespace, `prometheus`, and create the following resources: `ClusterRole`, `ServiceAccount`, `ClusterRoleBinding` to let Prometheus to access Kubernetes API server. `ConfigMap` that contains minimum configurations to let Prometheus run Kubernetes service discovery. `Deployment` that run Prometheus in `Pod`. `Service` with `type` of `NodePort`(`30909` in this how to), that we can access Prometheus through `<hostIP>:30909`. In production environment, this `type` may be `LoadBalancer` or `Ingress` resource. After the Prometheus server is running, run `curl -s http://hostIP:NodePort:30909/metrics`, if Prometheus is working correctly, you will get response like these: ``` gogcduration_seconds{quantile="0"} 3.9403e-05 gogcduration_seconds{quantile="0.25"} 0.000169907 gogcduration_seconds{quantile="0.5"} 0.000207421 gogcduration_seconds{quantile="0.75"} 0.000229911 ``` `kata-monitor` can be started on the cluster as follows: ``` $ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/kata-monitor-daemonset.yml ``` This will create a new namespace `kata-system` and a `daemonset` in it. Once the `daemonset` is running, Prometheus should discover `kata-monitor` as a target. You can open `http://<hostIP>:30909/service-discovery` and find `kubernetes-pods` under the `Service Discovery` list Run this command to run Grafana in Kubernetes: ``` $ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/grafana.yml ``` This will create deployment and service for Grafana under namespace `prometheus`. After the Grafana deployment is ready, you can open `http://hostIP:NodePort:30000/` to access Grafana server. For Grafana 7.0.5, the default user/password is `admin/admin`. You can modify the default account and adjust other security settings by editing the . To use Grafana show data from Prometheus, you must create a Prometheus `datasource` and dashboard. Open `http://hostIP:NodePort:30000/datasources/new` in your browser, select Prometheus from time series databases list. Normally you only need to set `URL` to `http://hostIP:NodePort:30909` to let it work, and leave the name as `Prometheus` as default. A for Kata Containers metrics is provided which can be imported to Grafana for evaluation. You can import this dashboard using Grafana UI, or using `curl` command in console. ``` $ curl -XPOST -i localhost:3000/api/dashboards/import \ -u admin:admin \ -H "Content-Type: application/json" \ -d "{\"dashboard\":$(curl -sL https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/dashboard.json )}" ```