Project: Inspektor Gadget (Observability and Analysis / Observability)
File: prometheus.md

title: 'Using prometheus' weight: 30 description: > Expose metrics using prometheus The Prometheus gadget collects and exposes metrics in Prometheus format. It's available in both, for Kubernetes (`ig-k8s`) and in Linux hosts (`ig`). ```bash $ kubectl gadget prometheus --config @<path> $ ig prometheus --config @<path> --metrics-listen-address $IP:$PORT --metrics-path /metrics ``` The configuration file defines the metrics to be exposed and their settings. The structure of this file is: ```yaml metricsname: metricsname metrics: name: metric_name type: counter or gauge or histogram category: trace # category of the gadget to collect the metric. trace, snapshot, etc. gadget: exec # gadget used to collect the metric. exec, open, etc. selector: labels: ``` It's possible to configure Inspektor Gadget to only update metrics for some specific labels. This is useful to keep the cardinality of the labels low. ```yaml selector: "columnName:value" # matches if the content of the column is equals to value "columnName:!value" # matches if the content of the column is not equal to value "columnName:>=value" # matches if the content of the column is greater and equal to value "columnName:>value" # matches if the content of columnName is greater than the value "columnName:<=value" # matches, if the content of columnName is lower or equal to the value "columnName:<value" # matches, if the content of columnName is lower than the value "columnName:~value" # matches if the content of column matches the regular expression 'value'. ``` Some examples are: Only metrics for default namespace ```yaml selector: k8s.namespace: default ``` Only events with retval != 0 ```yaml selector: "retval:!0" ``` Only events executed by pid 1 by non root users ```yaml selector: "pid:0" "uid:>=1" ``` This is the most intuitive metric: "A counter is a cumulative metric that represents a single whose value can only increase or be reset to zero on restart. For example, you can use a counter to represent the number of requests served, tasks completed, or errors." from . The following are examples of counters we can support with the existing gadgets. The first one counts the number of executed processes by namespace, pod and container. ```yaml metricsname: mymetrics metrics: name: executed_processes type: counter category: trace gadget: exec labels: k8s.namespace k8s.pod k8s.container ``` By default, a counter is increased by one each time there is an event, however it's possible to increase a counter using a field on the event too. Executed processes by pod and container in the default namespace ```yaml metricsname: metricsname metrics: name: executed_processes type: counter category: trace gadget: exec labels: k8s.pod k8s.container selector: "k8s.namespace:default" ``` Or only count events for a given command: `cat` executions by namespace, pod and container ```yaml metricsname: metricsname metrics: name: executed_cats # ohno! type: counter category: trace gadget: exec labels: k8s.namespace k8s.pod

selector: "comm:cat" ``` DNS requests aggregated by namespace and pod ```yaml metricsname: metricsname metrics: name: dns_requests type: counter category: trace gadget: dns labels: k8s.namespace k8s.pod selector: "qr:Q" # Only count query events ``` "A gauge is a metric that represents a single numerical value that can arbitrarily go up and down" from . Right now only snapshotters are supported. Examples of gauges are: Number of processes by namespace, pod and container. ```yaml metricsname: metricsname metrics: name: numberofprocesses type: gauge category: snapshot gadget: process labels: k8s.namespace k8s.pod k8s.container ``` Number of sockets in `CLOSE_WAIT` state ```yaml metricsname: metricsname metrics: name: numberofsocketsclosewait type: gauge category: snapshot gadget: socket labels: k8s.namespace k8s.pod k8s.container selector: "status:CLOSE_WAIT" ``` "A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values." from . We support the same bucket configuration as described in Right now only trace gadgets are supported. Example of histograms is: Latency of DNS requests for all pods ```yaml metricsname: metricsname metrics: name: dnsrequestslatency type: histogram category: trace field: latency bucket: min: 0 max: 10 multiplier: 100000 # 0.1ms type: exp2 unit: ns selector: "qr:R" # Latency is only calculated for response events ``` Let's see how we can use this gadget in different environments. In this guide we'll use the Prometheus Service Discovery: it automatically detects the endpoints to scrape metrics from. If you already have a Prometheus instance running in your cluster, be sure you provide it with the following configuration: ```yaml scrape_configs: job_name: 'kubernetes-pods' scrape_interval: 1s scrape_timeout: 1s kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodannotationprometheusio_scrape] action: keep regex: true sourcelabels: [metakubernetespodannotationprometheusio_scheme] action: replace targetlabel: scheme_ regex: (https?) sourcelabels: [metakubernetespodannotationprometheusio_path] action: replace targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetespodannotationprometheusio_port] action: replace targetlabel: address_ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 ``` Otherwise, you can just apply the config provided with this guide: ```bash $ kubectl apply -f docs/examples/prometheus.yaml namespace/monitoring created serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/discoverer created clusterrolebinding.rbac.authorization.k8s.io/prometheus-discoverer created configmap/prometheus-server-conf created deployment.apps/prometheus created ``` Create a port-forward session to Prometheus: ```bash $ kubectl port-forward --namespace monitoring deployment/prometheus 9090:9090 & ``` Let's create a metric that reports processes executed: ```yaml metrics_name: guide metrics: name: executed_processes type: counter category: trace gadget: exec labels: k8s.namespace k8s.pod k8s.container ``` Start the gadget ```bash $ kubectl gadget prometheus --config @myconfig.yaml INFO[0000] Running. Press Ctrl + C to finish INFO[0000] minikube | Publishing metrics... ``` <!-- markdown-link-check-disable-next-line --> Now, the `executedprocessestotal` counter is available in Prometheus http://localhost:9090/graph?g0.expr=executedprocessestotal&g0.tab=0&g0.stacked=0&g0.showexemplars=0&g0.rangeinput=1m: You can see that the counters are already going up for some

Let's create a pod to execute from more processes: ```bash $ kubectl run mypod1 -it --image busybox --restart Never -- sh -c 'for i in $(seq 0 1 1000); do cat /dev/null ; ping -c 1 localhost > /dev/null; done' ``` If we check the counter again, we can see that it shows that our pod has executed a lot of processes: Now, update the configuration file to only take into considerations executions of the `cat` binary: ```yaml metrics_name: guide metrics: name: executed_processes type: counter category: trace gadget: exec labels: k8s.namespace k8s.pod k8s.container selector: "comm:cat" ``` Restart the gadget ```bash $ kubectl gadget prometheus --config @myconfig.yaml INFO[0000] Running. Press Ctrl + C to finish INFO[0000] minikube | Publishing metrics... ``` Create a new pod that executes processes: ```bash $ kubectl run mypod2 -it --image busybox --restart Never -- sh -c 'for i in $(seq 0 1 1000); do cat /dev/null ; ping -c 1 localhost > /dev/null; done' ``` The counter only takes into consideration the cat commands now: It's also possible to use the prometheus gadget without Kubernetes. In this case, we have to configure Prometheus to point to the endpoint exposed by ig, it's `localhost:2223` by default: ```yaml scrape_configs: job_name: ig scrape_interval: 1s static_configs: targets: localhost:2223 ``` Start prometheus with above configuration (please refer to [docker installation](https://prometheus.io/docs/prometheus/latest/installation/#using-docker) in case you want to run prometheus in a container). ```bash $ prometheus --config.file prometheus.yaml ``` Then, start the prometheus gadget with the same configuration as above Kubernetes section: ```bash $ sudo ig prometheus --config @myconfig.yaml INFO[0000] Running. Press Ctrl + C to finish INFO[0000] Publishing metrics... ``` <!-- markdown-link-check-disable-next-line --> You can check in http://localhost:9090/targets and check that the ig endpoint is reporting metrics: Let's execute some commands inside a container: ```bash docker run --rm -ti --name=mycontainer busybox sh -c 'for i in $(seq 0 1 1000); do cat /dev/null ; ping -c 1 localhost > /dev/null; done' ``` <!-- markdown-link-check-disable-next-line --> We can see how the counter for `mycontainer` is increased in http://localhost:9090/graph?g0.expr=executedprocessestotal&g0.tab=0&g0.stacked=0&g0.showexemplars=0&g0.rangeinput=1m. It's possible to visualize the metrics in Grafana. As an example we will plot a histogram for DNS requests latency. We can use the to prepare the environment: ```bash $ pushd tools/monitoring $ docker compose up -d $ popd ``` <!-- markdown-link-check-disable-next-line --> At this point, Grafana is available at http://localhost:3000 and Prometheus at http://localhost:9090. We can start `ig` with the following configuration: ```bash $ sudo ig prometheus --config @tools/monitoring/config/histogram.yaml INFO[0000] Running. Press Ctrl + C to finish ``` Now, generate some DNS requests: ```bash $ docker run --rm -ti busybox sh -c 'for i in $(seq 0 1 1000); do cat /dev/null ; nslookup -querytype=a microsoft.com. > /dev/null; done' ``` <!-- markdown-link-check-disable --> We should now be able to see the visualized histogram at: http://localhost:3000/d/e1981f70-308c-4784-b986-9b5f1a895444/inspektor-gadget?orgId=1&viewPanel=1 <!-- markdown-link-check-enable --> The `kubectl gadget` instance has to keep running in order to update the metrics. It's not possible to configure the metrics endpoint in ig-k8s