Project: VictoriaMetrics (Observability and Analysis / Observability)
File: k8s-monitoring-via-vm-single.md

weight: 2 title: Kubernetes monitoring via VictoriaMetrics Single menu: docs: parent: "guides" weight: 2 aliases: /guides/k8s-monitoring-via-vm-single.html This guide covers: The setup of a in via Helm charts How to scrape metrics from k8s components using service discovery How to visualize stored data How to store metrics in tsdb Precondition We will use: We use GKE cluster from but this guide is also applied on any Kubernetes cluster. For example . <img src="k8s-monitoring-via-vm-single_k8s-scheme.webp" alt="VictoriaMetrics Single on Kubernetes cluster"> For this guide we will use Helm 3 but if you already use Helm 2 please see this You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. Were going to use . You can do this by running the following command: ```shell helm repo add vm https://victoriametrics.github.io/helm-charts/ ``` Update Helm repositories: ```shell helm repo update ``` To verify that everything is set up correctly you may run this command: ```shell helm search repo vm/ ``` The expected output is: ```text NAME CHART VERSION APP VERSION DESCRIPTION vm/victoria-metrics-agent 0.7.20 v1.62.0 Victoria Metrics Agent - collects metrics from ... vm/victoria-metrics-alert 0.3.34 v1.62.0 Victoria Metrics Alert - executes a list of giv... vm/victoria-metrics-auth 0.2.23 1.62.0 Victoria Metrics Auth - is a simple auth proxy ... vm/victoria-metrics-cluster 0.8.32 1.62.0 Victoria Metrics Cluster version - high-perform... vm/victoria-metrics-k8s-stack 0.2.9 1.16.0 Kubernetes monitoring on VictoriaMetrics stack.... vm/victoria-metrics-operator 0.1.17 0.16.0 Victoria Metrics Operator vm/victoria-metrics-single 0.7.5 1.62.0 Victoria Metrics Single version - high-performa... ``` Run this command in your terminal: ```text helm install vmsingle vm/victoria-metrics-single -f https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml ``` Here is full file content `guide-vmsingle-values.yaml` ```yaml server: scrape: enabled: true configMap: "" config: global: scrape_interval: 15s scrape_configs: job_name: victoriametrics static_configs: targets: [ "localhost:8428" ] job_name: "kubernetes-apiservers" kubernetessdconfigs: role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureskipverify: true bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: source_labels: [ metakubernetesnamespace, metakubernetesservice_name, metakubernetesendpointportname, ] action: keep regex: default;kubernetes;https job_name: "kubernetes-nodes" scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureskipverify: true bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetessdconfigs: role: node relabel_configs: action: labelmap regex: metakubernetesnodelabel(.+) targetlabel: address_ replacement: kubernetes.default.svc:443 sourcelabels: [ metakubernetesnodename ] regex: (.+) targetlabel: metricspath replacement: /api/v1/nodes/$1/proxy/metrics job_name: "kubernetes-nodes-cadvisor" scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureskipverify: true bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetessdconfigs: role: node relabel_configs: action: labelmap regex: metakubernetesnodelabel(.+) targetlabel: address_ replacement: kubernetes.default.svc:443 sourcelabels: [ metakubernetesnodename ] regex: (.+) targetlabel: metricspath replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor metricrelabelconfigs: action: replace source_labels: [pod] regex: '(.+)' targetlabel: podname replacement: '${1}' action: replace source_labels: [container] regex: '(.+)' targetlabel: containername replacement: '${1}' action: replace target_label: name replacement: k8s_stub action: replace source_labels: [id] regex: '^/system\.slice/(.+)\.service$' targetlabel: systemdservice_name replacement: '${1}' ``` By running `helm install vmsingle vm/victoria-metrics-single` we install to default inside your cluster By adding `scrape: enable: true` we add and enable autodiscovery scraping from kubernetes cluster to On line 166 from we added `metricrelabelconfigs` section that will help us to show Kubernetes metrics on Grafana

As a result of the command you will see the following output: ```text NAME: victoria-metrics LAST DEPLOYED: Fri Jun 25 12:06:13 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The VictoriaMetrics write api can be accessed via port 8428 on the following DNS name from within your cluster: vmsingle-victoria-metrics-single-server.default.svc.cluster.local Metrics Ingestion: Get the Victoria Metrics service URL by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace default -l "app=server" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $POD_NAME 8428 Write url inside the kubernetes cluster: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write Metrics Scrape: Pull-based scrapes are enabled Scrape config can be displayed by running this command:: kubectl get cm vmsingle-victoria-metrics-single-server-scrapeconfig -n default The targets information is accessible via api: Inside cluster: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/targets Outside cluster: You need to port-forward service (see instructions above) and call http://<service-host-port>/targets Read Data: The following url can be used as the datasource url in Grafana:: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428 ``` For us its important to remember the url for the datasource (copy lines from output). Verify that VictoriaMetrics pod is up and running by executing the following command: ```shell kubectl get pods ``` The expected output is: ```text NAME READY STATUS RESTARTS AGE vmsingle-victoria-metrics-single-server-0 1/1 Running 0 68s ``` Add the Grafana Helm repository. ```shell helm repo add grafana https://grafana.github.io/helm-charts helm repo update ``` By installing the Chart with the release name `my-grafana`, you add the VictoriaMetrics datasource with official dashboard and kubernetes dashboard: ```yaml cat <<EOF | helm install my-grafana grafana/grafana -f - datasources: datasources.yaml: apiVersion: 1 datasources: name: victoriametrics type: prometheus orgId: 1 url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428 access: proxy isDefault: true updateIntervalSeconds: 10 editable: true dashboardProviders: dashboardproviders.yaml: apiVersion: 1 providers: name: 'default' orgId: 1 folder: '' type: file disableDeletion: true editable: true options: path: /var/lib/grafana/dashboards/default dashboards: default: victoriametrics: gnetId: 10229 revision: 22 datasource: victoriametrics kubernetes: gnetId: 14205 revision: 1 datasource: victoriametrics EOF ``` By running this command we: Install Grafana from Helm repository. Provision VictoriaMetrics datasource with the url from the output above which we copied before. Add for VictoriaMetrics. Add to see Kubernetes cluster metrics. Check the output log in your terminal. To see the password for Grafana `admin` user use the following command: ```shell kubectl get secret --namespace default my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` Expose Grafana service on `127.0.0.1:3000`: ```shell export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=my-grafana" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $POD_NAME 3000 ``` Now Grafana should be accessible on the address. To check that VictoriaMetrics has collects metrics from the k8s cluster open in browser and choose `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously obtained from kubectl. <img src="k8s-monitoring-via-vm-single_grafana-dashboards.webp" alt=""> You will see something like this: <img src="k8s-monitoring-via-vm-single_grafana-k8s-dashboard.webp" alt=""> VictoriaMetrics dashboard also available to use: <img src="k8s-monitoring-via-vm-single_grafana.webp" alt=""> We have set up TimeSeries Database for your k8s cluster. Collected metrics from all running pods,nodes, and store them in VictoriaMetrics database. Visualize resources used in Kubernetes cluster by Grafana dashboards.