Project: KubeDB by AppsCode (App Definition and Development / Database)
File: builtin-prometheus.md

title: Monitoring KubeDB Operator with builtin Prometheus menu: docs_{{ .version }}: identifier: operator-monitoring-builtin name: Builtin Prometheus parent: operator-monitoring weight: 20 menuname: docs{{ .version }} sectionmenuid: setup aliases: /docs/{{ .version }}/setup/operator-monitoring/ This tutorial will show you how to configure builtin scraper to monitor KubeDB operator. At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using . If you are not familiar with how to configure Prometheus to scrape metrics from various Kubernetes resources, please read the tutorial from . To keep Prometheus resources isolated, we are going to use a separate namespace called `monitoring` to deploy respective monitoring resources. ```bash $ kubectl create ns monitoring namespace/monitoring created ``` Enable Prometheus monitoring using `prometheus.io/builtin` agent while installing KubeDB. To know details about how to enable monitoring see . Let's install KubeDB with operator monitoring enabled. Helm 3: ```bash $ helm install kubedb oci://ghcr.io/appscode-charts/kubedb \ --version {{< param "info.version" >}} \ --namespace kubedb --create-namespace \ --set kubedb-provisioner.monitoring.enabled=true \ --set kubedb-provisioner.monitoring.agent=prometheus.io/builtin \ --set kubedb-provisioner.monitoring.prometheus.namespace=monitoring \ --set kubedb-provisioner.monitoring.serviceMonitor.labels.release=prometheus ``` YAML (with Helm 3): ```bash $ helm template kubedb oci://ghcr.io/appscode-charts/kubedb \ --version {{< param "info.version" >}} \ --namespace kubedb --create-namespace \ --set kubedb-provisioner.monitoring.enabled=true \ --set kubedb-provisioner.monitoring.agent=prometheus.io/builtin \ --set kubedb-provisioner.monitoring.prometheus.namespace=monitoring \ --set kubedb-provisioner.monitoring.serviceMonitor.labels.release=prometheus | kubectl apply -f - ``` This will add necessary annotations to `kubedb` service created in `kubedb` namespace. Prometheus server will scrape metrics using those annotations. Let's check which annotations are added to the service, ```yaml $ kubectl get service -n kubedb kubedb -o yaml apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"kubedb"},"name":"kubedb","namespace":"kubedb"},"spec":{"ports":[{"name":"api","port":443,"targetPort":8443}],"selector":{"app":"kubedb"}}} prometheus.io/path: /metrics prometheus.io/port: "8443" prometheus.io/scheme: https prometheus.io/scrape: "true" creationTimestamp: 2018-12-31T08:44:05Z labels: app: kubedb name: kubedb namespace: kubedb resourceVersion: "22287" selfLink: /api/v1/namespaces/kubedb/services/kubedb uid: 3af092c3-0cd8-11e9-9662-080027e8eafe spec: clusterIP: 10.108.131.64 ports: name: api port: 443 protocol: TCP targetPort: 8443 selector: app: kubedb sessionAffinity: None type: ClusterIP status: loadBalancer: {} ``` Here, `prometheus.io/scrape: "true"` annotation indicates that Prometheus should scrape metrics for this service. The following three annotations point to `api` endpoints which provides operator metrics. ```ini prometheus.io/path: /metrics prometheus.io/port: "8443" prometheus.io/scheme: https ``` Now, we are ready to configure our Prometheus server to scrape those metrics. Now, we have to configure a Prometheus scraping job to scrape the metrics using this service. We are going to configure scraping job similar to this job. However, as we are going to collect metrics from a TLS secured endpoint that exports Kubernetes extension apiserver metrics, we have to add following configurations: section to establish TLS secured connection. `bearertokenfile` to authorize Prometheus server to KubeDB extension

KubeDB has created a secret named `kubedb-apiserver-cert` in `monitoring` namespace as we have specified it through `--prometheus-namespace`. This secret holds the public certificate of KubeDB extension apiserver that is necessary to configure `tls_config` section. Verify that the secret `kubedb-apiserver-cert` has been created in `monitoring` namespace. ```bash $ kubectl get secret -n monitoring -l=app=kubedb NAME TYPE DATA AGE kubedb-apiserver-cert kubernetes.io/tls 2 3h33m ``` We are going to mount this secret in `/etc/prometheus/secret/kubedb-apiserver-cert` directory of Prometheus deployment. Let's configure a Prometheus scraping job to collect the operator metrics. ```yaml job_name: kubedb kubernetessdconfigs: role: endpoints tls_config: ca_file: /etc/prometheus/secret/kubedb-apiserver-cert/tls.crt server_name: kubedb.kubedb.svc bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: sourcelabels: [metakubernetesserviceannotationprometheusio_scrape] regex: true action: keep sourcelabels: [metakubernetesservicelabel_app] regex: kubedb action: keep sourcelabels: [metakubernetesendpointport_name] regex: api action: keep sourcelabels: [metakubernetesserviceannotationprometheusio_path] regex: (.+) targetlabel: metricspath action: replace sourcelabels: [metakubernetesserviceannotationprometheusio_scheme] action: replace targetlabel: scheme_ regex: (https?) sourcelabels: [address, metakubernetesserviceannotationprometheusio_port] action: replace targetlabel: address_ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 sourcelabels: [metakubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace sourcelabels: [metakubernetesservicename] separator: ; regex: (.*) target_label: service replacement: $1 action: replace ``` Note that, `bearertokenfile` denotes the `ServiceAccount` token of the Prometheus server. Kubernetes automatically mount it in `/var/run/secrets/kubernetes.io/serviceaccount/token` directory of Prometheus pod. For, an RBAC enabled cluster, we have to grand some permissions to this `ServiceAccount`. If you already have a Prometheus server running, update the respective `ConfigMap` and add above scraping job. Then, you have to mount `kubedb-apiserver-cert` secret in Prometheus deployment. Add the secret as volume: ```yaml volumes: name: kubedb-apiserver-cert secret: defaultMode: 420 name: kubedb-apiserver-cert items: # avoid mounting private key key: tls.crt path: tls.crt ``` Then, mount this volume in `/etc/prometheus/secret/kubedb-apiserver-cert` directory. ```yaml volumeMounts: name: kubedb-apiserver-cert # mount the secret volume with public certificate of the kubedb extension apiserver mountPath: /etc/prometheus/secret/kubedb-apiserver-cert ``` Warning: Updating deployment will cause restart of your Prometheus server. If you don't use a persistent volume for Prometheus storage, you will lose your previously scraped data. If you don't have any existing Prometheus server running, you have to deploy one. In this section, we are going to deploy a Prometheus server to collect metrics from KubeDB operator. Create ConfigMap: At first, create a ConfigMap with the scraping configuration. Bellow, the YAML of ConfigMap that we are going to create in this tutorial. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kubedb-prom-config labels: app: kubedb namespace: monitoring data: prometheus.yml: |- global: scrape_interval: 30s scrape_timeout: 10s evaluation_interval: 30s scrape_configs: job_name: kubedb kubernetessdconfigs: role: endpoints tls_config: ca_file: /etc/prometheus/secret/kubedb-apiserver-cert/tls.crt server_name: kubedb.kubedb.svc bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: sourcelabels: [metakubernetesserviceannotationprometheusio_scrape] regex: true action: keep sourcelabels: [metakubernetesservicelabel_app] regex: kubedb action: keep sourcelabels: [metakubernetesendpointport_name] regex: api action: keep sourcelabels: [metakubernetesserviceannotationprometheusio_path] regex:

targetlabel: metricspath action: replace sourcelabels: [metakubernetesserviceannotationprometheusio_scheme] action: replace targetlabel: scheme_ regex: (https?) sourcelabels: [address, metakubernetesserviceannotationprometheusio_port] action: replace targetlabel: address_ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 sourcelabels: [metakubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace sourcelabels: [metakubernetesservicename] separator: ; regex: (.*) target_label: service replacement: $1 action: replace ``` Let's create the ConfigMap we have shown above, ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/monitoring/operator/prom-config.yaml configmap/kubedb-prom-config created ``` Create RBAC: If you are using an RBAC enabled cluster, you have to give necessary RBAC permissions for Prometheus. Let's create necessary RBAC stuffs for Prometheus, ```bash $ kubectl apply -f https://github.com/appscode/third-party-tools/raw/master/monitoring/prometheus/builtin/artifacts/rbac.yaml clusterrole.rbac.authorization.k8s.io/prometheus created serviceaccount/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created ``` YAML for the RBAC resources created above can be found . Deploy Prometheus: Now, we are ready to deploy Prometheus server. YAML for the deployment that we are going to create for Prometheus is shown below. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitoring spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: serviceAccountName: prometheus containers: name: prometheus image: prom/prometheus:v2.4.3 args: "--config.file=/etc/prometheus/prometheus.yml" "--storage.tsdb.path=/prometheus/" ports: containerPort: 9090 volumeMounts: name: prometheus-config-volume mountPath: /etc/prometheus/ name: prometheus-storage-volume mountPath: /prometheus/ name: kubedb-apiserver-cert # mount the secret volume with public certificate of the kubedb extension apiserver mountPath: /etc/prometheus/secret/kubedb-apiserver-cert volumes: name: prometheus-config-volume configMap: defaultMode: 420 name: kubedb-prom-conf name: prometheus-storage-volume emptyDir: {} name: kubedb-apiserver-cert secret: defaultMode: 420 secretName: kubedb-apiserver-cert items: # avoid mounting private key key: tls.crt path: tls.crt ``` Notice that, we have mounted `kubedb-apiserver-cert` secret as a volume at `/etc/prometheus/secret/kubedb-apiserver-cert` directory. Use a persistent volume instead of `emptyDir` for `prometheus-storage` volume if you don't want to lose collected metrics on Prometheus pod restart. Now, let's create the deployment, ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/monitoring/operator/prom-deploy.yaml deployment.apps/prometheus created ``` Prometheus server is listening to port `9090`. We are going to use to access Prometheus dashboard. At first, let's check if the Prometheus pod is in `Running` state. ```bash $ kubectl get pod -n monitoring -l=app=prometheus NAME READY STATUS RESTARTS AGE prometheus-5bcb9678c-kh8vt 1/1 Running 0 149m ``` Now, run following command on a separate terminal to forward 9090 port of `prometheus-5bcb9678c-kh8vt` pod, ```bash $ kubectl port-forward -n monitoring prometheus-5bcb9678c-kh8vt 9090 Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 ``` Now, we can access the dashboard at `localhost:9090`. Open in your browser. You should see `api` endpoint of `kubedb` service as target. <p align="center"> <img alt="Prometheus Target" src="/docs/images/monitoring/operator/builtin-prom-target.png" style="padding:10px"> </p> To cleanup the Kubernetes resources created by this tutorial, run: ```bash kubectl delete clusterrole -l=app=prometheus-demo kubectl delete clusterrolebinding -l=app=prometheus-demo kubectl delete -n monitoring deployment prometheus kubectl delete -n monitoring serviceaccount/prometheus kubectl delete -n monitoring configmap/kubedb-prom-config kubectl delete -n monitoring secret kubedb-apiserver-cert kubectl delete ns monitoring ``` To uninstall KubeDB follow this . Learn what metrics KubeDB operator exports from . Learn how to monitor KubeDB operator using Prometheus operator from .