Project: KubeDB by AppsCode (App Definition and Development / Database)
File: using-builtin-prometheus.md

title: Monitoring Redis Using Builtin Prometheus Discovery menu: docs_{{ .version }}: identifier: rd-using-builtin-prometheus-monitoring name: Builtin Prometheus parent: rd-monitoring-redis weight: 20 menuname: docs{{ .version }} sectionmenuid: guides New to KubeDB? Please start . This tutorial will show you how to monitor Redis server using builtin scraper. At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using . Install KubeDB operator in your cluster following the steps . If you are not familiar with how to configure Prometheus to scrape metrics from various Kubernetes resources, please read the tutorial from . To learn how Prometheus monitoring works with KubeDB in general, please visit . To keep Prometheus resources isolated, we are going to use a separate namespace called `monitoring` to deploy respective monitoring resources. We are going to deploy database in `demo` namespace. ```bash $ kubectl create ns monitoring namespace/monitoring created $ kubectl create ns demo namespace/demo created ``` Note: YAML files used in this tutorial are stored in folder in GitHub repository . At first, let's deploy an Redis server with monitoring enabled. Below is the Redis object that we are going to create. ```yaml apiVersion: kubedb.com/v1alpha2 kind: Redis metadata: name: builtin-prom-redis namespace: demo spec: version: 6.0.20 terminationPolicy: WipeOut storage: storageClassName: "standard" accessModes: ReadWriteOnce resources: requests: storage: 1Gi monitor: agent: prometheus.io/builtin ``` Here, `spec.monitor.agent: prometheus.io/builtin` specifies that we are going to monitor this server using builtin Prometheus scraper. Let's create the Redis crd we have shown above. ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/redis/monitoring/builtin-prom-redis.yaml redis.kubedb.com/builtin-prom-redis created ``` Now, wait for the database to go into `Running` state. ```bash $ kubectl get rd -n demo builtin-prom-redis NAME VERSION STATUS AGE builtin-prom-redis 4.0-v1 Running 41s ``` KubeDB will create a separate stats service with name `{Redis crd name}-stats` for monitoring purpose. ```bash $ kubectl get svc -n demo --selector="app.kubernetes.io/instance=builtin-prom-redis" NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE builtin-prom-redis ClusterIP 10.109.162.108 <none> 6379/TCP 59s builtin-prom-redis-stats ClusterIP 10.106.243.251 <none> 56790/TCP 41s ``` Here, `builtin-prom-redis-stats` service has been created for monitoring purpose. Let's describe the service. ```bash $ kubectl describe svc -n demo builtin-prom-redis-stats Name: builtin-prom-redis-stats Namespace: demo Labels: app.kubernetes.io/name=redises.kubedb.com app.kubernetes.io/instance=builtin-prom-redis Annotations: monitoring.appscode.com/agent: prometheus.io/builtin prometheus.io/path: /metrics prometheus.io/port: 56790 prometheus.io/scrape: true Selector: app.kubernetes.io/name=redises.kubedb.com,app.kubernetes.io/instance=builtin-prom-redis Type: ClusterIP IP: 10.106.243.251 Port: prom-http 56790/TCP TargetPort: prom-http/TCP Endpoints: 172.17.0.14:56790 Session Affinity: None Events: <none> ``` You can see that the service contains following annotations. ```bash prometheus.io/path: /metrics prometheus.io/port: 56790 prometheus.io/scrape: true ``` The Prometheus server will discover the service endpoint using these specifications and will scrape metrics from the exporter. Now, we have to configure a Prometheus scraping job to scrape the metrics using this service. We are going to configure scraping job similar to this job that scrapes metrics from endpoints of a service. Let's configure a Prometheus scraping job to collect metrics from this service. ```yaml job_name: 'kubedb-databases' honor_labels: true scheme: http kubernetessdconfigs: role: endpoints relabel_configs: sourcelabels: [metakubernetesserviceannotationprometheusioscrape, metakubernetesserviceannotationprometheusio_port] separator: ; regex: true;(.*) action: keep sourcelabels: [metakubernetesserviceannotationprometheusio_scheme] action: drop regex: https sourcelabels: [metakubernetesservicename] separator: ; regex: (.*-stats) action: keep sourcelabels: [metakubernetesservicelabelappkubernetesioname] separator: ; regex: (.*) action: keep sourcelabels: [metakubernetesserviceannotationprometheusio_path] action: replace targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetesserviceannotationprometheusio_port] action: replace targetlabel: address_ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 sourcelabels: [metakubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace sourcelabels: [metakubernetesservicename] separator: ; regex: (.*) target_label: service replacement: $1 action: replace action: labelmap regex:

``` If you already have a Prometheus server running, you have to add above scraping job in the `ConfigMap` used to configure the Prometheus server. Then, you have to restart it for the updated configuration to take effect. If you don't use a persistent volume for Prometheus storage, you will lose your previously scraped data on restart. If you don't have any existing Prometheus server running, you have to deploy one. In this section, we are going to deploy a Prometheus server in `monitoring` namespace to collect metrics using this stats service. Create ConfigMap: At first, create a ConfigMap with the scraping configuration. Bellow, the YAML of ConfigMap that we are going to create in this tutorial. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config labels: app: prometheus-demo namespace: monitoring data: prometheus.yml: |- global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: job_name: 'kubedb-databases' honor_labels: true scheme: http kubernetessdconfigs: role: endpoints relabel_configs: sourcelabels: [metakubernetesserviceannotationprometheusioscrape, metakubernetesserviceannotationprometheusio_port] separator: ; regex: true;(.*) action: keep sourcelabels: [metakubernetesserviceannotationprometheusio_scheme] action: drop regex: https sourcelabels: [metakubernetesservicename] separator: ; regex: (.*-stats) action: keep sourcelabels: [metakubernetesservicelabelappkubernetesioname] separator: ; regex: (.*) action: keep sourcelabels: [metakubernetesserviceannotationprometheusio_path] action: replace targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetesserviceannotationprometheusio_port] action: replace targetlabel: address_ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 sourcelabels: [metakubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace sourcelabels: [metakubernetesservicename] separator: ; regex: (.*) target_label: service replacement: $1 action: replace action: labelmap regex: metakubernetesservicelabel(.+) ``` Let's create above `ConfigMap`, ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/monitoring/builtin-prometheus/prom-config.yaml configmap/prometheus-config created ``` Create RBAC: If you are using an RBAC enabled cluster, you have to give necessary RBAC permissions for Prometheus. Let's create necessary RBAC stuffs for Prometheus, ```bash $ kubectl apply -f https://github.com/appscode/third-party-tools/raw/master/monitoring/prometheus/builtin/artifacts/rbac.yaml clusterrole.rbac.authorization.k8s.io/prometheus created serviceaccount/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created ``` YAML for the RBAC resources created above can be found . Deploy Prometheus: Now, we are ready to deploy Prometheus server. We are going to use following to deploy Prometheus server. Let's deploy the Prometheus server. ```bash $ kubectl apply -f https://github.com/appscode/third-party-tools/raw/master/monitoring/prometheus/builtin/artifacts/deployment.yaml deployment.apps/prometheus created ``` Prometheus server is listening to port `9090`. We are going to use to access Prometheus dashboard. At first, let's check if the Prometheus pod is in `Running` state. ```bash $ kubectl get pod -n monitoring -l=app=prometheus NAME READY STATUS RESTARTS AGE prometheus-8568c86d86-95zhn 1/1 Running 0 77s ``` Now, run following command on a separate terminal to forward 9090 port of `prometheus-8568c86d86-95zhn` pod, ```bash $ kubectl port-forward -n monitoring prometheus-8568c86d86-95zhn 9090 Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 ``` Now, we can access the dashboard at `localhost:9090`. Open in your browser. You should see the endpoint of `builtin-prom-redis-stats` service as one of the targets. <p align="center"> <img alt="Prometheus Target" height="100%" src="/docs/images/redis/monitoring/redis-builtin-prom-target.png" style="padding:10px"> </p> Check the labels marked with red rectangle. These labels confirm that the metrics are coming from `Redis` server `builtin-prom-redis` through stats service `builtin-prom-redis-stats`. Now, you can view the collected metrics and create a graph from homepage of this Prometheus dashboard. You can also use this Prometheus server as data source for and create beautiful dashboard with collected metrics. To cleanup the Kubernetes resources created by this tutorial, run following commands ```bash $ kubectl delete -n demo rd/builtin-prom-redis $ kubectl delete -n monitoring deployment.apps/prometheus $ kubectl delete -n monitoring clusterrole.rbac.authorization.k8s.io/prometheus $ kubectl delete -n monitoring serviceaccount/prometheus $ kubectl delete -n monitoring clusterrolebinding.rbac.authorization.k8s.io/prometheus $ kubectl delete ns demo $ kubectl delete ns monitoring ``` Monitor your Redis server with KubeDB using . Detail concepts of . Use to deploy Redis with KubeDB. Want to hack on KubeDB? Check our .