Project: Grafana Mimir (Observability and Analysis / Observability)
File: collecting-metrics-and-logs.md

aliases: ../../operators-guide/monitoring-grafana-mimir/collecting-metrics-and-logs/ ../../operators-guide/monitor-grafana-mimir/collecting-metrics-and-logs/ description: Learn how to collect metrics and logs from Grafana Mimir itself menuTitle: Collecting metrics and logs title: Collecting metrics and logs from Grafana Mimir weight: 60 You can collect logs and metrics from a Mimir or GEM cluster. To set up dashboards and alerts, see or . It is easier and best to monitor a cluster if it was installed via the Grafana Mimir Helm chart. For more information, see the . It is also possible to use this integration if Mimir was deployed another way. For more information, see . You can still use the dashboards and rules in the monitoring-mixin, even if Mimir or GEM is not deployed via the Helm chart. If you are not using the Helm chart, start by using the Agent configuration from . You might need to modify it. For more information, see . The Agent configuration relies on Kubernetes service discovery and Pod labels to constrain the collected metrics and logs to ones that are strictly related to the Grafana Mimir deployment. If you are deploying Grafana Mimir on something other than Kubernetes, then replace the `kubernetessdconfigs` block with a block from the that can discover the Mimir processes. Set up a Grafana Agent that collects logs and metrics from Mimir or GEM. To set up Grafana Agent, see . After your Agent is deployed, use the to configure the Agent to scrape Mimir or GEM. Managing your own Agent comes with some caveats: You will have to keep the Agent configuration up to date manually as you update the Mimir Helm chart. While we will try to keep this article up to date, we cannot guarantee that the will always work. The static configuration makes some assumptions about the naming of the chart, such as that you have not overridden the `fullnameOverride` in the Helm chart. The static configuration cannot be selective in the PersistentVolumes metrics it collects from Kubelet, so it will scrape metrics for all PersistentVolumes. The static configuration hardcodes the value of the `cluster` label on all metrics and logs. This means that the configuration cannot account for multiple installations of the Helm chart. If possible, upgrade the Mimir Helm chart to version

or higher and use the . Using the Agent operator allows the chart to automatically configure the Agent, eliminating the aforementioned caveats. In the following example Grafana Agent configuration file for collecting logs and metrics, replace `url`, `password`, and `username` in the `logs` and `metrics` blocks with the details of your Prometheus and Loki clusters. ```yaml logs: configs: clients: basic_auth: password: xxx username: xxx url: https://example.com/loki/api/v1/push name: integrations positions: filename: /tmp/positions.yaml scrape_configs: job_name: integrations/grafana-mimir-logs kubernetessdconfigs: role: pod pipeline_stages: cri: {} relabel_configs: action: keep regex: mimir-distributed-.* source_labels: metakubernetespodlabelhelmshchart source_labels: metakubernetespodnodename targetlabel: host_ action: replace replacement: $1 separator: / source_labels: metakubernetesnamespace metakubernetespodcontainername target_label: job action: replace regex: "" replacement: k8s-cluster separator: "" source_labels: cluster target_label: cluster action: replace source_labels: metakubernetesnamespace target_label: namespace action: replace source_labels: metakubernetespod_name target_label: pod action: replace source_labels: metakubernetespodcontainername target_label: name action: replace source_labels: metakubernetespodcontainername target_label: container replacement: /var/log/pods/$1/.log separator: / source_labels: metakubernetespod_uid metakubernetespodcontainername targetlabel: path_ target_config: sync_period: 10s metrics: configs: name: integrations remote_write: basic_auth: password: xxx username: xxx url: https://example.com/api/prom/push scrape_configs: job_name: integrations/grafana-mimir/kube-state-metrics kubernetessdconfigs: role: pod metricrelabelconfigs: action: keep regex: (.-mimir-)?alertmanager.|(.-mimir-)?compactor.|(.-mimir-)?distributor.|(.-mimir-)?(gateway|cortex-gw|cortex-gw).|(.-mimir-)?ingester.|(.-mimir-)?querier.|(.-mimir-)?query-frontend.|(.-mimir-)?query-scheduler.|(.-mimir-)?ruler.|(.-mimir-)?store-gateway. separator: "" source_labels: deployment statefulset pod relabel_configs: action: keep regex: kube-state-metrics source_labels: metakubernetespodlabelappkubernetesio_name action: replace regex: "" replacement: k8s-cluster separator: "" source_labels: cluster target_label: cluster bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token job_name: integrations/grafana-mimir/kubelet kubernetessdconfigs: role: node metricrelabelconfigs: action: keep regex: kubeletvolumestats.* source_labels: name relabel_configs: replacement: kubernetes.default.svc.cluster.local:443 targetlabel: address_ regex: (.+) replacement: /api/v1/nodes/${1}/proxy/metrics source_labels: metakubernetesnode_name targetlabel: metricspath action: replace regex: "" replacement: k8s-cluster separator: "" source_labels: cluster target_label: cluster scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureskipverify: false server_name: kubernetes bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token job_name: integrations/grafana-mimir/cadvisor kubernetessdconfigs: role: node metricrelabelconfigs: action: keep regex: (.-mimir-)?alertmanager.|(.-mimir-)?compactor.|(.-mimir-)?distributor.|(.-mimir-)?(gateway|cortex-gw|cortex-gw).|(.-mimir-)?ingester.|(.-mimir-)?querier.|(.-mimir-)?query-frontend.|(.-mimir-)?query-scheduler.|(.-mimir-)?ruler.|(.-mimir-)?store-gateway. source_labels: pod relabel_configs: replacement: kubernetes.default.svc.cluster.local:443 targetlabel: address_ regex: (.+) replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor source_labels: metakubernetesnode_name targetlabel: metricspath action: replace regex: "" replacement: k8s-cluster separator: "" source_labels: cluster target_label: cluster scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureskipverify: false server_name: kubernetes job_name: integrations/grafana-mimir/metrics kubernetessdconfigs: role: pod relabel_configs: action: keep regex: .*metrics source_labels: metakubernetespodcontainerport_name action: keep regex: mimir-distributed-.* source_labels: metakubernetespodlabelhelmshchart action: replace regex: "" replacement: k8s-cluster separator: "" source_labels: cluster target_label: cluster action: replace source_labels: metakubernetesnamespace target_label: namespace action: replace source_labels: metakubernetespod_name target_label: pod action: replace source_labels: metakubernetespodcontainername target_label: container action: replace separator: "" source_labels: metakubernetespodlabelname metakubernetespodlabelappkubernetesio_component targetlabel: tmpcomponent_name action: replace separator: / source_labels: metakubernetesnamespace tmpcomponentname target_label: job action: replace source_labels: metakubernetespodnodename target_label: instance global: scrape_interval: 15s wal_directory: /tmp/grafana-agent-wal ```