Project: Alluxio (Runtime / Cloud Native Storage)
File: Metrics-On-Kubernetes.md

layout: global title: Metrics System On Kubernetes For general information about the metrics of Alluxio, refer to . To deploy Alluxio on Kubernetes, refer to . This documentation focus on how to configure and get metrics of different metrics sinks from Alluxio deployed on Kubernetes. The metrics are exposed through the web ports of different components. The web ports of Alluxio masters and workers are opened by default. Alluxio standalone Fuse web port is not opened by default. It can be opened by setting `alluxio.fuse.web.enabled` to true. You can send an HTTP request to an Alluxio process to get a snapshot of the metrics in JSON format. ```shell $ kubectl exec <COMPONENTHOSTNAME> -c <CONTAINERNAME> -- curl 127.0.0.1:<COMPONENTWEBPORT>/metrics/json/ ``` ```shell $ kubectl exec <alluxio-master-x> -c alluxio-master -- curl 127.0.0.1:19999/metrics/json/ ``` ```shell $ kubectl exec <alluxio-worker-xxxxx> -c alluxio-worker -- curl 127.0.0.1:30000/metrics/json/ ``` ```shell $ kubectl exec <alluxio-master-x> -c alluxio-job-master -- curl 127.0.0.1:20002/metrics/json/ ``` ```shell $ kubectl exec <alluxio-worker-xxxxx> -c alluxio-job-worker -- curl 127.0.0.1:30003/metrics/json/ ``` ```shell $ kubectl exec <alluxio-fuse-xxxxx> -- curl 127.0.0.1:49999/metrics/json/ ``` is a monitoring tool that can help to monitor Alluxio metrics changes. `PrometheusMetricsServlet` needs to be enabled for Prometheus in Alluxio. Set the following properties in helm chart `value.yaml` to enable Prometheus metrics sink: ```yaml metrics: enabled: true PrometheusMetricsServlet: true podAnnotations: prometheus.io/scrape: "true" prometheus.io/masterWebPort: "<MASTERWEBPORT>" prometheus.io/jobMasterWebPort: "<JOBMASTERWEB_PORT>" prometheus.io/workerWebPort: "<WORKERWEBPORT>" prometheus.io/jobWorkerWebPort: "<JOBWORKERWEB_PORT>" prometheus.io/fuseWebPort: "<FUSEWEBPORT>" prometheus.io/path: "/metrics/prometheus/" ``` Note that similar to HTTP JSON Sink, fuse web port needs to be opened for accessing metrics by setting `alluxio.fuse.web.enabled` to true. For a Prometheus client to get the metrics from Alluxio, configure the `prometheus.yml` of the client. For example, to read the master metrics: ```yaml scrape_configs: job_name: 'alluxio master' kubernetessdconfigs: role: pod namespaces: names: alluxio # Only look at pods in namespace named `alluxio` relabel_configs: sourcelabels: [metakubernetespodlabel_role] action: keep regex: alluxio-master sourcelabels: [metakubernetespodannotationprometheusio_scrape] action: keep regex: true sourcelabels: [metakubernetespodannotationprometheusio_path] action: replace targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetespodannotationprometheusio_masterWebPort] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 targetlabel: address_ action: labelmap regex:

sourcelabels: [metakubernetes_namespace] action: replace target_label: namespace sourcelabels: [metakubernetespodname] action: replace targetlabel: podname sourcelabels: [metakubernetespodnode_name] action: replace target_label: node sourcelabels: [metakubernetespodlabel_release] action: replace targetlabel: clustername ``` To read other components' metrics, use the respective pod role label and web port label. An example configuration reading worker metrics ```yaml scrape_configs: job_name: 'alluxio worker' kubernetessdconfigs: role: pod namespaces: names: alluxio # Only look at pods in namespace named `alluxio` relabel_configs: sourcelabels: [metakubernetespodlabel_role] action: keep regex: alluxio-worker sourcelabels: [metakubernetespodannotationprometheusio_scrape] action: keep regex: true sourcelabels: [metakubernetespodannotationprometheusio_path] action: replace targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetespodannotationprometheusio_workerWebPort] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 targetlabel: address_ action: labelmap regex: metakubernetespodlabel(.+) sourcelabels: [metakubernetes_namespace] action: replace target_label: namespace sourcelabels: [metakubernetespodname] action: replace targetlabel: podname sourcelabels: [metakubernetespodnode_name] action: replace target_label: node sourcelabels: [metakubernetespodlabel_release] action: replace targetlabel: clustername ``` An example configuration reading job master metrics ```yaml scrape_configs: job_name: 'alluxio job worker' kubernetessdconfigs: role: pod namespaces: names: alluxio # Only look at pods in namespace named `alluxio` relabel_configs: sourcelabels: [metakubernetespodlabel_role] action: keep regex: alluxio-master sourcelabels: [metakubernetespodannotationprometheusio_scrape] action: keep regex: true sourcelabels: [metakubernetespodannotationprometheusio_path] action: replace targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetespodannotationprometheusio_jobMasterWebPort] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 targetlabel: address_ action: labelmap regex: metakubernetespodlabel(.+) sourcelabels: [metakubernetes_namespace] action: replace target_label: namespace sourcelabels: [metakubernetespodname] action: replace targetlabel: podname sourcelabels: [metakubernetespodnode_name] action: replace target_label: node sourcelabels: [metakubernetespodlabel_release] action: replace targetlabel: clustername ``` An example configuration reading job worker metrics ```yaml scrape_configs: job_name: 'alluxio job worker' kubernetessdconfigs: role: pod namespaces: names: alluxio # Only look at pods in namespace named `alluxio` relabel_configs: sourcelabels: [metakubernetespodlabel_role] action: keep regex: alluxio-worker sourcelabels: [metakubernetespodannotationprometheusio_scrape] action: keep regex: true sourcelabels: [metakubernetespodannotationprometheusio_path] targetlabel: metricspath regex: (.+) sourcelabels: [address, metakubernetespodannotationprometheusio_jobWorkerWebPort] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 targetlabel: address_ action: labelmap regex: metakubernetespodlabel(.+) sourcelabels: [metakubernetes_namespace] action: replace target_label: namespace sourcelabels: [metakubernetespodname] action: replace targetlabel: podname sourcelabels: [metakubernetespodnode_name] action: replace target_label: node sourcelabels: [metakubernetespodlabel_release] action: replace targetlabel: clustername ``` You can send an HTTP request to the Prometheus endpoint of an Alluxio process to get a snapshot of the metrics in Prometheus format. ```shell $ kubectl exec <COMPONENTHOSTNAME> -c <CONTAINERNAME> -- curl 127.0.0.1:<COMPONEMTWEBPORT>/metrics/prometheus/ ``` ```shell $ kubectl exec <alluxio-master-x> -c alluxio-master -- curl 127.0.0.1:19999/metrics/prometheus/ ``` ```shell $ kubectl exec <alluxio-worker-xxxxx> -c alluxio-worker -- curl 127.0.0.1:30000/metrics/prometheus/ ``` ```shell $ kubectl exec <alluxio-master-x> -c alluxio-job-master -- curl 127.0.0.1:20002/metrics/prometheus/ ``` ```shell $ kubectl exec <alluxio-worker-xxxxx> -c alluxio-job-worker -- curl 127.0.0.1:30003/metrics/prometheus/ ``` ```shell $ kubectl exec <alluxio-fuse-xxxxx> -- curl 127.0.0.1:49999/metrics/prometheus/ ```