Project: Fluid (Orchestration & Management / Scheduling & Orchestration)
File: monitoring.md

Note: Prometheus requires In-Cluster deployment. If there is no Prometheus in your cluster, you can use the following example for a quick try. However, this method is not recommended for production use. Please follow the to set up Prometheus correctly in your production environment. ```shell $ cd community $ kubectl apply -f integration/prometheus/prometheus.yaml ``` If you have Prometheus in your cluster, you can write the following configuration to the Prometheus configuration file: ```yaml scrape_configs: job_name: 'alluxio runtime' metrics_path: /metrics/prometheus kubernetessdconfigs: role: endpoints relabel_configs: sourcelabels: [metakubernetesservicelabel_monitor] regex: alluxioruntimemetrics action: keep sourcelabels: [metakubernetesendpointport_name] regex: web action: keep sourcelabels: [metakubernetes_namespace] target_label: namespace replacement: $1 action: replace sourcelabels: [metakubernetesservicelabel_release] targetlabel: fluidruntime replacement: $1 action: replace sourcelabels: [metakubernetesendpointaddresstargetname] target_label: pod replacement: $1 action: replace ``` ```shell $ docker run -d \ -p 3000:3000 \ --name=grafana \ --restart=always \ --name grafana \ grafana/grafana ``` You can install grafana in Kubernetes by following . Log-in Grafana For docker deployment, visit `http://$grafana-node-ip:3000`; for In-CLuster deployment, visit `http://$grafana-node-ip:NodePort` with default account and password `admin:admin`: ``` $ kubectl describe svc monitoring-grafana -n kube-system ``` First check the Prometheus svc port ``` $ kubectl get svc -n kube-system | grep prometheus-svc prometheus-svc NodePort 10.100.0.144 <none> 9090:31245/TCP 22h $ kubectl describe svc prometheus-svc -n kube-system Name: prometheus-svc Namespace: kube-system Labels: kubernetes.io/name=Prometheus name=prometheus-svc Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"kubernetes.io/name":"Prometheus","name":"prometheus-svc"},"nam... Selector: app=prometheus Type: NodePort IP: 10.100.0.144 Port: prometheus 9090/TCP TargetPort: 9090/TCP NodePort: prometheus 31245/TCP Endpoints: 10.99.224.138:9090 Session Affinity: None External Traffic Policy: Cluster Events: <none> ``` Configure Prometheus data source Note: For Grafana In-Cluster deployment, the URL should be Service Endpoints; for docker deployment, the URL should be Prometheus deployment node-ip:NodePort. After importing, click Save & Test to show that the Data source is working. Import the template file Grafana chooses to import the template Json file `fluid-prometheus-grafana-monitor.json`, which is located at `integration/prometheus/fluid-prometheus-grafana-monitor.json` Start Fluid Job ```yaml $ cat<<EOF >dataset.yaml apiVersion: data.fluid.io/v1alpha1 kind: Dataset metadata: name: spark spec: mounts: mountPoint: https://mirrors.bit.edu.cn/apache/spark/ name: spark apiVersion: data.fluid.io/v1alpha1 kind: AlluxioRuntime metadata: name: spark spec: replicas: 2 tieredstore: levels: mediumtype: MEM path: /dev/shm quota: 1Gi high: "0.95" low: "0.7" EOF ``` Note: Prometheus is turned on by default. If you need to turn off Prometheus, you can set disablePrometheus: true, default is false Checking the monitor In grafana HOME, you can find the view named Fluid-Prometheus-Grafana-Monitor, as follows: NoteUser of runtime correspond to Fluid Alluxio runtime user; fluid_runtime correspond to Fluid runtime name; namespace correspond to Fluid runtime namespace.