Project: CubeFS (Runtime / Cloud Native Storage)
File: view.md

::: tip Note The monitoring metrics template of the erasure coding subsystem is not included in the corresponding configuration module example yet, and will be improved later. ::: You can use Grafana to display monitoring data, as shown in the figure below Refer to the *.json files in docker/monitor/grafana/provisioning/dashboards/ for template configuration You can configure monitoring alarm capabilities through Prometheus Alertmanager, refer to the Related references + You can configure personalized monitoring panels based on the monitoring metrics items introduced earlier, such as the following panel examples. ::: tip Note Replace `cluster/cluster_id` with the cluster ID The `service` label is the service name, configured in the audit log metrics configuration item. ::: Cluster Physical Space Overview ```bash sum by (item)(blobstoreclusterMgrspacestatinfo{cluster="${clusterid}",item="FreeSpace",isleader="true"}) sum by (item)(blobstoreclusterMgrspacestatinfo{cluster="${clusterid}",item="UsedSpace",isleader="true"}) sum by (item)(blobstoreclusterMgrspacestatinfo{cluster="${clusterid}1",item="UsedSpace",isleader="true"}) ``` Volume Overview ```bash sum by (status) (blobstoreclusterMgrvolstatusvolcount{cluster="${clusterid}",is_leader="true"}) ``` SLA ```bash ((sum by (service)(rate(serviceresponsecode{service="ACCESS",code=~"4..|3..|2..|1.."}[5m]))>0)/(sum by (service)(rate(serviceresponsecode{service="ACCESS"}[5m]))>0))*100 ``` Upload Bandwidth ```bash sum by (idc) (rate(servicerequestlength{service="ACCESS",api=~"access.put|access.putat"}[5m]))*8 ``` Download Bandwidth ```bash sum by (idc) (rate(serviceresponselength{service="ACCESS",api=~"access.get"}[5m]))*8 ``` Five-Minute Request Exceptions ```bash sum by (api,service,code)(increase(serviceresponsecode{code!~"200|206|404|700|702|621|622|651|654|923"}[5m]))>0 ``` Request QPS ```bash sum by (service,idc,api)(rate(serviceresponsecode{}[5m])) ``` Average Upload Latency ```bash (histogramquantile(0.95, sum by(idc,le) (rate(serviceresponsedurationms_bucket{code=~"2..",service="ACCESS",api=~"access.put|access.putat"}[5m]))))>0 ``` Average Download Latency ```bash (histogramquantile(0.95, sum by(idc,le) (rate(serviceresponsedurationms_bucket{code=~"2..",service="ACCESS",api=~"access.get"}[5m]))))>0 ``` Average Deletion Latency ```bash histogramquantile(0.95, sum by(idc,le) (rate(serviceresponsedurationms_bucket{code=~"2..",service="ACCESS",api="access.delete"}[5m])))>0 ``` Proxy Codemode Overview ```bash sum by(idc,codemode) (blobstoreproxyvolumestatus{cluster="${clusterid}",type="totalfreesize"}) ``` Background Migration Rate ```bash sum by (tasktype) (rate(schedulertaskdatasize{clusterid="${clusterid}"}[5m]))*8 ``` Shard Migration Count ```bash sum by (tasktype) (rate(schedulertaskshardcnt{clusterid="${clusterid}"}[5m])) ``` Task Cancellation or Redistribution Count ```bash sum (increase(schedulertaskreclaim{clusterid="${clusterid}"}[5m])) by (task_type) sum (increase(schedulertaskcancel{clusterid="${clusterid}"}[5m])) by (task_type) ``` Background Task Count ```bash sum by (tasktype,taskstatus) (schedulertaskcnt{clusterid="${clusterid}"}) ``` Repair Message Consumption Latency ```bash min by (clusterid,topic,partition)(kafkatopicpartitionconsumelag{clusterid="${clusterid}",topic=~".*.shardrepair.|shard.",module_name="SCHEDULER"}) ``` Deletion Message Consumption Latency ```bash min by (clusterid,topic,partition)(kafkatopicpartitionconsumelag{clusterid="${clusterid}",topic=~"blobdelete.|..blobdelete.*",modulename="SCHEDULER"}) ``` Free Chunk Distribution ```bash sum(increase(schedulerfreechunkcntrange_bucket{}[1m])) by(le)>0 ```