Project: CubeFS (Runtime / Cloud Native Storage)
File: metrics.md

CubeFS integrates Prometheus as the monitoring metric collection module, which can be combined with specific situations to enable monitoring metrics or configure monitoring panels. Support reporting go gc stats and mem stats, as follows ```bash gogcdurationsecondssum gogcduration_seconds ... gomemstatsalloc_bytes gomemstatsheapidlebytes ... processresidentmemory_bytes processstarttime_seconds processopenfds ... ``` The monitoring metrics reported by the `Master` module mainly include the health status, usage, and statistical data of nodes in the cluster. | Metric Name | Description | |--|| | cfsmasterdataNodes_count | Number of data nodes in the cluster | | cfsmasterdataNodes_inactive | Number of abnormal data nodes in the cluster | | cfsmasterdataNodesincreasedGB | Change in disk space usage in the cluster within 2 minutes | | cfsmasterdataNodestotalGB | Total disk space size in the cluster | | cfsmasterdataNodesusedGB | Disk space size used in the cluster | | cfsmasterdisk_error{addr="xx",path="xx"} | Bad disk monitoring in the cluster, including abnormal node IP and disk path | | cfsmastermetaNodes_count | Total number of metadata nodes in the cluster | | cfsmastermetaNodes_inactive | Number of abnormal metadata nodes in the cluster | | cfsmastermetaNodesincreasedGB | Change in memory usage of metadata within 2 minutes in the cluster | | cfsmastermetaNodestotalGB | Total memory size of metadata in the cluster | | cfsmastermetaNodesusedGB | Memory size used by metadata in the cluster | | cfsmastervol_count | Number of volumes in the cluster | | cfsmastervolmetacount{type="dentry",volName="vol"} | Details of the specified volume, type: dentry, inode, dp, mp | | cfsmastervoltotalGB{volName="xx"} | Capacity of the specified volume | | cfsmastervolusageratio{volName="xx"} | Usage rate of the specified volume | | cfsmastervolusedGB{volName="xx"} | Used capacity of the specified volume | | cfsmasternodesetdatatotal_GB{nodeset="xx"} | The sum of the total space of all data nodes on the specified nodeset | | cfsmasternodesetdatausage_ratio{nodeset="xx"} | The used data space ratio on the specified nodeset | | cfsmasternodesetdataused_GB{nodeset="xx"} | The sum of available space of all data nodes on the specified nodeset | | cfsmasternodesetdpreplica_count{nodeset="xx"} | The number of data partition replicas on the specified nodeset | | cfsmasternodesetmetatotal_GB{nodeset="xx"} | The sum of the total space of all meta nodes on the specified nodeset | | cfsmasternodesetmetausage_ratio{nodeset="xx"} | The used meta space ratio on the specified nodeset | | cfsmasternodesetmetaused_GB{nodeset="xx"} | The sum of available space of all meta nodes on the specified nodeset | | cfsmasternodesetmpreplica_count{nodeset="xx"} | The number of meta partition replicas on the specified nodeset | The monitoring metrics of the `MetaNode` can be used to monitor the qps and latency data of various metadata operations of each volume, such as lookup, createInode, createDentry,

| Metric Name | Description | ||-| | cfsmetanode$op_count | Total number of requests for the corresponding operation of the meta node, which can be used to calculate the request qps | | cfsmetanode$ophistbucket | Hist data of the corresponding operation request of the meta node, which can be used to calculate the 95 value of the latency | | cfsmetanode$ophistcount | Total number of corresponding requests for the meta node, same as cfsmetanode$op_count | | cfsmetanode$ophistsum | Total time consumption of the corresponding operation request of the meta node, which can be used to calculate the average latency with hist_count | The monitoring metrics of the `DataNode` can be used to monitor the qps and latency data of various data operations of each volume, as well as bandwidth, such as read, write, etc. | Metric Name | Description | ||-| | cfsdataNode$op_count | Total number of requests for the corresponding operation of the data node, which can be used to calculate the request qps | | cfsdataNode$ophistbucket | Hist data of the corresponding operation request of the data node, which can be used to calculate the 95 value of the latency | | cfsdataNode$ophistcount | Total number of corresponding requests for the data node, same as cfsdatanode$op_count | | cfsdataNode$ophistsum | Total time consumption of the corresponding operation request of the data node, which can be used to calculate the average latency with hist_count | | cfsdataNodedataPartitionIOBytes | Total amount of data read and written by the data node, which can be used to calculate the bandwidth data of the specified disk and volume | | cfsdataNodedataPartitionIO_count | Total number of IOs of the data node, which can be used to calculate the disk IO qps data | | cfsdataNodedataPartitionIOhistbucket | Histogram data of the IO operation of the data node, which can be used to calculate the 95 value of the IO | | cfsdataNodedataPartitionIOhistcount | Total number of IO operations of the data node, same as above | | cfsdataNodedataPartitionIOhistsum | Total delay of the IO operation of the data node, which can be used to calculate the average delay with hist_count | The monitoring metrics of the `ObjectNode` are mainly used to monitor the request volume and time consumption of various S3 operations, such as copyObject, putObject, etc. | Metric Name | Description | |--|| | cfsobjectnode$op_count | Total number of requests for the corresponding operation of the object node, which can be used to calculate the qps | | cfsobjectnode$ophistcount | Same as above | | cfsobjectnode$ophistsum | Total time consumption of the corresponding operation request of the object node, which can be used to calculate the average latency with hist_count | | cfsobjectnode$ophistbucket | Histogram data of the corresponding request of the object node, which can be used to calculate the 95 value and 99 value of the request latency | The monitoring metrics of the `FuseClient` module are mainly used to monitor the request volume, time consumption, cache hit rate, etc. of the interaction with the data module or metadata module, such as fileread, filewrite,

The details are as follows: | Metric Name | Description | |--|-| | cfsfuseclient$dp_count | Total number of requests for the corresponding operation of the client, which can be used to calculate the qps | | cfsfuseclient$dphistcount | Same as above | | cfsfuseclient$dphistsum | Total time consumption of the corresponding operation request of the client, which can be used to calculate the average latency with hist_count | | cfsfuseclient$dphistbucket | Histogram data of the corresponding request of the client, which can be used to calculate the 95 value of the request latency | | Label | Description | ||-| | api | Request API name, can configure path depth | | code | Response status code | | host | Host name, automatically obtain the current host name | | idc | Configuration file configuration item | | method | Request type | | service | Service name | | tag | Custom tag | | team | Custom team name | You can modify the service audit log configuration item to enable related metrics | Configuration Item | Description | Required | |:--|:|:-| | idc | IDC name | No, it is recommended to fill in if the metric is enabled | | service | Module name | No, it is recommended to fill in if the metric is enabled | | tag | Custom tag, such as configuring clusterid | No, it is recommended to fill in if the metric is enabled | | enablehttpmethod | Whether to enable status code statistics | No, default is off | | enablereqlength_cnt | Whether to enable request length statistics | No, default is off | | enableresplength_cnt | Whether to enable response length statistics | No, default is off | | enablerespduration | Whether to enable request/response interval time consumption statistics | No, default is off | | maxapilevel | Maximum API path depth | No | ```json { "auditlog": { "metric_config": { "idc": "z0", "service": "SCHEDULER", "tag": "100", "team": "cubefs", "enablehttpmethod": true, "enablereqlength_cnt": true, "enableresplength_cnt": true, "enablerespduration": true, "maxapilevel": 3 } } } ``` service_response_code Request status code metrics, record request API, status code, module name and other information, generally used to calculate request error rate, request volume, etc. ```bash serviceresponsecode{api="scheduler.inspect.acquire",code="200",host="xxx",idc="z0",method="GET",service="SCHEDULER",tag="100",team=""} 8.766433e+06 ``` service_request_length Request body length statistics metrics, generally used to calculate request bandwidth, traffic. ```bash servicerequestlength{api="clustermgr.chunk.report",code="200",host="xxx",idc="z0",method="POST",service="CLUSTERMGR",tag="",team=""} 27631 ``` service_response_length Request response body length statistics metrics, generally used to calculate download bandwidth, traffic. ```bash serviceresponselength{api="clustermgr.chunk.report",code="200",host="xxxx",idc="z0",method="POST",service="CLUSTERMGR",tag="",team=""} 6 ``` service_response_duration_ms Request response time statistics metrics, which can be used to calculate the 95 latency of the request or response. ```bash serviceresponsedurationmsbucket{api="clustermgr.config.get",code="200",host="xxx",idc="z0",method="GET",reqlength="",resplength="",service="CLUSTERMGR",tag="",team="",le="1"} 22 ``` blobstore_access_cache_hit_rate Cache hit rate, status includes four types: none (the hit value is nil), hit (hit), miss (not hit), expired (expired) | Label | Description | ||--| | cluster | Cluster ID | | service | Monitoring component, such as memcache, proxy, etc. | | status | Status, none, hit, miss, expired | ```bash blobstoreaccesscachehitrate{cluster="100",service="memcache",status="hit"} 3.4829103e+13 blobstoreaccesscachehitrate{cluster="100",service="proxy",status="hit"} 2.4991594e+07 ``` blobstore_access_unhealth Failure degradation metrics statistics | Label | Description | ||--| | cluster | Cluster ID | | action | allocate, punish, repair.msg,

| | host | Failed node | | reason | Failure reason | | module | Degradation dimension, diskwith, volume, service | ```bash blobstoreaccessunhealth{action="punish",cluster="100",host="xxx",module="diskwith",reason="Timeout"} 7763 ``` blobstore_access_download Download failure statistics metrics | Label | Description | ||--| | cluster | Cluster ID | | way | Download method, EC read or direct read (Direct) | ```bash blobstoreaccessdownload{cluster="100",way="Direct"} 37 blobstoreaccessdownload{cluster="100",way="EC"} 3016 ``` blobstore_clusterMgr_chunk_stat_info Chunk status metrics, statistics of the total number of chunks and available chunks | Label | Description | |--|-| | cluster | Cluster ID | | idc | IDC | | region | Region information | | is_leader | Whether it is the master node | | item | TotalChunk, TotalFreeChunk | ```bash blobstoreclusterMgrchunkstatinfo{cluster="100",idc="z0",is_leader="false",item="TotalChunk",region="cn-south-2"} 55619 ``` blobstore_clusterMgr_disk_stat_info Disk status metrics | Label | Description | |--|-| | cluster | Cluster ID | | idc | IDC | | region | Region information | | is_leader | Whether it is the master node | | item | Available, Broken, Dropped, Dropping, Expired, Readonly, Repaired, Repairing, Total | ```bash blobstoreclusterMgrdiskstatinfo{cluster="100",idc="z0",is_leader="false",item="Available",region="cn-south-2"} 107 ``` blobstore_clusterMgr_raft_stat Raft status metrics | Label | Description | |--|| | cluster | Cluster ID | | idc | IDC | | region | Region information | | is_leader | Whether it is the master node | | item | appliedindex, committedindex, peers, term | ```bash blobstoreclusterMgrraftstat{cluster="100",isleader="false",item="applied_index",region="cn-south-2"} 2.97061597e+08 ``` blobstore_clusterMgr_space_stat_info Cluster space metrics | Label | Description | |--|| | cluster | Cluster ID | | idc | IDC | | region | Region information | | is_leader | Whether it is the master node | | item | FreeSpace, TotalBlobNode, TotalDisk, TotalSpace, UsedSpace, WritableSpace | ```bash blobstoreclusterMgrspacestatinfo{cluster="100",is_leader="false",item="FreeSpace",region="cn-south-2"} 1.76973072064512e+15 ``` blobstore_clusterMgr_vol_status_vol_count Volume status metrics | Label | Description | |--|| | cluster | Cluster ID | | idc | IDC | | region | Region information | | is_leader | Whether it is the master node | | status | active, allocatable, idle, lock, total, unlocking | ```bash blobstoreclusterMgrvolstatusvolcount{cluster="100",isleader="false",region="cn-south-2",status="active"} 316 ``` blobstore_blobnode_disk_stat Disk status metrics | Label | Description | ||| | cluster_id | Cluster ID | | idc | IDC | | disk_id | Disk ID | | host | Local service address | | rack | Rack information | | item | free, reserved, totaldisksize, used | ```bash blobstoreblobnodediskstat{clusterid="100",disk_id="243",host="xxx",idc="z2",item="free",rack="testrack"} 6.47616868352e+12 ``` scheduler_task_shard_cnt Task shard count | Label | Description | ||-| | cluster_id | Cluster ID | | kind | success, failed | | tasktype | Task type, delete, shardrepair, balance, diskdrop, diskrepair, manual_migrate | ```bash schedulertaskshardcnt{clusterid="100",kind="failed",task_type="delete"} 7.4912551e+07 ``` scheduler_task_reclaim Task redistribution metrics | Label | Description | ||| | cluster_id | Cluster ID | | kind | success, failed | | tasktype | Task type, balance, diskdrop, diskrepair, manualmigrate | ```bash schedulertaskreclaim{clusterid="100",kind="success",tasktype="balance"} 0 ``` scheduler_task_data_size Task data migration metrics, unit: byte | Label | Description | ||| | cluster_id | Cluster ID | | kind | success, failed | | tasktype | Task type, balance, diskdrop, diskrepair, manualmigrate | ```bash schedulertaskdatasize{clusterid="100",kind="success",task_type="balance"} 0 ``` scheduler_task_cnt Task count metrics | Label | Description | |-|| | cluster_id | Cluster ID | | kind | success, failed | | tasktype | Task type, balance, diskdrop, diskrepair, manualmigrate | | taskstatus | finishing, preparing, workerdoing | ```bash schedulertaskcnt{clusterid="100",kind="success",taskstatus="finishing",task_type="balance"} 0 ``` scheduler_task_cancel Task cancellation metrics | Label | Description | ||| | cluster_id | Cluster ID | | kind | success, failed | | tasktype | Task type, balance, diskdrop, diskrepair, manualmigrate | ```bash schedulertaskcancel{clusterid="100",kind="success",tasktype="balance"} 0 ``` scheduler_free_chunk_cnt_range Cluster idle chunk statistics | Label | Description | ||-| | cluster_id | Cluster ID | | idc | IDC | | rack | Rack | ```bash schedulerfreechunkcntrangebucket{clusterid="100",idc="z0",rack="testrack",le="5"} 0 ``` kafka_topic_partition_consume_lag Kafka