Project: MinIO (Runtime / Cloud Native Storage)
File: v3.md

In metrics version 3, all metrics are available under the endpoint: ``` /minio/metrics/v3 ``` however, a specific path under this is required. Metrics are organized into groups at paths relative to the top-level endpoint above. Each endpoint below can be queried at different intervals as needed via a scrape configuration in Prometheus or a compatible metrics collection tool. For ease of configuration, each (non-empty) parent of the path serves all metric endpoints that are at descendant paths. For example, to query all system metrics one needs to only scrape `/minio/metrics/v3/system/`. Some metrics are bucket specific. These will have a `/bucket` component in their path. As the number of buckets can be large, the metrics scrape operation needs to be provided with a specific list of buckets via the `bucket` query parameter. Only metrics for the given buckets will be returned (with the bucket label set). For example to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`. Instead of a metrics scrape, it is also possible to list the metrics that would be returned by a path. This is done by adding a `?list` query parameter. The MinIO server will then list all possible metrics that could be returned. During an actual metrics scrape, only available metrics are returned - not all of them. With the `list` query parameter, the output format can be selected - just set the request `Content-Type` to `application/json` for JSON output, or `text/plain` for a simple markdown formatted table. The latter is the default. At a high level metrics are grouped into three categories, listed in the following sub-sections. The path in each of the tables is relative to the top-level endpoint. These are metrics about requests served by the (current) node. | Path | Description | |--|--| | `/api/requests` | Metrics over all requests | | `/api/bucket` | Metrics over all requests split by bucket labels | | | | These are metrics about the minio process and the node. | Path | Description | |-|-| | `/audit` | Metrics related to audit functionality | These are metrics about the minio logger webhooks | Path | Description | |-|| | `/logger/webhook` | Metrics related to logger webhooks | These are metrics about the minio process and the node. | Path | Description | |--|| | `/system/drive` | Metrics about drives on the system | | `/system/memory` | Metrics about memory on the system | | `/system/network/internode` | Metrics about internode requests made by the node | | `/system/process` | Standard process metrics | | | | These are metrics for debugging | Path | Description | |--|| | `/debug/go` | Standard Go lang metrics | | | | These present metrics about the whole MinIO cluster. | Path | Description | |--|--| | `/cluster/health` | Cluster health metrics | | `/cluster/usage/objects` | Object statistics | | `/cluster/usage/buckets` | Object statistics by bucket | | `/cluster/erasure-set` | Erasure set metrics | | | | Each of the following sub-sections list metrics returned by each of the endpoints. The standard metrics group for GoCollector is not shown

| Name | Type | Help | Labels | ||--||-| | `minioapirequestsrejectedauthtotal` | `counter` | Total number of requests rejected for auth failure | `type,poolindex,server` | | `minioapirequestsrejectedheadertotal` | `counter` | Total number of requests rejected for invalid header | `type,poolindex,server` | | `minioapirequestsrejectedtimestamptotal` | `counter` | Total number of requests rejected for invalid timestamp | `type,poolindex,server` | | `minioapirequestsrejectedinvalidtotal` | `counter` | Total number of invalid requests | `type,poolindex,server` | | `minioapirequestswaitingtotal` | `gauge` | Total number of requests in the waiting queue | `type,pool_index,server` | | `minioapirequestsincomingtotal` | `gauge` | Total number of incoming requests | `type,pool_index,server` | | `minioapirequestsinflighttotal` | `gauge` | Total number of requests currently in flight | `name,type,pool_index,server` | | `minioapirequeststotal` | `counter` | Total number of requests | `name,type,poolindex,server` | | `minioapirequestserrorstotal` | `counter` | Total number of requests with (4xx and 5xx) errors | `name,type,pool_index,server` | | `minioapirequests5xxerrorstotal` | `counter` | Total number of requests with 5xx errors | `name,type,poolindex,server` | | `minioapirequests4xxerrorstotal` | `counter` | Total number of requests with 4xx errors | `name,type,poolindex,server` | | `minioapirequestscanceledtotal` | `counter` | Total number of requests canceled by the client | `name,type,pool_index,server` | | `minioapirequeststtfbsecondsdistribution` | `counter` | Distribution of time to first byte across API calls | `name,type,le,poolindex,server` | | `minioapirequeststrafficsentbytes` | `counter` | Total number of bytes sent | `type,poolindex,server` | | `minioapirequeststrafficreceivedbytes` | `counter` | Total number of bytes received | `type,poolindex,server` | | Name | Type | Help | Labels | |-|--||--| | `miniobucketapitrafficreceivedbytes` | `counter` | Total number of bytes sent for a bucket | `bucket,type,server,poolindex` | | `miniobucketapitrafficsentbytes` | `counter` | Total number of bytes received for a bucket | `bucket,type,server,poolindex` | | `miniobucketapiinflighttotal` | `gauge` | Total number of requests currently in flight for a bucket | `bucket,name,type,server,pool_index` | | `miniobucketapitotal` | `counter` | Total number of requests for a bucket | `bucket,name,type,server,poolindex` | | `miniobucketapicanceledtotal` | `counter` | Total number of requests canceled by the client for a bucket | `bucket,name,type,server,pool_index` | | `miniobucketapi4xxerrorstotal` | `counter` | Total number of requests with 4xx errors for a bucket | `bucket,name,type,server,poolindex` | | `miniobucketapi5xxerrorstotal` | `counter` | Total number of requests with 5xx errors for a bucket | `bucket,name,type,server,poolindex` | | `miniobucketapittfbsecondsdistribution` | `counter` | Distribution of time to first byte across API calls for a bucket | `bucket,name,le,type,server,poolindex` | | Name | Type | Help | Labels | |--|--|-|--| | `minioauditfailedmessages` | `counter` | Total number of messages that failed to send since start | `targetid,server` | | `minioaudittargetqueuelength` | `gauge` | Number of unsent messages in queue for target | `target_id,server` | | `minioaudittotalmessages` | `counter` | Total number of messages sent since start | `targetid,server` | | Name | Type | Help | Labels | ||--|--|--| | `miniosystemdriveusedbytes` | `gauge` | Total storage used on a drive in bytes | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivefreebytes` | `gauge` | Total storage free on a drive in bytes | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivetotalbytes` | `gauge` | Total storage available on a drive in bytes | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdriveusedinodes` | `gauge` | Total used inodes on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivefreeinodes` | `gauge` | Total free inodes on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivetotalinodes` | `gauge` | Total inodes available on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivetimeouterrorstotal` | `counter` | Total timeout errors on a drive | `drive,setindex,driveindex,poolindex,server` | | `miniosystemdriveioerrorstotal` | `counter` | Total I/O errors on a drive | `drive,setindex,driveindex,poolindex,server` | | `miniosystemdriveavailabilityerrorstotal` | `counter` | Total availability errors (I/O errors, timeouts) on a drive | `drive,setindex,driveindex,poolindex,server` | | `miniosystemdrivewaitingio` | `gauge` | Total waiting I/O operations on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdriveapilatencymicros` | `gauge` | Average last minute latency in s for drive API storage operations | `drive,api,setindex,driveindex,poolindex,server` | | `miniosystemdriveofflinecount` | `gauge` | Count of offline drives | `pool_index,server` | | `miniosystemdriveonlinecount` | `gauge` | Count of online drives | `pool_index,server` | | `miniosystemdrivecount` | `gauge` | Count of all drives | `poolindex,server` | | `miniosystemdrivehealth` | `gauge` | Drive health (0 = offline, 1 = healthy, 2 = healing) | `drive,setindex,driveindex,poolindex,server` | | `miniosystemdrivereadspersec` | `gauge` | Reads per second on a drive | `drive,setindex,driveindex,poolindex,server` | | `miniosystemdrivereadskbpersec` | `gauge` | Kilobytes read per second on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivereadsawait` | `gauge` | Average time for read requests served on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivewritespersec` | `gauge` | Writes per second on a drive | `drive,setindex,driveindex,poolindex,server` | | `miniosystemdrivewriteskbpersec` | `gauge` | Kilobytes written per second on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivewritesawait` | `gauge` | Average time for write requests served on a drive | `drive,setindex,driveindex,pool_index,server` | | `miniosystemdrivepercutil` | `gauge` | Percentage of time the disk was busy | `drive,setindex,driveindex,pool_index,server` | | Name | Type | Help | Labels | |-|||-| | `miniosystemmemory_used` | `gauge` | Used memory on the node | `server` | | `miniosystemmemoryusedperc` | `gauge` | Used memory percentage on the node | `server` | | `miniosystemmemory_free` | `gauge` | Free memory on the node | `server` | | `miniosystemmemory_total` | `gauge` | Total memory on the node | `server` | | `miniosystemmemory_buffers` | `gauge` | Buffers memory on the node | `server` | | `miniosystemmemory_cache` | `gauge` | Cache memory on the node | `server` | | `miniosystemmemory_shared` | `gauge` | Shared memory on the node | `server` | | `miniosystemmemory_available` | `gauge` | Available memory on the node | `server` | | Name | Type | Help | Labels | |-|||-| | `miniosystemcpuavgidle` | `gauge` | Average CPU idle time | `server` | | `miniosystemcpuavgiowait` | `gauge` | Average CPU IOWait time | `server` | | `miniosystemcpu_load` | `gauge` | CPU load average 1min | `server` | | `miniosystemcpuloadperc` | `gauge` | CPU load average 1min (percentage) | `server` | | `miniosystemcpu_nice` | `gauge` | CPU nice time | `server` | | `miniosystemcpu_steal` | `gauge` | CPU steal time | `server` | | `miniosystemcpu_system` | `gauge` | CPU system time | `server` | | `miniosystemcpu_user` | `gauge` | CPU user time | `server` | | Name | Type | Help | Labels | ||--|-|| | `miniosystemnetworkinternodeerrorstotal` | `counter` | Total number of failed internode calls | `server,poolindex` | | `miniosystemnetworkinternodedialerrorstotal` | `counter` | Total number of internode TCP dial timeouts and errors | `server,pool_index` | | `miniosystemnetworkinternodedialavgtimenanos` | `gauge` | Average dial time of internodes TCP calls in nanoseconds | `server,poolindex` | | `miniosystemnetworkinternodesentbytestotal` | `counter` | Total number of bytes sent to other peer nodes | `server,pool_index` | | `miniosystemnetworkinternoderecvbytestotal` | `counter` | Total number of bytes received from other peer nodes | `server,pool_index` | | Name | Type | Help | Labels | |-|--|-|-| | `locksreadtotal` | `gauge` | Number of current READ locks on this peer | `server` | | `lockswritetotal` | `gauge` | Number of current WRITE locks on this peer | `server` | | `cputotalseconds` | `counter` | Total user and system CPU time spent in seconds | `server` | | `goroutinetotal` | `gauge` | Total number of go routines running | `server` | | `iorcharbytes` | `counter` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar | `server` | | `ioreadbytes` | `counter` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes | `server` | | `iowcharbytes` | `counter` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar | `server` | | `iowritebytes` | `counter` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes | `server` | | `starttimeseconds` | `gauge` | Start time for MinIO process in seconds since Unix epoc | `server` | | `uptime_seconds` | `gauge` | Uptime for MinIO process in seconds | `server` | | `filedescriptorlimit_total` | `gauge` | Limit on total number of open file descriptors for the MinIO Server process | `server` | | `filedescriptoropen_total` | `gauge` | Total number of open file descriptors by the MinIO Server process | `server` | | `syscallreadtotal` | `counter` | Total read SysCalls to the kernel. /proc/[pid]/io syscr | `server` | | `syscallwritetotal` | `counter` | Total write SysCalls to the

/proc/[pid]/io syscw | `server` | | `residentmemorybytes` | `gauge` | Resident memory size in bytes | `server` | | `virtualmemorybytes` | `gauge` | Virtual memory size in bytes | `server` | | `virtualmemorymax_bytes` | `gauge` | Maximum virtual memory size in bytes | `server` | | Name | Type | Help | Labels | |-|||--| | `minioclusterhealthdrivesoffline_count` | `gauge` | Count of offline drives in the cluster | | | `minioclusterhealthdrivesonline_count` | `gauge` | Count of online drives in the cluster | | | `minioclusterhealthdrivescount` | `gauge` | Count of all drives in the cluster | | | `minioclusterhealthnodesoffline_count` | `gauge` | Count of offline nodes in the cluster | | | `minioclusterhealthnodesonline_count` | `gauge` | Count of online nodes in the cluster | | | `minioclusterhealthcapacityrawtotalbytes` | `gauge` | Total cluster raw storage capacity in bytes | | | `minioclusterhealthcapacityrawfreebytes` | `gauge` | Total cluster raw storage free in bytes | | | `minioclusterhealthcapacityusabletotalbytes` | `gauge` | Total cluster usable storage capacity in bytes | | | `minioclusterhealthcapacityusablefreebytes` | `gauge` | Total cluster usable storage free in bytes | | | Name | Type | Help | Labels | |-||-|| | `minioclusterusageobjectssincelastupdate_seconds` | `gauge` | Time since last update of usage metrics in seconds | | | `minioclusterusageobjectstotal_bytes` | `gauge` | Total cluster usage in bytes | | | `minioclusterusageobjectscount` | `gauge` | Total cluster objects count | | | `minioclusterusageobjectsversions_count` | `gauge` | Total cluster object versions (including delete markers) count | | | `minioclusterusageobjectsdeletemarkerscount` | `gauge` | Total cluster delete markers count | | | `minioclusterusageobjectsbuckets_count` | `gauge` | Total cluster buckets count | | | `minioclusterusageobjectssize_distribution` | `gauge` | Cluster object size distribution | `range` | | `minioclusterusageobjectsversioncountdistribution` | `gauge` | Cluster object version count distribution | `range` | | Name | Type | Help | Labels | |--|||-| | `minioclusterusagebucketssincelastupdate_seconds` | `gauge` | Time since last update of usage metrics in seconds | | | `minioclusterusagebucketstotal_bytes` | `gauge` | Total bucket size in bytes | `bucket` | | `minioclusterusagebucketsobjects_count` | `gauge` | Total objects count in bucket | `bucket` | | `minioclusterusagebucketsversions_count` | `gauge` | Total object versions (including delete markers) count in bucket | `bucket` | | `minioclusterusagebucketsdeletemarkerscount` | `gauge` | Total delete markers count in bucket | `bucket` | | `minioclusterusagebucketsquotatotalbytes` | `gauge` | Total bucket quota in bytes | `bucket` | | `minioclusterusagebucketsobjectsizedistribution` | `gauge` | Bucket object size distribution | `range,bucket` | | `minioclusterusagebucketsobjectversioncount_distribution` | `gauge` | Bucket object version count distribution | `range,bucket` | | Name | Type | Help | Labels | |--||--|| | `minioclustererasuresetoverallwritequorum` | `gauge` | Overall write quorum across pools and sets | | | `minioclustererasuresetoverall_health` | `gauge` | Overall health across pools and sets (1=healthy, 0=unhealthy) | | | `minioclustererasuresetreadquorum` | `gauge` | Read quorum for the erasure set in a pool | `poolid,set_id` | | `minioclustererasuresetwritequorum` | `gauge` | Write quorum for the erasure set in a pool | `poolid,set_id` | | `minioclustererasuresetonlinedrivescount` | `gauge` | Count of online drives in the erasure set in a pool | `poolid,setid` | | `minioclustererasuresethealingdrivescount` | `gauge` | Count of healing drives in the erasure set in a pool | `poolid,setid` | | `minioclustererasuresethealth` | `gauge` | Health of the erasure set in a pool (1=healthy, 0=unhealthy) | `poolid,setid` | | `minioclustererasuresetreadtolerance` | `gauge` | No of drive failures that can be tolerated without disrupting read operations | `poolid,set_id` | | `minioclustererasuresetwritetolerance` | `gauge` | No of drive failures that can be tolerated without disrupting write operations | `poolid,set_id` | | `minioclustererasuresetreadhealth` | `gauge` | Health of the erasure set in a pool for read operations (1=healthy,