Project: Longhorn (Runtime / Cloud Native Storage)
File: 20200909-prometheus-support.md

We currently do not have a way for users to monitor and alert about events happen in Longhorn such as volume is full, backup is failed, CPU usage, memory consumption. This enhancement exports Prometheus metrics so that users can use Prometheus or other monitoring systems to monitor Longhorn. https://github.com/longhorn/longhorn/issues/1180 We are planing to expose 22 metrics in this release: longhornvolumecapacity_bytes longhornvolumeactualsizebytes longhornvolumestate longhornvolumerobustness longhornnodestatus longhornnodecount_total longhornnodecpucapacitymillicpu longhornnodecpuusagemillicpu longhornnodememorycapacitybytes longhornnodememoryusagebytes longhornnodestoragecapacitybytes longhornnodestorageusagebytes longhornnodestoragereservationbytes longhorndiskcapacity_bytes longhorndiskusage_bytes longhorndiskreservation_bytes longhorninstancemanagercpuusage_millicpu longhorninstancemanagercpurequests_millicpu longhorninstancemanagermemoryusage_bytes longhorninstancemanagermemoryrequests_bytes longhornmanagercpuusagemillicpu longhornmanagermemoryusagebytes See the section for definition of each metric. We are not planing to expose 6 metrics in this release: longhornbackupstatsnumberfailed_backups longhornbackupstatsnumbersucceed_backups longhornbackupstatsbackupstatus (status for this backup (0=InProgress,1=Done,2=Failed)) longhornvolumeio_ops longhornvolumeioreadthroughput longhornvolumeiowritethroughput Longhorn already has a great UI with many useful information. However, Longhorn doesn't have any alert/notification mechanism yet. Also, we don't have any dashboard or graphing support so that users can have overview picture of the storage system. This enhancement will address both of the above issues. In many cases, a problem/issue can be quickly discovered if we have a monitoring dashboard. For example, there are many times users ask us for supporting and the problems were that the Longhorn engines were killed due to over-use CPU limit. If there is a CPU monitoring dashboard for instance managers, those problems can be quickly detected. User want to be notified about abnormal event such as disk space limit approaching. We can expose metrics provide information about it and user can scrape the metrics and setup alert system. After this enhancement is merged, Longhorn expose metrics at end point `/metrics` in Prometheus' . Users can use Prometheus or other monitoring systems to collect those metrics by scraping the end point `/metrics` in longhorn manager. Then, user can display the collected data using tools such as Grafana. User can also setup alert by using tools such as Prometheus Alertmanager. Below are the descriptions of metrics which Longhorn exposes and how users can use them: longhornvolumecapacity_bytes This metric reports the configured size in bytes for each volume which is managed by the current longhorn manager. This metric contains 2 labels (dimensions): `node`: the node of the longhorn manager which is managing this volume `volume`: the name of this volume Example of a sample of this metric could be: ``` longhornvolumecapacity_bytes{node="worker-2",volume="testvol"} 6.442450944e+09 ``` Users can use this metrics to draw graph about and quickly see the big volumes in the storage system. longhornvolumeactualsizebytes This metric reports the actual space used by each replica of the volume on the corresponding nodes This metric contains 2 labels (dimensions): `node`: the node of the longhorn manager which is managing this volume `volume`: the name of this volume Example of a sample of this metric could be: ``` longhornvolumeactualsizebytes{node="worker-2",volume="testvol"} 1.1917312e+08 ``` Users can use this metrics to the actual size occupied on disks of Longhorn volumes longhornvolumestate This metric reports the state of the volume. The states are: 1=creating, 2=attached, 3=Detached, 4=Attaching, 5=Detaching, 6=Deleting. This metric contains 2 labels (dimensions): `node`: the node of the longhorn manager which is managing this volume `volume`: the name of this volume Example of a sample of this metric could be: ``` longhornvolumestate{node="worker-3",volume="testvol1"} 2 ``` longhornvolumerobustness This metric reports the robustness of the

Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted This metric contains 2 labels (dimensions): `node`: the node of the longhorn manager which is managing this volume `volume`: the name of this volume Example of a sample of this metric could be: ``` longhornvolumerobustness{node="worker-3",volume="testvol1"} 1 ``` longhornnodestatus This metric reports the `ready`, `schedulable`, `mountPropagation` condition for the current node. This metric contains 3 labels (dimensions): `node` `condition`: the name of the condition (`ready`, `schedulable`, `mountPropagation`) `condition_reason` Example of a sample of this metric could be: ``` longhornnodestatus{condition="allowScheduling",condition_reason="",node="worker-3"} 1 longhornnodestatus{condition="mountpropagation",condition_reason="",node="worker-3"} 1 longhornnodestatus{condition="ready",condition_reason="",node="worker-3"} 1 longhornnodestatus{condition="schedulable",condition_reason="",node="worker-3"} 1 ``` Users can use this metrics to setup alert about node status. longhornnodecount_total This metric reports the total nodes in Longhorn system. Example of a sample of this metric could be: ``` longhornnodecount_total 3 ``` Users can use this metric to detect the number of down nodes longhornnodecpucapacitymillicpu Report the maximum allocatable cpu on this node Example of a sample of this metric could be: ``` longhornnodecpucapacitymillicpu{node="worker-3"} 2000 ``` longhornnodecpuusagemillicpu Report the cpu usage on this node Example of a sample of this metric could be: ``` longhornnodecpuusagemillicpu{node="worker-3"} 149 ``` longhornnodememorycapacitybytes Report the maximum allocatable memory on this node Example of a sample of this metric could be: ``` longhornnodememorycapacitybytes{node="worker-3"} 4.031217664e+09 ``` longhornnodememoryusagebytes Report the memory usage on this node Example of a sample of this metric could be: ``` longhornnodememoryusagebytes{node="worker-3"} 1.643794432e+09 ``` longhornnodestoragecapacitybytes Report the storage capacity of this node Example of a sample of this metric could be: ``` longhornnodestoragecapacitybytes{node="worker-3"} 8.3987283968e+10 ``` longhornnodestorageusagebytes Report the used storage of this node Example of a sample of this metric could be: ``` longhornnodestorageusagebytes{node="worker-3"} 9.060212736e+09 ``` longhornnodestoragereservationbytes Report the reserved storage for other applications and system on this node Example of a sample of this metric could be: ``` longhornnodestoragereservationbytes{node="worker-3"} 2.519618519e+10 ``` longhorndiskcapacity_bytes Report the storage capacity of this disk. Example of a sample of this metric could be: ``` longhorndiskcapacity_bytes{disk="default-disk-8b28ee3134628183",node="worker-3"} 8.3987283968e+10 ``` longhorndiskusage_bytes Report the used storage of this disk Example of a sample of this metric could be: ``` longhorndiskusage_bytes{disk="default-disk-8b28ee3134628183",node="worker-3"} 9.060212736e+09 ``` longhorndiskreservation_bytes Report the reserved storage for other applications and system on this disk Example of a sample of this metric could be: ``` longhorndiskreservation_bytes{disk="default-disk-8b28ee3134628183",node="worker-3"} 2.519618519e+10 ``` longhorninstancemanagercpurequests_millicpu This metric reports the requested CPU resources in Kubernetes of the Longhorn instance managers on the current node. The unit of this metric is milliCPU. See more about the unit at https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units This metric contains 3 labels (dimensions): `node` `instance_manager` `instancemanagertype` Example of a sample of this metric could be: ``` longhorninstancemanagercpurequestsmillicpu{instancemanager="instance-manager-r-61ffe369",instancemanagertype="replica",node="worker-3"} 250 ``` longhorninstancemanagercpuusage_millicpu This metric reports the CPU usage of the Longhorn instance managers on the current node. The unit of this metric is milliCPU. See more about the unit at https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units This metric contains 3 labels (dimensions): `node` `instance_manager` `instancemanagertype` Example of a sample of this metric could be: ``` longhorninstancemanagercpuusagemillicpulonghorninstancemanagermemoryrequestsbytes{instancemanager="instance-manager-r-61ffe369",instancemanager_type="replica",node="worker-3"} 0 ``` longhorninstancemanagermemoryrequests_bytes This metric reports the requested memory in Kubernetes of the Longhorn instance managers on the current node. This metric contains 3 labels (dimensions): `node` `instance_manager` `instancemanagertype` Example of a sample of this metric could be: ``` longhorninstancemanagermemoryrequestsbytes{instancemanager="instance-manager-e-0a67975b",instancemanagertype="engine",node="worker-3"} 0 ``` longhorninstancemanagerusagememory_bytes This metrics reports the memory usage of the Longhorn instance managers on the current node. This metric contains 3 labels (dimensions): `node` `instance_manager` `instancemanagertype` Example of a sample of this metric could be: ``` longhorninstancemanagermemoryusagebytes{instancemanager="instance-manager-e-0a67975b",instancemanagertype="engine",node="worker-3"} 1.374208e+07 ``` longhornmanagercpuusagemillicpu This metric reports the CPU usage of the Longhorn manager on the current node. The unit of this metric is milliCPU. See more about the unit

https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units This metric contains 2 labels (dimensions): `node` `manager` Example of a sample of this metric could be: ``` longhornmanagercpuusagemillicpu{manager="longhorn-manager-x5cjj",node="phan-cluster-23-worker-3"} 15 ``` longhornmanagermemoryusagebytes This metric reports the memory usage of the Longhorn manager on the current node. This metric contains 2 labels (dimensions): `node` `manager` Example of a sample of this metric could be: ``` longhornmanagermemoryusagebytes{manager="longhorn-manager-x5cjj",node="worker-3"} 2.7979776e+07 ``` We add a new end point `/metrics` to exposes all longhorn Prometheus metrics. We follow the , each Longhorn manager reports information about the components it manages. Prometheus can use service discovery mechanism to find all longhorn-manager pods in longhorn-backend service. We create a new collector for each type (volumeCollector, backupCollector, nodeCollector, etc..) and have a common baseCollector. This structure is similar to the controller package: we have volumeController, nodeController, etc.. which have a common baseController. The end result is a structure like a tree: ``` a custom registry <- many custom collectors share the same base collector <- many metrics in each custom collector ``` When a scrape request is made to endpoint `/metric`, a handler gathers data in the Longhorn custom registry, which in turn gathers data in custom collectors, which in turn gathers data in all metrics. Below are how we collect data for each metric: longhornvolumecapacity_bytes We get the information about volumes' capacity by reading volume CRD from datastore. When volume move to a different node, the current longhorn manager stops reporting the vol. The volume will be reported by a new longhorn manager. longhornactualsize_bytes We get the information about volumes' actual size by reading volume CRD from datastore. When volume move to a different node, the current longhorn manager stops reporting the vol. The volume will be reported by a new longhorn manager. longhornvolumestate We get the information about volumes' state by reading volume CRD from datastore. longhornvolumerobustness We get the information about volumes' robustness by reading volume CRD from datastore. longhornnodestatus We get the information about node status by reading node CRD from datastore. Nodes don't move likes volume, so we don't have to decide which longhorn manager reports which node. longhornnodecount_total We get the information about total number node by reading from datastore longhornnodecpucapacitymillicpu We get the information about the maximum allocatable cpu on this node by reading Kubernetes node resource longhornnodecpuusagemillicpu We get the information about the cpu usage on this node from metric client longhornnodememorycapacitybytes We get the information about the maximum allocatable memory on this node by reading Kubernetes node resource longhornnodememoryusagebytes We get the information about the memory usage on this node from metric client longhornnodestoragecapacitybytes We get the information by reading node CRD from datastore longhornnodestorageusagebytes We get the information by reading node CRD from datastore longhornnodestoragereservationbytes We get the information by reading node CRD from datastore longhorndiskcapacity_bytes We get the information by reading node CRD from datastore longhorndiskusage_bytes We get the information by reading node CRD from datastore longhorndiskreservation_bytes We get the information by reading node CRD from datastore longhorninstancemanagercpurequests_millicpu We get the information by reading instance manager Pod objects from datastore. longhorninstancemanagercpuusage_millicpu We get the information by using kubernetes metric client. longhorninstancemanagermemoryusage_bytes We get the information by using kubernetes metric client. longhorninstancemanagermemoryrequests_bytes We get the information by reading instance manager Pod objects from datastore. longhornmanagercpuusagemillicpu We get the information by using kubernetes metric client. longhornmanagermemoryusagebytes We get the information by using kubernetes metric client. The manual test plan is detailed at This enhancement doesn't require any upgrade.