Project: OpenEBS (Runtime / Cloud Native Storage)
File: 20190905-ndm-exporter-integration.md

oep-number: NDM 0002 title: Metrics Collection using Node-Disk-Manager authors: "@akhilerm" owners: "@kmova" "@vishnuitta" editor: "@akhilerm" creation-date: 2019-09-05 last-updated: 2020-03-26 status: provisional * * * * * This proposal brings out the design details to implement a metrics collector for block devices. The metrics will include static data like NDM assigned UUID, Device state along with continuously varying data like used capacity, temperature data etc. NDM currently stores all the data related to the block devices in etcd in a custom resource. This is not a good approach for continuously varying metrics. Also the metrics should be available on demand at an end point from where users can query and get relevant information. This will help in monitoring of the block devices rather at the pool or volume level. A cluster level exporter that exposes static and rarely changing metrics of block devices like device state and UUIDs A node level exporter that exposes continuously varying metrics like temperature and free space on the block device The exporter will be working only at the block level and will not provide metrics about pool or volumes that are built on top of these devices. For a high level view of storage on the cluster, data points like when the disk went offline, total capacity of the disk etc should be available to the cluster admin. For deep monitoring of the storage devices, metrics like IOPS, drive temperature etc are required and these data should be queried from each block device from each node. The implementation details of exporter for collecting block level metrics. The current implementation just stores all the data related to the block devices in etcd, which is difficult to retrieve by monitoring systems like prometheus. In the current implementation all the details related to the block devices are stored in etcd. Even information like temperature is stored in the custom resource. The data in etcd is updated only when a udev event happens on the system. Since the time at which udev events occur cannot be predicted and is not so frequent the metric data stored in etcd will be obsolete most of the time. Also, this approach cannot be used in cases of continuously varying metrics like temperature and free space on the drive. There will be 2 components to the exporter. One will be running at the cluster level and the other at node level. The components can be customized depending on the metrics you need to fetch from the cluster. This component of the exporter will run at the cluster level and there will be only one running instance of this component at any time in the cluster. The primary responsibility of this exporter is to take static / rarely changing data about the block devices from etcd and expose it as prometheus

The static data like Model number, UUID etc will be cached in the exporter, and metrics like state(Offline/Online) of the device will be fetched from etcd every time a request comes at the end point for the metrics. The node level exporter runs as a daemon in each node and collects all the metrics of the block device like IOPS, temperature, free space etc. It exposes a rest endpoint which can be queried to get the data. Every time the exporter is queried, all the metrics related are fetched from the disk using SMART and Seachest libraries and returned. ``` ++ ++ 2 ++ | | 1 | +>+ | | +>+ cluster level | | etcd | | Prometheus | | exporter +<+ | | | | | 3 ++ | +<+ | | | 4 | | ++--++-+--+ ++ | ^ | ^ | | | +--8-+ | | | | | | +--5-+ | 5 | | | | 8 +-+ +--v-+--+ | | | | | | | +--+ | | | | | Node1 | | Node2 | | | | | | +->+ exporter-d | | exporter-d | +--+-+ +--+-+ 6| ^ 6| ^ v |7 v |7 +-+-+ +-+-+ |::Disk | |::Disk | ++ ++ ``` Cluster Level exporter Queries the cluster level operator for static data about all block devices in the cluster Requests etcd for all blockdevices and their properties Response from etcd with all blockdevices Response from cluster level exporter to prometheus with static metrics Sample metrics when cluster exporter endpoint is queried ``` nodeblockdevice_state{blockdevicename="blockdevice-6a0ec0732d1f709810a3fbbde81fc3bb",hostname="minikube",nodename="minikube",path="sda"} 0 nodeerrorrequest_count 2 noderejectrequest_count 0 ``` Node Level exporter Queries each storage node for disk metrics exporter-d queries certain pages on the disk to get relevant metrics data The information from the pages is analysed using SMART and seachest libraries to get relevant metric about the disk The live metrics are send back to prometheus. Sample metrics when node level exporter is queried ``` seachestblockdevicecurrenttemperature_celsius{blockdevicename="blockdevice-6a0ec0732d1f709810a3fbbde81fc3bb",hostname="minikube",nodename="minikube",path="sda"} 38 seachestblockdevicecurrenttemperature_valid{blockdevicename="blockdevice-6a0ec0732d1f709810a3fbbde81fc3bb",hostname="minikube",nodename="minikube",path="sda"} 1 seachesterrorrequest_count 0 seachestrejectrequest_count 0 ``` NDM Exporter can be installed by following the YAML . Make sure that the exporter deployment and daemonset is installed in the same namespace as of OpenEBS-NDM. The NDM exporter exposes metrics such that it can be used along with prometheus node exporter. The `hostname`, `nodename` and `path` labels are made available in all metrics exposed by NDM exporter. This labels help to club together metrics from NDM exporter and node exporter, thus giving a complete details view of the block devices in the system. New collectors for fetching additional metrics can be added by following the steps described . The exporter should be able to expose both static and dynamic metrics of the block devices. The exporter should be also customizable such that depending on the type of metrics required, the cluster or node level exporter should be deployed. Owner acceptance of `Summary` and `Motivation` sections - YYYYMMDD Agreement on `Proposal` section - YYYYMMDD Date implementation started - YYYYMMDD First OpenEBS release where an initial version of this OEP was available - YYYYMMDD Version of OpenEBS where this OEP graduated to general availability - YYYYMMDD If this OEP was retired or superseded - YYYYMMDD NA NA