SMART

As part of we are now exporting SMART attributes from disks installed on all physical hosts.
= Overview =
We want to track SMART attributes values over time, both for operational purposes (a disk is reporting errors) and auditing purposes (e.g. disks usually are within these temperatures). To this end, from each host we export attributes known to smartmontools as Prometheus metrics. The metrics are then collected as part of the regular monitoring pipeline and available for alerting/graphing/querying. The grunt work is done by a custom python script (smart-data-dump) that periodically enumerates physical disks (including ones behind raid controllers) and parses smartctl output for those disks, writing the resulting metrics into a plaintext file for node_exporter to pick up and present via HTTP.
= Metrics =
Each attribute is exporter as a metric with the attribute name (as reported by smartctl as the metric name, prefixed with device_smart_. The tags attached to all metrics are at least instance and device, the former is used to uniquely identify the disk: sdX in case of directly-attached disks or DRIVERNAME,DISK_ID in case of disks behind raid controllers. In case of raid controllers, the device tag is what is passed as the argument to smartctl -d.
For example:
# smart-data-dump | grep -v ^# | grep sda
device_smart_udma_crc_error_count{device="sda"} 0.0
device_smart_raw_read_error_rate{device="sda"} 205493718.0
device_smart_media_wearout_indicator{device="sda"} 365324.0
device_smart_healthy{device="sda"} 1.0
device_smart_used_rsvd_blk_cnt_tot{device="sda"} 0.0
device_smart_offline_uncorrectable{device="sda"} 0.0
device_smart_power_cycle_count{device="sda"} 24.0
device_smart_power_on_hours{device="sda"} 14065.0
device_smart_read_soft_error_rate{device="sda"} 124759545302.0
device_smart_reallocated_sector_ct{device="sda"} 0.0
device_smart_end_to_end_error{device="sda"} 0.0
device_smart_info{device="sda",firmware="G201DL2B",model="INTEL SSDSC2BX200G4R"} 1.0
device_smart_erase_fail_count_total{device="sda"} 0.0
device_smart_hardware_ecc_recovered{device="sda"} 0.0
device_smart_unused_rsvd_blk_cnt_tot{device="sda"} 5455.0
device_smart_program_fail_cnt_total{device="sda"} 0.0
device_smart_temperature_celsius{device="sda"} 27.0
= Operations =
The smart-data-dump script can be invoked from the command line to see the resulting metrics on stdout and debug eventual problems. Since the script is normally invoked by cron, most of the non-metrics output will be normally logged to syslog for easier auditing and to avoid cronspam.
Some attributes are purposefully not reported as metrics because are not useful/interesting, invoking with --debug will show what attributes are being ignored and which are found to be output by smartmontools, but ignored.
To run a full audit of unknown attributes it is sufficient to run smart-data-dump in debug and syslog mode, across the fleet:
 cumin -s20 -b50 '*' 'if [ -x /usr/local/sbin/smart-data-dump ]; then /usr/local/sbin/smart-data-dump --debug --syslog >/dev/null; fi'
= Alerts =
== SMART not healthy ==
This alert checks whether disks are healthy according to smartmontools. The alert contains the device(s) affected, these can be passed to smartctl to get the full SMART status, e.g for an alert on db1064 such as cluster=mysql device={megaraid,2,megaraid,6} instance=db1064:9100 job=node site=eqiad
 root@db1068:~# smart-data-dump --debug >/dev/null
 ...
 DEBUG:__main__:Running: /usr/bin/timeout 60 /usr/sbin/smartctl --info --health -d megaraid,2 /dev/bus/0
 DEBUG:__main__:Running: /usr/bin/timeout 60 /usr/sbin/smartctl --attributes -d megaraid,2 /dev/bus/0
 root@db1068:~# /usr/sbin/smartctl --info --health -d megaraid,2 /dev/bus/0
 smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.0-1-amd64] (local build)
 Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
 
 === START OF INFORMATION SECTION ===
 Vendor: SEAGATE
 Product: ST3600057SS
 Revision: 0008
 User Capacity: 600,127,266,816 bytes [600 GB]
 Logical block size: 512 bytes
 Rotation Rate: 15000 rpm
 Form Factor: 3.5 inches
 Logical Unit id: 0x5000c50076ad15a3
 Serial number: 6SL8SR0K0000N45004XL
 Device type: disk
 Transport protocol: SAS (SPL-3)
 Local Time is: Tue May 15 11:11:18 2018 UTC
 SMART support is: Available - device has SMART capability.
 SMART support is: Enabled
 Temperature Warning: Enabled
 
 === START OF READ SMART DATA SECTION ===
 SMART Health Status: OK
 
 root@db1068:~# /usr/sbin/smartctl --attributes -d megaraid,2 /dev/bus/0
 smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.0-1-amd64] (local build)
 Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
 
 === START OF READ SMART DATA SECTION ===
 Current Drive Temperature: 43 C
 Drive Trip Temperature: 68 C
 
 Elements in grown defect list: 0
 
 Vendor (Seagate) cache information
 Blocks sent to initiator = 443295929
 Blocks received from initiator = 2624070826
 Blocks read from cache and sent to initiator = 784713905
 Number of read and write commands whose size <= segment size = 130821891
 Number of read and write commands whose size > segment size = 0
 
 Vendor (Seagate/Hitachi) factory information
 number of hours powered up = 7375.38
 number of minutes until next internal SMART test = 45
= TODO =
* We are exporting the raw attributes now, consider exporting normalized values too