Project: datamanagement
Issue DM-3125: basic monitoring of jenkins nodes with notification
This last weekend, the build slaves el6-2 and el7-2 ran out of disk space and were causing stack-os-matrix build failures. We should have an active monitoring system that sends notifications via at least one of hipchat/email/pagerduty. There is disk utilitization information present in jenkins itself, aws cloudwatch, and ganglia (as of v0.2.x of the demo). However, it may make be more convenient (read: expedient) to use a dedicated monitoring system such as sensu instead of mining existing data sources. We should also investigate if we can configure jenkins to not schedule jobs on a slaves with low disk space.
Story points: 6