sensu/sensu-community-plugins — pull request #368 by johanek (2013-11-27 21:25:44+00:00)
Title: If we can't get data from graphite, return UNKNOWN rather than CRITICAL

The graphite check-data.rb plugin will raise a CRITICAL error if it can't contact the graphite server, or if the required data can't be obtained somehow.

I don't think this is appropriate - when I define a check based on graphite data, it's for a specific problem we want to know about. The only case where I want the check to raise critical is when value falls outside what's defined in the check. If graphite isn't working, or the data collection isn't working, that's a different issue entirely.

We run a monitoring pipeline where we try and collect data, store it in graphite, and then alert off the data stored in graphite. So we use this check heavily. This means that if graphite is down, then we get lots of CRITICAL failing checks.

Instead I'd rather see the check reporting there's a problem, and we can define specific checks to make sure graphite or the collection agents are working.