alphagov/whitehall — pull request #266 by samjsharpe (2013-02-21 22:24:14+00:00)
Title: Add a publishing:due:check Rake task

NB: Apologies if I've messed up anything in this Code - I'm not a Ruby person and this is my first commit to Whitehall ;)

At the moment, out Whitehall checks for scheduled_publishing consist of checking a graphite counter which is incremented when scheduled publishing is run. That's sub-optimal in many ways:

1. It's actually only checking that Cron works. It doesn't check that the document is actually published.
1. It uses Graphite as an intermediary, which is prone to missing hits - leading to false alerts
1. It doesn't actually test that there are no overdue editions, which is what we actually care about.

This Rake task (publishing:due:check) outputs a message in standard Nagios format showing the number of overdue editions (I hope). It is designed to be checked via Nagios, an appropriate exit code determined and an error raised if there are any overdue publications.

My plan is to ask for this to be merged in, wait for it to be deployed and then write a quick wrapper for running this via Nagios Remote Plugin Executor and then write a check on the Monitoring machine which randomly picks a server running Whitehall Admin, connects, runs this check and barfs in a spectacular fashion if there are any overdue publication.

Does that sound sensible?