alphagov/smokey — pull request #21 by samjsharpe (2013-01-26 23:46:17+00:00)
Title: External json parser

This commit does not change current behaviour, it simply adds something that will not currently be used.

* `cron_json.sh /tmp/smokey.json` will run cucumber for all the tests needed in skyscape.
* It will generate a JSON file describing the output of the tests
* It runs Ruby, Bundler and Cucumber exactly once for all the tests
* `nagios_check_cache.py calendars normal /tmp/smokey.json` is designed to run as a Nagios check.
* It parses the output of the json file and emits the correct nagios messages and exit codes.
* Additionally it outputs to log/calendars_normal.log a full log of what it found in the JSON cache file, so that we can determine *which* tests failed at any particular time.
```
2013-01-26 23:34:02: Feature: features/calendars.feature
2013-01-26 23:34:02: Scenario: check calendars loads
2013-01-26 23:34:02: Step: the "calendars" application has booted - failed
2013-01-26 23:34:02: Step: I am testing through the full stack - skipped
2013-01-26 23:34:02: Step: I force a varnish cache miss - skipped
2013-01-26 23:34:02: Step: I should be able to visit: - skipped
2013-01-26 23:34:02: Scenario: check alternative formats are available
2013-01-26 23:34:02: Step: the "calendars" application has booted - failed
2013-01-26 23:34:02: Step: I am testing through the full stack - skipped
2013-01-26 23:34:02: Step: I should be able to visit: - skipped
```

Once I have this merged and deployed, I will evaluate how long it takes to run on the Nagios server and from that determine how frequently we can run it ;)