drothlis/stb-tester — pull request #191 by szemate (2014-06-26 17:10:56+00:00)
Title: stbt-batch.d: New hooks for custom test report and summary generators

`report.py` has certain limitations: generating static pages and not
using a database back-end makes it somewhat unsuitable for reporting
large volumes of test results; it doesn't allow sending stb-tester
results to an external reporting system.

To allow `stbt batch` users to overcome the above issues, add two new
user hooks, `batch.report` and `batch.summary` that may replace some or
all of `report.py` functionality with user programs.

The idea to replace `report.py` is the result of the following
considerations:
- Forking and building stb-tester with a customised `stbt batch` would
involve the maintenance burden of continuously backporting changes
from upstream. We'd prefer to use official releases.
- Implementing a custom test runner instead of using `stbt batch` is
certainly viable, but replacing just the report generator saves the
hassle of reimplementing a very similar test runner.
- Replacing `stbt batch report` would be ambiguous, because this is the
script that and invokes the `classify` hook. Also, it generates
`failure-reason`, which is actually very useful.

`report.py` has two different functionalities:
- `report.py ` generates a test report of a single test
run,
- `report.py index.html` generates a summary report of all test runs.
The former may be replaced with `batch.report` and the latter with
`batch.summary` user programs, using single-word hook names for
simplicity. Specifying `batch.report` or `batch.summary` with no
value skips test report or summary generation respectively.

A possible use case is generating dynamic report pages. The `report`
hook fills a database using the contents of files provided by `stbt
batch run`. The `summary` hook is skipped, or it may e.g. send e-mail
notifications if the ratio of failed tests exceeds a threshold.

See the additions to README.rst for further details.