asciidoc/asciidoc — pull request #27 by florentx (2014-05-05 17:14:43+00:00)
Title: "Errors should never pass silently"

> _Errors should never pass silently_ - [The Zen of Python](https://www.python.org/dev/peps/pep-0020/)

This pull request is based on the previous one #25. It improves the `testasciidoc.py` tool to report better the failures and the other warnings.
I recommend to review and merge #25 before considering this one.

When preparing PR #25 I discovered that the test suite is a bit too indulgent:
* if some dependency is missing, errors are printed but test is marked PASSED
* if a filter spawns a subprocess and it crashes, test is marked PASSED
* if the source of a test was deleted or a backend is missing, the test is SKIPPED but the exit code is 0
* if a macro or anything else returns a warning, the warning is silenced, and result is PASSED

This PR addresses the above issues:
* it returns exit code 1 on FAILED tests, but also on SKIPPED tests, and on warnings too
* it prints all the unexpected messages on the screen when running the test suite
* a new directive is created for `testasciidoc.conf` to declare the expected messages: `% messages`
* this new directive is documented
* test is counted as WARNED if some message is unexpected, and the exit code is set to 1

I've chosen to add all the current warnings to the expected `% messages`, so the Travis-CI build is still green after the merge.
However, we should challenge this in the future, and probably fix some tests.