Perl-Toolchain-Gang/Test-Harness — pull request #5 by markstos (2012-08-20 11:05:16+00:00)
Title: Documentation for the 'rules' system and the 'Scheduler'

This is a follow-up to the pull request to my [previous pull request](https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/3), in which I created a patch to add sequence-only exceptions for parallel test runs, only to discover that there was an undocumented option to prove called '--rules' which already has this feature.
--rules is handled by TAP::Parser::Scheduler, and much of the parts of the code for --rules and the Scheduler were not documented since they were introduced in 2008.

Despite being difficult to discover by most people, the rules system has been well exercised over time, as it is used by [t/harness in the Perl distribution](http://perl5.git.perl.org/perl.git/blob/HEAD:/t/harness), which uses a more advanced syntax for it.

The ability to mark some tests as a not parallel-ready will be a very welcome feature for users with large test suites. Indeed, it turns out it was key to speeding up the Perl testing process. With this feature available, it can be a fairly simple process to convert a complex test suite to parallel testing:

0. Start with tests 100% passing as a baseline (or at least inventory those which are failing)
1. Try a fully parallel test run (or 5 or 10)
2. Note which tests fail when run in parallel
3. Add these exceptions to your .proverc using the --rules syntax.

Profit!

There are a couple of discussion points going forward:

1. In the little documentation that was pre-existing, the rule system was marked as "experimental" in 2008. Is it still experimental because it's been largely undocumented and undiscovered, or is it considered stable now before because it's 4 years old? I would inclined to continue to mark it as experimental until more people can discover it and provide feedback. I left it marked as such in the documentation.

2. Now that I figured out how to use it via prove, I think the syntax for `prove` could use particular scrutiny to see if that's what we want to keep, or if we want to simply it further. Here's my concern. The primary use-case I see for having "rules" options in prove is to make some tests as not-parallel-ready. Yet, here's the cumbersome, non-memorable recipe for that:

# Run all tests in parallel, except those starting with "p"
--rules='seq=t/p*.t' --rules='par=**'

That's a pretty gross syntax for what could be a common usage. On one hand, people may copy/paste this syntax into their .proverc file and forget. On the other, I think we can do better. My initial inclination is to leave the "--rules" flag exposed for advanced uses, but support a simpler shorthand, for example, this could mean the same as the above:

# Tests starting with "p" are not parallel-ready, they must be run in sequence.
--not-parallel='t/p*.t'

There would be some details to work out there, like how the "simple" and advanced syntaxes would interact, but it seems worth considering.