w3c/ldp-testsuite — pull request #36 by fserena (2014-06-17 10:30:36+00:00)
Title: Allow testing with a custom subset of tests

The implementation of the test suite consists of launching all existing and enabled tests. There is not a way to programmatically choose a custom subset of them you want to test. It is true that they are splitted into groups (MUST, SHOULD and MAY mainly), but what if you only want to test a subset of the MUSTs?

It is highly desirable to provide a simple way for users to build their own dynamic/virtual groups programmatically (at least). This can also help in integrating the test suite with integration testing.

This feature implements an IAnnotationTransformer (TestNG) which changes the enable state of each test method just before they are invoked. This enable state switch will depend on a customized list of test method names. There should be two ways to operate with this list. If we only want to enlist the methods to be tested, the default value for enable annotations will be false. The opposite shall work the same way.

We have implemented a MethodEnabler (IAnnotationTransformer) that behaves as described above. It is a non-intrusive solution because Transformers are binded to the TestSuite as listeners. We only had to modify the LdpTestSuite class to make it capable of accepting a list of listeners in its options map input.

This pull request uses the feature in https://github.com/w3c/ldp-testsuite/pull/35.