carrotsearch/randomizedtesting — Issue #17: Add arbitrary execution groups (not only @Nightly).
Author: dweiss
Created: 2011-11-04 18:43:11+00:00
State: open

@Nightly is cool, but it's limited. We could add arbitrary test groups (applicable to types and test methods). TestNG has this too, as in:

@Test(groups = { "fast" })

Having separate annotations would be nice in that you then can search for which tests/ methods are annotated with a given annotation using an IDE. Annotations don't inherit though, so I don't see a clear way of doing that. For now, I'd suggest doing:

@TestGroups({@GroupName1, @GroupName1, @GroupName1})

This would allow arbitrary annotations to be used as groups. There could also be an interface an annotation could implement for extending the functionality of a group (for example, indicate a group's tests can be executed in parallel?).

Execution of test groups could be controlled via system properties -Dtests.group.GroupName1=[boolean].

@Nightly would be an exception from the above rule in that it could be applied directly, but would follow all other rules for groups too (i.e., could be disabled by -Dtests.group.Nightly=false).