CB-7459 Allow automatic tests to be run for specific plugin(s)

Implementation for https://issues.apache.org/jira/browse/CB-7459

See comments in commits for more details.
May we rename this to `setupCheckboxes` or something like that?