CB-7459 Allow automatic tests to be run for specific plugin(s)

Implementation for https://issues.apache.org/jira/browse/CB-7459

See comments in commits for more details.
Also, its typical to put the `callback` argument last.