Create a test runner which tries out different configurations.