Unify test harness

This is basic support for what I introduced in #475. 
This is far from finished, but I think this should considerably improve the CI output.