Suite runner extensibility

These changes would allow much easier customization, to resolve #486