Runner improvements

This supercedes #1192 and takes it a bit further.  I also did some additional refactoring in related areas.

/cc @benhoskings @JonRowe @xaviershay @cupakromer