Fix running test files directly and add test coverage

I broke this in 0.5.3.  

This adds tests to cover expected behaviour.