Fix rerunning non-rspec tests

Fixes #1