Modified our spec tests so they set the same flag as the command runner,
causing invalid directory exceptions to be seen when running tests. This
commit provides the test coverage for #4d562ba.