Record a set of features (suites) that failed so they can be reported to the test runner
No need for passing as every test is guaranteed to have at least one test in its suite
Add mkdirp
Write out test suite failures and results to file