We could shift the work to the test runner, or reuse rimraf from the npm distribution