Get cli tests to run in parallel without clobbering each other results.