DynamoDS/Dynamo — pull request #2647 by Benglin (2014-10-09 05:33:34+00:00)
Title: Enable parallel test execution

Background
-----
We cannot run unit tests on our own machine prior to submission because of issues like memory leaks, the tests ended up consuming all memory available on any machine. For other cases, we have UI elements left over from a previous test case (which owned them) and accessing such UI elements in current test case causes *sharing violation* which fail all subsequent tests (think ```Helix 3D``` related exceptions).

Wouldn't it be ideal, if we could run each of the tests in its own process? I have checked on both ```process``` and ```domain``` command flags for ```nunit-console.exe```, the [highest level of isolation](http://www.nunit.org/index.php?p=consoleCommandLine&r=2.5.5) is done on a per-assembly basis.

So I wrote [a tool called Isolation](https://github.com/Benglin/Isolation) last week to do just that. The idea behind this is very simple, each test case with attribute ```[Test]``` can be obtained through reflection, and then each of these can be specified as command input to ```nunit-console.exe``` to run them in complete isolation. This way none of the test cases can interfere with one another. What's more, ```IsolationConsole.exe``` spawns up to 8 (or as many logical processors as there are) ```nunit-console.exe``` to run the tests in parallel, one test does not need to wait for other to complete (all 1318 tests completed within 40 minutes on a 8 core lab machine).

However, that causes resource sharing issue when more than one concurrent test cases try to deal with the same file on disk. I discovered, by running ```IsolationConsole.exe``` that few of our test cases have unpredictable outcome when they tried to access the same ```TempFolder``` created by the ```[Setup]``` methods. Some concurrent access to files in this ```TempFolder``` causes I/O API calls to throw exceptions, resulting in test errors.

Changes
-----
```UnitTestBase```, ```DynamoTestUIBase``` and ```UpdateManagerUITests``` base classes are updated to produce ```TempFolder``` based on ```System.GUID``` which is guaranteed to be unique. Also, since the ```[TearDown]``` method can safely delete the entire ```TempFolder``` (due to it having a ```GUID``` name), the ```EmptyTempFolder``` helper method is removed -- the temporary folder removal is now much cleaner.

Notes to Reviewers
-----
Hi @lukechurch, this is what I wanted to show you last week but did not. I have get help from intern to run the complete test suite for more than 3 times, the results are consistent. There is no longer unpredictable outcome now. Note also that this will unlikely affect any existing test run, as the temporary folder is transparent to all test cases (they only refer to temporary folder as ```TempFolder``` property).

Please have a look :)