DynamoDS/Dynamo — pull request #2947 by Benglin (2014-11-03 09:46:44+00:00)
Title: Fixed flakiness in recorded tests

Background
-----
```AutomationSettings``` is a class that runs the recorded tests, it contains a ```DispatcherTimer playbackTimer``` that picks up commands one by one from its internal ```loadedCommands``` list. When these commands are exhausted, ```AutomationSettings``` turns this ```playbackTimer``` into a ```shutdownTimer```, counting down for a predefined duration (defined as ```PauseAfterPlaybackInMs``` attribute in the test XML) before shutting down ```DynamoView``` for control to return to the test case before terminating the test run.

Some recorded tests have ```DynamoModel.DynamicRunEnabled``` set to ```true```. For these test cases there is another ```DispatcherTimer``` at play -- ```HomeWorkspaceModel.runExpressionTimer```. The ```runExpressionTimer``` gets reset to tick in ```100ms``` every time the ```HomeWorkspaceModel``` is modified, including the final user action in the recorded test. These test cases are said to run the tests in *Run Automatically* mode.

The following code example shows an example of such test with ```autoRun``` set to ```true```:

```cs
public void Defect_MAGN_747_MultiReference()
{
var fileName = "Defect_MAGN_747_MultiReference.xml";
RunCommandsFromFile(fileName, autoRun: true);

// When all the commands exhausted, it gets here for validation.
AssertPreviewValue("a76409a1-1280-428c-9cf7-16580c48ff96", 1);
}
```

When dynamic run is enabled, both ```shutdownTimer``` and ```runExpressionTimer``` will be active at the very end of ```RunCommandsFromFile``` method. If ```shutdownTimer``` ticks before ```runExpressionTimer```, it shuts down ```DynamoView``` and get to ```AssertPreviewValue``` without ```runExpressionTimer``` running for the final time. This naturally results in ```AssertPreviewValue``` failing, since the evaluation has not taken place.

Changes
-----
This pull request addresses this by enhancing ```AutomationSettings``` to check if there is still any outstanding evaluation in ```HomeWorkspaceModel``` when its ```shutdownTimer``` ticks. If the ```runExpressionTimer``` has yet to tick, it then returns immediately, waiting for the ```shutdownTimer``` to tick next.

Test scenario
-----
This fix is verified by setting ```runExpressionTimer``` to tick after ```10,000ms```, and the tests still pass as expected (these tests naturally failed consistently before the fix when I increased the ```runExpressionTimer``` to ```10,000ms```).

Next steps
-----
After this is merged in, I will restore ```PauseAfterPlaybackInMs``` in recorded test cases to their default values of ```20ms```, and all these tests should still retain their results predictably.

Hi @lukechurch, please have a go with it, thanks!