trackabout/speceasy — pull request #24 by jsternal (2014-06-20 21:07:10+00:00)
Title: Adding test case showing when SpecEasy incorrectly reports an assertion ...

**TL;DR**
SpecEasy results sometimes hide exceptions under bogus assertion errors. To fix this, I'm recommending that we replace `AssertWasThrown` with a new mechanism for asserting exceptions. (Not included in this PR.)

**The Problem**
When the method under test throws an exception, SpecEasy sometimes reports the result incorrectly: it reports an assertion failure instead of the exception.

Consider the following SUT and spec:

public class BrokenClass {
public bool Invert(bool value) {
throw new Exception("exception while trying to invert value");
return !value;
}
}

class BrokenClassSpec : Spec {
private bool input;
private bool result;

public void RunSpec() {
When("inverting the value", () => result = SUT.Invert(input));

Given("a true input", () => input = true).Verify(() =>
Then("the result is false", () => result.ShouldBeFalse()));

Given("a false input", () => input = false).Verify(() =>
Then("the result is true", () => result.ShouldBeTrue()));
}
}

The first spec correctly shows the exception. There is no assertion failure: SpecEasy evaluates the spec, but the assertion ‘passes’ (since false is the result variable’s default value).

The second spec incorrectly reports “Failed: Should.Core.Exceptions.TrueException : Assert.True() Failure”. SpecEasy always executes its specs, even if the SUT throws an exception during the When. (It appears to be doing this in order to support `AssertWasThrown`.) It seems likely that the assertion failure overwrites or somehow takes priority over the exception that truly causes the failure.

This does not appear to be unique to the Resharper test runner: I ran the test case from my pull request using the native Nunit test running and got identical results.

**Possible Solutions**
* Add a new API, something along the lines of `ThenItThrows(string, Action)` Store a flag indicating whether or not the spec in question asserts that an exception was thrown. If the "When" throws, only execute those specs flagged as asserting exceptions. This is a **breaking change**, and would render `AssertWasThrown` unusable.
* Using a library like [Cecil](https://github.com/jbevain/cecil), modify the SpecEasy `Then` methods to inspect the Action (and Func). If it includes an `AssertWasThrown` call, store a flag as outlined above.

I think the first option is the only realistic one. First, Cecil is not yet mature, and there don't seem to be any alternatives. Second, it wouldn't be airtight: SpecEasy would *only* handle exception assertions using our API.

**The Test Demonstrating the Problem**
* In order to verify that the NUnit TestResults were correctly reporting exceptions, I used the following code: `ft.Message.StartsWith("System.Exception")`. I don’t like looking for strings, but there doesn't seem to be any other way of differentiating results that report assertion failures from results that report exceptions.
* If you debug and look at the test results in `ExceptionReportingSpecs.RunSpecsForBrokenClass`, you'll see that we get two test results per spec. This is not true for the NUnit equivalent.
It looks like SpecEasy is building them twice, possibly due to the way the test harness is set up. It does not appear to influence the outcome