nunit/nunit — pull request #263 by CharliePoole (2014-10-07 00:06:22+00:00)
Title: Issue 255

Do not merge!

I'm creating this PR so we can discuss options for handling various kinds of failures that can occur in a suite. There are three places failures can occur: in OneTimeSetUp (including Before actions), in a child test or in OneTimeTearDown (including After actions).

In this commit, I have created separate failure states for each of those alternatives: SetUpFailure, ChildFailure and TearDownFailure. I made corresponding changes to the reporting so that it continues to work correctly and was able to do away with checking for a specific message text to recognize that a failure was due to a problem in a child test.

However, I'm not entirely happy. Other things can also happen in these three places: unexpected exceptions, ignoring the test failure of an Assume. So do we take every outcome and multiply it by three?
Do we need SetUpError, SetUpInconclusive, etc?

In NUnit 2.x, we had the FailureSite enum and the FailureSite property of TestResult. The enum had values of SetUp, Test, TearDown, Child and Parent. Instead of proliferating result states, we could add a new Site property to ResultState to indicate where any non-Success result originated. Assert.Ignore in setup would give a ResultState.Ignored, with the Site set to SetUp. Would this lead to clearer code than the present approach of adding entirely new ResultStates?