xp-framework/rfc — Issue #188: Test outcome
Author: thekid
Created: 2011-10-17 16:04:37+00:00
State: closed
Labels: framework, implemented

Scope of Change
====
At the moment, there are three possible outcomes of a unittest: Succeeded,
Skipped and Failed. This RFC suggests adding more kinds of outcomes:
Warnings, Error and NotRun.

Rationale
====
Be able to distinguish between the different non-success reasons.

Functionality
====
The new outcomes are represented by these new classes:

```php

```

The existing classes TestSuccess, TestFailure and TestSkipped are turned
into interfaces and the concrete implementations moved to the following
new classes:

```php

```

Definitions
----

These current outcomes are defined:
* "." - Succeeded (Test was successful)
* "S" - Skipped (Test was skipped by either an error inside setUp() or by
an @ignore annotation)
* "F" - Failed (Test failed due to an assertion error, an expected exception
not being caught or by an exception raised by the code inside).

Example output:

```
[..SSFFF]
2 succeeded, 2 skipped, 3 failed

```

This RFC suggests to add the following new outcomes:
* "W" - Warning (One or more PHP errors were raised)
* "E" - Error (An exception other than AssertionFailedError was thrown when
executing the test and no @expected annotation existed)
* "N" - Not run (Test was ignored by @ignore, as stated above, this is currently
also reported as "skipped")

Example output:

```
[..SNFFE]
2 succeeded, 2 skipped, 3 failed

```

Security considerations
====
n/a

Speed impact
====
n/a

Dependencies
====
No BC break: All instanceof checks on TestFailure, TestSuccess and
TestSkipped will continue to work as expected due to the classes being
converted to interfaces.

Related documents
====
* http://msdn.microsoft.com/en-us/library/ms243180(VS.100).aspx
* http://www.phpunit.de/manual/current/en/textui.html
* http://www.nunit.org/index.php?p=nunit-console&r=2.5.2

Implementing patch:
* http://xp-framework.net/rfc/contrib/rfc0188.diff