allending/Kiwi — pull request #449 by modocache (2014-02-13 17:26:20+00:00)
Title: Output formatters

**Do not merge this yet. I'm issuing this pull request as an invitation for feedback.**

One of the goals for issue #306 is to allow test custom test output. This pull request implements this feature in [a similar way to how RSpec does it](http://modocache.svbtle.com/code-reading-rspec-output-formatting).

## Feature Summary

Users may add a `KiwiConfiguration.plist` file to their test bundle. This plist may contain a `formatters` key, which refers to a dictionary of key-value pairs in the form `formatter_name: output_file_path`. For example:

```xml

formatters

KWJSONFormatter
/Users/modocache/Desktop/test-output.json

KWTextFormatter
stderr

```

All formatter classes specified in the plist are registered to receive notifications regarding test execution via a new `KWReporter` singleton class.

For example, specifying `KWJSONFormatter: /Users/modocache/Desktop/test-output.json` writes the following content to a file named `test-output.json` on the Desktop:

```json
{
"summary" : {
"failure_count" : 2,
"duration" : 0,
"summary_line" : "5 examples, 2 failures, 1 pending",
"example_count" : 5,
"pending_count" : 1
},
"examples" : [
{
"result" : "passed",
"description" : "something, does something"
},
{
"result" : "passed",
"description" : "something, does another thing"
},
{
"result" : "failed",
"failure_message" : "expected subject to equal (NSString) \"goodbye\", got (NSString) \"hello\"",
"description" : "something, doesn't know how to say goodbye"
},
{
"failure_message" : "NSInternalInconsistencyException \"oh wow such exception so halting\" raised",
"exception" : {
"call_stack_symbols" : [
"0 CoreFoundation 0x017505e4 __exceptionPreprocess + 180",
"1 libobjc.A.dylib 0x014bf8b6 objc_exception_throw + 44",
"2 CoreFoundation 0x017503bb +[NSException raise:format:] + 139",
"3 KWTestAppTests 0x08c822a4 __37+[KiwiTestAppSpec buildExampleGroups]_block_invoke50 + 68",
"4 KWTestAppTests 0x08c8f1e2 __25-[KWExample visitItNode:]_block_invoke + 114",
// ...
],
"name" : "NSInternalInconsistencyException",
"reason" : "oh wow such exception so halting"
},
"result" : "failed",
"description" : "something, raises an exception"
},
{
"result" : "pending",
"description" : "something, will soon do something else(PENDING)"
}
],
"messages" : [
"This is an arbitrary message test writers may send to the reporter."
]
}
```

## Implementation Summary

Here are the key players:

- `KWReporter`: A singleton which maintains an array of listeners. `KWExample` informs the reporter whenever it passes, fails, etc. The reporter then informs each of its listeners of these events.
- `KWListener`: A protocol defining how the reporter notifies listeners.
- `KWFormatterLoader`: A class responsible for loading the formatters from the plist. If the plist does not exist, `KWTextFormatter: stderr` is used.
- `KWTextFormatter`: A formatter class which outputs the exact same text as the `NSLog` statements in Kiwi 2.0 (minus the timestamps).
- `KWJSONFormatter`: A formatter class which outputs JSON.

The rest is in the diff. Sorry it's so huge. :frowning:

## Specific Requests for Feedback

#### The API for Specifying Formatters

I don't like plists, but I chose to use one because I couldn't figure out a way to get access to arguments passed on the command line to `xcodebuild`. It would be much nicer to users if they could execute the following:

```
xcodebuild -project MyProject.xcodeproj -scheme MyProject -sdk iphonesimulator test \
-formatters="KWTextFormatter:stderr KWJSONFormatter:/Users/modocache/Desktop/test-output.json"
```

Any thoughts on how to do this?

#### Printing to stderr

As far as I know, there is no way to "silence" XCTest output. Therefore, in order to ensure end users can actually parse the Kiwi formatter output, I've decided to redirect their output to a file. I'm not sure if this is the best way to do things. Ideally, I'd like to "silence" XCTest output and instead print the formatter output to stderr.