square/zapp — Issue #27: Tests reporting success when they've actually failed.
Author: lawrence-forooghian
Created: 2011-11-24 14:27:49+00:00
State: open

One of our KIF scenarios, somewhere in the middle of our test suite, is failing. Zapp then seems to be noticing this, and decides to run the tests again, starting from the next scenario after the failing one. This remaining set of scenarios all passes, and it seems like Zapp then hence decides that this means the whole build was a success. This doesn't seem right :-)

I'm not quite sure why Zapp is doing this business of stopping and continuing the tests if one fails, but I'm sure there's a good reason for it. However, I think that the reason this bug is happening lies in line 385 of ZappBuild.m:

```objc
[self runSimulatorWithAppPath:appPath initialSkip:lastStartedScenario - 1 failureCount:newFailureCount - 1 startsWithRetry:YES];
```
I've got a feeling that you shouldn't be subtracting one from newFailureCount? In our example, this means that failureCount in the recursive call is 0, when I believe it should be 1.