apache/maven-surefire — pull request #47 by jontodd (2014-09-03 01:14:02+00:00)
Title: Support for TestNG successPercentage

TestNG has the concept of "successPercentage" where a probabalistic test can be run some number of innvoations and then if the successPercentage is met, the test is considered passing.

Example code might look like:

```java
@Test(invocationCount = 100, successPercentage = 25)
void testRandom() {
// ~50% chance of generating 0 & 1
int num = new Random().nextInt(2);
Assert.assertEquals(num, 1);
}
```

As detailed in SUREFIRE-654, the problem is that surefire is currently mapping each invocation to a new test run, and any failure, regardless of whether we're within the allowed success percentage or not will result in this test failing. This renders any other value for successPercentage aside from 100 useless.

Digging into the code it looks like this was just an oversight in the inital implementation of the TestNGReporter. The TestNG ITestListener#onTestFailedButWithinSuccessPercentage method API stipulates that it will only be called when we're still within the passing limit:

Invoked each time a method fails but has been annotated with successPercentage and this failure still keeps it within the success percentage requested.

This change maps tests which pass within the successPercentage as passing which is safe because if we go beyond the passing threshold the test will be marked failed as desired.

__What about tests?__
Yes, there are no test changes. It looks like the providers aren't being tested. I'd be happy to add coverage that asserts the correct stubbed reporter method is called but that seems like overkill given the existing testing conventions.

__What about breaking existing users?__
As mentioned earlier, essentially successPercentage wasn't even usable within surefire before, so this should only provide net-new functionality to users. In the example above, before my change that test would fail every time and would result in ~50 failures. After this change the above example would pass so long as there are only 25 or fewer failures.