cbeust/testng — Issue #156: Method that is failed because of missing parameters is passed to listener as a skipped one
Author: gaurnar
Created: 2012-01-13 07:07:25+00:00
State: open

I encountered a rather vicious problem with passing test results to listener. If a method fails because of missing parameters, it is passed to registered listener as a skipped one (i.e. onTestSkipped method is called and test result object has skipped status). I can see that it is actually failed because it is stated so in summary box that is written to STDOUT:

```
===============================================
test
Total tests run: 1, Failures: 1, Skips: 0
===============================================
```

and also because it is marked as failed in test-output/emailable-report.html. There is also relevant exception message displayed:

```
Parameter 'param' is required by @Test on method foo but has not been marked @Optional or defined in ...
```

I verified this problem in latest released version - 6.3.1.

Here is a simple test to reproduce this issue:

```java
package org.gaurnar;

import org.testng.annotations.BeforeClass;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class FooTest
{
@BeforeClass( description = "prep" )
public void preparation()
{

}

@Test( description = "foo" )
@Parameters( "param" )
public void foo( String param )
{

}
}
```

and a simple runner to launch the test and set listener:

```java
package org.gaurnar;

import java.util.LinkedList;

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.testng.TestNG;

public class Runner
{
public static class Listener implements ITestListener
{
public void onTestStart( ITestResult result )
{
System.out.print( "test start\n" );
}

public void onTestSuccess( ITestResult result )
{
System.out.print( "test success\n" );
}

public void onTestFailure( ITestResult result )
{
System.out.print( "test failure\n" );
}

public void onTestSkipped( ITestResult result )
{
System.out.print( "test skipped\n" );
}

public void onTestFailedButWithinSuccessPercentage( ITestResult result )
{
}

public void onStart( ITestContext context )
{
System.out.print( "class start\n" );

}

public void onFinish( ITestContext context )
{
System.out.print( "class finish\n" );
}
}

public static void main( String[] args ) throws Exception
{
TestNG testNG = new TestNG();
testNG.setListenerClasses( new LinkedList() {{ add( Listener.class ); }} );
testNG.setTestClasses( new Class[] { FooTest.class } );
testNG.run();
}
}
```