Mutation Testing with PITest and Spock 2
By: Ruben Mondejar
Tags: Spock, Java, Junit, Mutation Testing, Gradle

Gradle Project

First of all, we are going to take advantage here from Gradle and create our basic project from scratch by using SDKMAN!:

$ mkdir pitest-spock-example

$ cd pitest-spock-example

$ sdk install gradle

$ ./gradlew init

For this first contact with mutation testing, we are going to implement an extremely simple calculator package, which contains only two classes: Operations and Numbers.

public class Operations {



public static int add(int num1, int num2) {

return num1 + num2;

}



public static int subtract(int num1, int num2) {

return num1 - num2;

}

} public class Numbers {



public static boolean isNatural(int num) {

boolean result = false;

if (num >= 0) {

result = true;

}

return result;

}

}

Spock Specifications

Once our implementation is clear, in order to start using Spock in our project, we only need to add the Groovy plugin and the Spock dependencies into our build.gradle:

plugins {

id 'groovy'

}



repositories {

mavenCentral()

maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }

}



dependencies {

testCompile

platform("org.spockframework:spock-bom:2.0-M4-groovy-3.0")

testCompile "org.spockframework:spock-core"

}

Once Spock is enabled, we can take advantage of this super handy tool, which doesn't need any extra library to cover all the unit test requirements that we need to use, like mocks or asserts.

Note that, as a collateral benefit in this configuration, everything is aligned to take advantage of Groovy 3

On the other hand, a parametrized test is the best way to automate a specification against a specific dataset. Therefore, instead of creating different tiny tests for each scenario, we can cluster them into a single test block, which will be executed for each of those cases.

Conveniently, one of the best features of Spock, still being the “where” block, which enables the implementation of parameterized tests in a really readable way thanks to its DSL:

class OperationsSpec extends Specification {



@Unroll

def "Should return #result given #num1 + #num2"() {

expect:

Operations.add(num1, num2) == result



where:

num1 | num2 | result

50 | 0 | 50

76 | 0 | 76

}

} class NumbersSpec extends Specification {



@Unroll

def "Should return #result given #num"() {

expect:

Numbers.isNatural(num) == result



where:

num | result

10 | true

50 | true

-10 | false

-50 | false

}

}

Then, by running the tests with the proper Gradle command, we will verify if it is working and the unit tests are passed:

$ ./gradlew test BUILD SUCCESSFUL in 4s

3 actionable tasks: 3 executed

Mutation Tests

As already mentioned, to execute our mutations we are going to use the best tool in the Java ecosystem to do it, which is PITest. Please check the official documentation to learn more about it.

Fortunately, to start using mutation testing in our project, we only need to add the Gradle plugin for PITest and configure it into our final build.gradle:

plugins {

id 'groovy'

id "info.solidsoft.pitest" version '1.5.2'

}



pitest {

junit5PluginVersion = '0.12'

targetClasses = ['mutations.*']

threads = 4

outputFormats = ['HTML']

timestampedReports = false

}

In addition, thanks to the fact that Spock 2 is built on top of JUnit5 and the PITest last version is fully compatible with this framework, the combination of both should work out of the box.

Specially important is the junit5PluginVersion parameter, which adds the dependency to pitest-junit5-plugin and sets “testPlugin” to “junit5”.

That’s it, we are ready to execute the mutation testing in our project just with the PITest command and take a look at the generated report:

$ ./gradlew pitest >> (...)

>> Generated 8 mutations Killed 4 (50%)

>> Ran 57 tests (7.12 tests per mutation) $ open build/reports/pitest/index.html

Exploring the report, we are able to observe the current line and mutation coverages of our code:

The results are quite interesting but not as good as they should be with 88% of line coverage, but with an especially improvable 50% of mutation coverage.

Quickly analyzing the problems that we have in these tests, we can distinguish between three main issues:

Math Mutation survived after switching the addition operator in the Operation::add method

Lack of test coverage on the Operations::subtract method

Conditional Boundary Mutation survived after modifying the conditional in the Numbers::isNatural method

To improve the coverage, let’s go step by step in the following sections to understand what is happening and how to fix each of these cases.

Math Mutator

First of all, the math mutator replaces binary arithmetic operations for either integer or floating-point arithmetic with another operation. The replacements will be selected according to the operations found in the code.

For our first case, one of these math mutators changed the operation in our code, and this variation of our code has survived the test (false positive), which could be a problem for us:

Our tests are not accurate enough since (50 + 0) or (50–0) are == 0

Secondly, although it is not marked in red (still white) in the previous report, the other problem here is that our test coverage is incomplete. Particularly, we are not testing at all the subtract method in our specification.

To fix both of these issues, we need to write a new case where the result of the addition (x+y) operation is different from the subtraction (x-y) operation, and to cover the subtract method, we must implement another test in our spec:

@Unroll

def "Should return #result given #num1 + #num2"() {

expect:

operations.add(num1, num2) == result



where:

num1 | num2 | result

50 | 0 | 50

76 | 10 | 86

}



@Unroll

def "Should return #result given #num1 - #num2"() {

expect:

operations.subtract(num1, num2) == result



where:

num1 | num2 | result

50 | 0 | 50

76 | 10 | 66

}

Executing again the mutation testing Gradle task, all the previous errors should be fixed:

Conditionals Boundary Mutator

For our last scenario, let’s check the conditionals boundaries, where our mutator is capable to replace in our code the following relational operators between them: <, <=, >, or >=

This time our test didn’t cover the boundary of our conditional (num == 0)

By acknowledging the problem, that our mutation code (with num>0) survived this new round of tests, we should add this case in our dataset and cover the boundary case:

def "Should return #result given #num"() {

expect:

numbers.isNatural(num) == result



where:

num | result

0 | true

(...)

}

And running our PITest again, the previous error should be fixed: