Those runner classes that call TestNGCucumberRunner#runCukes directly in a TestNG Test method (like https://github.com/cucumber/cucumber-jvm/blob/master/examples/java-calculator-testng/src/test/java/cucumber/examples/java/calculator/RunCukesByCompositionTest.java currently does), is currently running all features and only then checking if there were any failures. When runCukes() is changed to call runCucumber(CucumberFeature), I cannot see it any other way than it will now throw an exception and stop running the remaining features, when the first failing feature has been run. Maybe the runCukes() method should be marked as deprecated, but its basic behavior should not change - to avoid breaking the current contract of the TestNG module for those that call it directly in their test methods.