cucumber/cucumber-jvm — Issue #241: Slow run times since RC16
Author: hutchy2570
Created: 2012-03-02 10:48:52+00:00
State: open

First a bit of background. We have a suite of acceptance tests which use Cucumber-jvm to rest a set of endpoints using rest requests. We have 42 Features, with 1302 Scenarios and 6465 steps executed per run. Of these around 75% are from Scenario Outlines with Examples tables. We started using Cucumber-jvm at 1.0.0.RC13 after previously using cuke4duke. A full run of the tests in RC13 took around 12 minutes. When we switched to RC16 this jumped to over 30 minutes, and hasn't improved with RC20.

We run cucumber-jvm from the cli using the Maven Exec plugin during the integration-test phase. Here is an example of our configuration:

```xml

org.codehaus.mojo
exec-maven-plugin
1.2.1

integration-test

exec

java

-Denv=${env}
-classpath

cucumber.cli.Main
${project.basedir}/target/classes/features/released
--glue
com.project.test.acceptance
--format
pretty
--format
json-pretty
--out
${project.basedir}/target/result.json
--tags
~@wip
--tags
~@broken
--tags
~@broken-${env}
--tags
~@ignore
--tags
~@deferred

```

We've done some investigation and tried runs with every release since RC13 and the change happens between RC15 and RC16.

RC13: 12 minutes
RC14: 11 minutes
RC15: 12 minutes
RC16: 32 minutes
RC20: 32 minutes

Watching the console output there doesn't seem to be any difference in the speed of individual Steps executing, but there does appear to be a significant delay before/after executing a Scenario.

These runs were performed on a Win7 x64 laptop with a Core 2 Duo T9400 with 4gb of RAM and running Java 64bit:
version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) Client VM (build 20.4-b02, mixed mode, sharing)