Faster Cucumber Tests
By: John Sextro
Tags: Testing, Software Development

Let’s face it you can never have enough money and your tests never run fast enough. I can’t help you with your money, but if you follow the steps in this post I can help you speed up your Cucumber tests.



The improvement process that I’ll walk you through is known commonly as DMAIC (Define, Measure, Analyze, Improve, Control). For those not familiar with DMAIC, Google can tell you everything you need to know. But, basically DMAIC is an improvement cycle commonly used in Six Sigma to improve and optimise business processes. We’ll use DMAIC in this post to improve the speed of your Cucumber tests.

Define

You’ve probably already defined the problem, your tests are too slow. What you haven’t done is define success. You will define success in terms of execution time. For instance, if your Cukes are taking 30 mins you might define success as, “Make all Cukes run in 20 mins or less”, others might define success as, “Cukes running in 5 mins or less”. This is up to you and your team to decide. Just remember, the amount of time it takes to run your tests will be inversely proportional to number of times you will run them. So the faster they are the more often you will run them.

Measure

You know how long it takes to run the suite. You might even know how long it takes to run each scenario or sets of scenarios. What you need to know now is how long it takes each step to run. Wait! Before you go diving back into your steps to instrument them with timers and such, let’s see what Cucumber can do to help us.

Fortunately for us Cucumber offers a number of helpful options for formatting the output of our test runs. One of those options is — format usage. Using the ‘usage’ format will show us which steps take the longest to run, how many times each step executed and if any steps are unused. I’ve slightly modified some of the example feature files provided with the Cucumber source to generate the following output.

Analyze

With our new data in hand we can begin to zero in on the trouble spots. The usage report displays the steps ordered from slowest to fastest. The report also shows the number of times each step executed. Your slowest step might not be the biggest problem. You should focus on the steps that run the slowest and the most often. Also, consider why the step executed so often. If you wrote the scenarios following a behavior driven development (BDD) cycle then the scenario was written to define behavior in the system. As a result you may find many scenarios exercising the same part of the system in only slightly different ways. Viewing the scenarios in hindsight you should consider if you can consolidate or eliminate some of the scenarios.

Improve

Now that we know where to focus our efforts it’s time to make our improvements.



In the ‘Analyze’ phase you considered if you could remove or consolidate scenarios. Now is the time to do that work. Deleting scenarios is fun and easy. Consolidating scenarios takes more effort and thought. Look for recurring patterns such as screen navigation and data setup. If you notice that you repeatedly setup the same data to run different scenarios, it’s a good bet that those scenarios are ripe for consolidation. From the regression testing perspective, you want to maximize the amount of testing performed while minimizing the amount of work you have to do in the system to prepare for the testing while maintaining the ability to independently execute the scenario.



One of the most common causes of slow downs in cukes is ‘sleep’ing. People often use sleep to wait for AJAX activity to complete. This is the most unpredictable and wasteful way to control test execution. Imagine that an AJAX action normally takes between .5 and 2 seconds to complete. If we sleep to wait for the action to complete how long should we sleep? Let’s say we decide to sleep for 3 seconds. If that action ever takes more than 3 seconds the test fails. Also, every time the action takes less than 3 seconds the difference is wasted doing nothing and if this is one of your most executed steps those seconds quickly turn into minutes.



Instead of sleeping you must learn to ‘wait’. The Selenium client includes a number of ‘wait_for’ methods that will improve the predictability of your steps and will remove the waste caused by sleeping.

wait_for_text(value) — Simply waits for the text you specify to appear in the DOM. Should this value already exist, execution will continue without waiting, so carefully select a value that you know won’t exist until the AJAX action completes.

wait_for_element(xpathLocator) — Allows you to specify an exact xpath locator to wait for before continuing.

wait_for_no_element(xpathLocator) and wait_for_no_text(value)– Allows you to wait for the removal of an xpath locator or text value from the DOM before continuing.

You can learn more about xpath and selenium from my Cucumber Pro Tips presentation, http://blip.tv/this-minute-in-programming/cucumber-pro-tips-the-secret-sauce-6078421

Good Software Engineering

Once you’ve ‘picked’ all the ‘low hanging fruit’, you’re left with your software engineering skills and training to profile, tune and tweak the last bits of improvement from your steps.

Scenario Tagging

Cucumber provides a simple yet powerful ability to add tags to scenarios using the @tagname. If you’ve made all the improvements that you can reasonably make but your tests still aren’t fast enough, then you can use tags to define a subset of scenarios to execute. For instance, you could tag the scenarios with @smokeTest tag and run them by specifying ‘ –tag @smokeTest’ at the end of your cucumber command. Don’t forget to run your full suite of Cukes nightly, or better yet schedule them to run more frequently via your continuous integration server. Checkout the Tags page on the Cucumber wiki for more details on tags, https://github.com/cucumber/cucumber/wiki/Tags

Control

Congratulations. Give yourself a pat on the back. Hopefully you’ve met and surpassed your success criteria. Wait a second though, we’re not done. You need to keep an eye on things. Don’t let your test times creep up. That’s what the ‘Control’ phase is all about and it goes on for as long as you care about the amount of time required to run the scenarios. Make the ‘care and feeding’ of your cukes a regular part of your development cycle and they will provide your application with benefits for years to come.

About the Author

John Sextro works as an Agile Coach for Asynchrony Solutions (www.asolutions.com), a provider of agile software development, enterprise architecture, and systems integration services. For more information, visit http://blog.asolutions.com or contact John at john.sextro@asolutions.com.