Automation Testing Expectations
By: Franco Santi
Tags: Automation Testing, Testing, Automation Software, Test Automation Strategy, QA

Introduction

Many companies try, struggle, fight, and many ultimately fail at adopting a successful test automation strategy, and as a result come to the wrong conclusion that test automation doesn’t work, when in fact, it is just not effective in the way they are trying to implement it. One main reason I have seen test automation fail, is due to unreasonable expectations or expecting the wrong things from automation. So, with that being said, and in order to avoid these common pitfalls, here are a few recommendations you can follow.

Define Realistic expectations.

Clarity of the benefits.

Make sure everyone is onboard.

Identify what’s good to be automated.

Pick the right automation strategy.

Define Realistic Expectations

You can expect your automated tests to save your team time, and the headache of having to test everything manually. However, you cannot completely rely on automation when it comes to software testing as not everything can be automated, or even should be. When you automate testing, you are really just having a computer program checking the system in ways that were predetermined ahead of time, but humans are still needed to do the non-automated aspects of testing. Also, keep in mind that you must have your manual testing done before, at least the test designs, so that you know what you’re automating in the first place. Crawl, walk, then run!

It’s important to remember that the value of an automated test comes from the information that it uncovers about software quality, not from the number of tests executed, nor the frequency with which they’re run. What’s essential is to obtain the right information that will be shared with the team upon which they can make critical business decisions. So, don’t expect that by automating everything, all of your problems will disappear.

Additionally, once your automation is done, there will still be a considerable amount of man-hours required to maintain scripts due to application changes, data updates, etc. Automation is hardly a “set it and forget it” way to test.

Targeting 100% Automated Testing

We all know that 100% testing is not possible. This is directly proportional to the design of the test, and the application development. It is advisable to aim at functionality of the testing that is what matters most to any organization. There is no automation tool that is able to do everything. Therefore, at the level of a simple test it's not difficult to automate, but at an integration level with several more sophisticated tests, that requires different test types, it will become complicated.

Even if you managed to automate all of your testing (which as it was mentioned it is unlikely to happen), you still need testers around to manage those automated tests. The application is constantly changing and growing, and as a result, you’ll need to update your tests as the application changes and add new tests as well. On top of that, it’s not usually cost effective to automate all of your testing. Non-functional testing aspects like usability, layout, accessibility and more, are incredibly difficult and expensive to automate all aspects of. Typically, it’s much cheaper to automate parts, and manually examine the rest.

Clarity of the Benefits

One of the biggest myths about test automation is that automation will create a downfall in the number of resources. Test automation tools will need people to support and script development. Test automation cannot replace manual testing. Manual testing is absolutely essential. Since test automation will follow manual testing, it needs to really embody this perspective. So, it’s not correct that in an organization automation will reduce your manual team.

What ends up happening, as a result of this misconception, is that managers and companies do not get the value they were expecting out of automation. They were expecting major cost cutting in personnel, but that didn’t happen. So even if you have succeeded in writing a fantastic framework, and automating all the things, you’ve failed. Instead, you need to set the expectations properly. No, we’re not going to fire all the testers, instead we’ll use their time more efficiently. Because what you will find is that when you don’t need to test X all of the time given it is automated, you can finally get around to test Y. We have limited time, and I have never met a tester who has enough time to test everything they want, automation can enable them to free up some time to get that overlooked work done. That in turn produces greater test coverage, which can lead to a better (and hopefully lower) risk profile when it is time to release.

A side effect of being able to expand coverage, is that more bugs should be able to be found sooner. This in turn saves money.

This added automation can also reduce development time, whether it is by speeding up CI, fewer deploys to QA, or even just a shorter regression cycle; it is an immediate productivity boost for everyone. While these things will take time to achieve, there are benefits, just not the ones your boss might be expecting. So it’s important to make sure to outline the real benefits, especially if you hear people expressing the wrong expectation. Reducing the repetitive tasks will also enable the team to perform better, impacting positively on their motivation.

Make Sure Everyone's Onboard

For any project to be successful, it needs the full support of all of the people and areas involved. Maybe, instead of your boss being the one to insist on implementing test automation, it’s the other way around. Perhaps you need some help to get the team and your leaders to realize the value of test automation. You might also need to get your developers on board with getting in the habit of automating unit tests, as those will give your team the greatest bang for your buck.

If that’s the case, you can present some information about the possible return on investment of test automation.

Remember to be very transparent with the team and make sure everyone shares the same realistic expectations that you set for the automation project. Don’t mislead them by asserting that automation doesn’t require much effort and resources upfront, because it certainly will.

Identify What's good to be automated

Now that we know that not everything can or should be automated, that begs the question, what should we automate?

Automation is particularly useful when your software has accumulated a lot of technical debt, when tests are very time consuming, and when you are working with several complexities, in a long-term context.

That's why, most teams start out by automating smoke tests, followed by regression, given the situation in which they already have a defined test suites that must be executed periodically before each product release. In this case, the effort to run them manually becomes repetitive and takes time away from other tasks that are not as easily automatable, yet highly valuable, like running accessibility tests.

Pick the Right Automation Strategy

More important than any tool you may choose to automate with, it’s important to follow a well-thought-out strategy to guide your automation. There are two that I’ll mention here: the test automation pyramid and risk-based testing.

Test Automation Pyramid

The agile test automation pyramid improves the ROI of automation and sets a guideline for receiving the most benefits from it.

The pyramid states that most testing should take place in the development stage, with developers running their own unit tests after every build. These tests are the easiest, cheapest, and fastest to complete. Here, any existing bugs will have the shortest life span, as developers can find and remove them almost immediately. After running unit tests and all of them pass, the pyramid suggests moving into the API/integration/component testing phase.

This is where the logic and business processes are tested without having to go through the UI. Compared to testing on the UI level, here you’ll have fewer problems, easier maintenance, and faster test execution. Lastly and least often, automate some UI tests. Run as few of these as possible since they’re the most costly, time-consuming and fragile. Be careful with these tests as they are the most likely to provide false positives and negatives. After reaching the top of the pyramid, manual and exploratory testing can be conducted.

Risk-Based Testing

Along with the test automation pyramid, it’s a good idea to consider risk-based testing. This test strategy places a higher priority on testing elements of the system that are the most at risk of failing and whose failures would be most critical to the business.

To get started with this strategy, it’s needed to run a risk analysis to decide which test cases to automate, taking into account different factors. For categorizing tests by priority, a widely used method is MoSCoW, which is an acronym for Must, Should, Could and Won’t. Once the priority of the tests has been established, it’s advisable to check them every once in a while, given that the business or client requirements might change.

Final thoughts

Having a successful automation project is not only about selecting an automation tool and starting coding. Sometimes it is even more important to have everyone on-board with clear/specific expectations and targets. Defining the right Strategy is as important as the experience of the automation engineers and tool selected, so, do the homework before starting, and do not rush to the code.