My Top 5 lessons on Test Automation
By: Andrejs Doronins
Tags: Coding, Testing, Test Automation, Programming, Quality Assurance

I published a story recently:

These lessons are my personal favorites from the book “Lessons Learned in Software Testing” and I discuss why they are of particular importance, but I deliberately left out lessons from Chapter 5: Automating Testing.

There are two reasons for this:

It’s a huge chapter with 40 lessons

I’m an automation engineer, so issues and challenges in automation are of particular importance to me

So I couldn’t resist but write a separate story discussing the most important lessons on automation. I like and agree with most of those 40 lessons, so it was a challenge to pick the top ones. I applied two criteria to do so:

Lessons that reflect the most frequent issues I encountered throughout my career or heard from others

issues I encountered throughout my career or heard from others Lessons that highlight high-impact issues

1. Don’t automate a mess

Lesson #107

Frequency: medium

Impact: very high

You’ve read some documentation about the System Under Test (SUT), you’ve had several sessions with your colleagues who explained the SUT to you, and now you’re tempted to get started, automate some tests, and show others how you add value to the project!

But you’re probably not ready yet! Before you write your first line of automation code, please invest more time into gaining a greater understanding of what you’re dealing with. Be curious, ask for greater detail. A single innocent fact might significantly alter your understanding and impact your course of action.

You may have heard this classic quote by Bill Gates:

The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.”

Or as the authors of the book put it even more succinctly:

If you have bad tests, automation can help you do bad testing faster.

Indeed, the universal Garbage-In-Garbage-Out (GIGO) law applies to automation as well.

Did you receive a suite of tests created by manual testers and your job is to convert them to code? I’ve never seen this work well. Manual testers create tests to suit their needs (that’s OK) without understanding the challenges of automation. The resulting automated tests are clunky at best, unmaintainable at worst.

Also, are you knowledgeable enough to challenge some of the tests done by others? Perhaps they’re wrong, or there is a gap, or there is an overlap, or many scenarios verify multiple things, or the suite is poorly structured? If you don’t know the SUT well enough, you will miss these flaws and “migrate” them right into your codebase.

As an Automation Engineer, it is absolutely your job to gain an understanding of the domain and the SUT that is deep and broad enough to create robust, valuable test scenarios yourself!

Tests and the testing process are directly under your control and as a QA professional, you should ensure high quality of what you do.

Of course, other factors may be less under your control, such as a large, complex, and unstable testing environment or developers not writing unit tests, but those are separate matters.

2. Avoid complex logic in your scripts

Lesson #125

Frequency: high

Impact: medium

OK, so you have ensured that you understand the SUT and the tests you are automating are of good quality.

But good input may still result in poor output!

Good test scenarios converted to bad scripts

There are dozens of ways to make your scripts worse than they could be, but one that occurs all too frequently is the presence of unnecessary complex logic. Sure, complex systems may require a fair amount of support or utility code to handle API calls and other actions and computations.

But those should not be present in the body of your tests. All such code should be wrapped into helper methods and moved out. These evolve into class structures and entire Test Automation Frameworks (TAFs).

Here are the top two signals that indicate your scripts are more complex than they should be:

if-else or another kind of branching, looping

Branching is frequently a hint that (a) your test is not making use of a parametrization or (b) it is verifying several things and it should be split.

(a) In Java, the TestNG framework gives us Dataproviders. JUnit has something very similar.

(b) “With input A, we verify one kind of behavior, and with input B we verify another”? That’s two tests, not one, so please split it.

2. Any kind of nesting

Your tests should be completely flat and thus read like a story. Can you draw a straight vertical arrow on the side of your script from top to bottom?

Some nesting in production or test frameworks is OK but should be minimized anyway via standard refactoring practices.

But in tests, I advocate for (near) zero tolerance towards nested code. Wrap all helper code into utility code and move it elsewhere for later reuse.

When tests get too complicated, they tend to become buggy. Keep it simple. Keep your tests linear.

Naturally, nesting should not be confused with indentation to improve readability, especially in case of libraries such as AssertJ that make use of Fluent Interface:

3. Don’t mandate 100 percent automation

Lesson # 105

Frequency: very high

Impact: medium

You and your team have written dozens or hundreds of tests this month. They pass, they are stable and they even caught a regression bug. Great job!

But they only cover a fraction of all possible test scenarios, right? And so the inevitable question arrives (from management, no doubt) — what is the coverage? When will we achieve 100%?

Forcing 100% coverage in unit tests is a bad idea. Some debate this by adjusting the definition of coverage. There’s also a difference between “code coverage” (poor) and “branch coverage” (better).

From Vladimir Khorikov’s excellent book “Unit Testing Principles, Practices, and Patterns”:

[…] relying on coverage metrics to determine the quality of your test suite is not enough. It can also lead to dangerous territory if you start making a specific coverage number a target, be it 100%, 90%, or even a moderate 70%. The best way to view a coverage metric is as an indicator, not a goal in and of itself.

Mandating 100% coverage for higher-level tests is even worse!

Why? Because it solely depends on the test scenarios you came up with!

Here’s how it goes:

You thought of 10 tests for a feature and you automated them You said that you have 100% coverage. Management inevitably interprets that to 100% safety. A bug is discovered in production because you didn’t think of that 11th case. Or it was a complex integration issue. Good luck explaining yourself.

Managers want and need numbers to gauge performance. You won’t change that. But never promise 100% coverage. Instead, communicate what you could and could not achieve with the time and resources you had.

4. Beware of automators who don’t understand testing

Lesson #134

Frequency: medium

Impact: high

You surely know that good Test Automation Engineers are a balanced blend of testing skills and programming skills, right?

Sometimes programmers are tasked with test automation. Their testing knowledge is nearly non-existent and their attitude may be even worse (“it’s not real programming”). At best, you will have clean, well-programmed tests that don’t verify much beyond the happy path.

Too much focus on programming and too little focus on testing is not a good thing.

The Testing Techniques section on guru99.com is a good start.

Chapter 3: Testing Techniques from the very book this article is based on is even better.

5. Test automation is a software development process

Lesson #118

Frequency: high

Impact: very high

The previous point stated that programmers underestimated the skill of testing and it results in (maybe) clean but low-value test automation.

This point is the opposite — good testers who learn some scripting, but otherwise copy-paste their way forward without applying clean code basics and other principles until the codebase reaches critical mass and the project is abandoned because it’s just unmaintainable.

In other words, testers who shift to automation don’t learn “real” programming. Everyone has an opinion on what“real” programming is.

As far as I’m concerned, you don’t have to implement classic sorting algorithms on a whiteboard or understand low-level memory management stuff of C++ (unless your job directly requires you to).

But here’s a list of skills and principles that I’d want my teammate to have and know:

It takes years to master these topics, but as long as the individual and the team keep striving — they should be fine. But without them, any long-term test automation endeavor is doomed to fail.

Honorable mention: Encourage the development of unit test suites

Lesson #133

Frequency: rare

Impact: unknown, but probably highly beneficial

Have you ever caught a bug with a higher-level functional test and thought to yourself “This really could’ve been caught by a simple unit test. Don’t they have one?”

I certainly did. But I only got as far as notifying developers and suggesting they add a unit test for something. I would never know if they did.

Everywhere I look, I see developers doing their own thing (writing unit tests) and Software Developers in Test (SDETs) doing their own (writing higher-level tests). Neither has a clue about the test suite of the other. I think this is hugely inefficient.

There is a lot of talk around “shifting left” — testing as early as possible and thus preventing bugs instead of reacting to them. I firmly believe that capable Automation Engineers can and should contribute to unit test suites.

No, not take over full responsibility and rid the developers of this laborious task, but contribute.

But as of 2020, this practice is anything but widespread. In fact, I only know of one company that does something like this — Google. In their book, “How Google Tests Software”, the authors explain that their SDETs are pretty much normal developers who sit and code with the rest of the team, but they also focus on, promote and enforce all possible quality standards and test automation principles.

Only time will tell if the rest of the IT world will follow their example.

Conclusion