Testing: Important Lessons You Can Learn From Bugs
By: Heidi Baker
Tags: Test Management, Software Development, Software Testing, Testing, Quality Assurance

When I first started testing, I was stoked when I found a bug. It was reassuring I was doing my job right. But by the time I became a Test Lead I had a significant amount of experience in finding bugs and realised it was only the really quirky bugs, the ones you really had to root out from the code using complex, yet realistic, test criteria that were truly satisfying to unearth.

By the time I was a Test Lead I liked to think I was helping the Developers I worked with learn how to do some ‘Dev Testing’ before handing work over to the Test Team. So, there was little joy in finding very obvious bugs I believed should never have got into a test release.

Typos, basic layout bugs: If I was testing against a wireframe or build spec and there were more than a handful of bugs, I would stop testing. Raising a long list of bugs in this scenario is not the best bang for the client’s buck. Either the documentation you are testing against differs from the one developed against, or the developer has not checked their work. Stopping testing to check the situation is the most efficient approach. You only have to spend hours testing against an out of date document once, to learn to ask early next time. If the document has changed, a chat about the value of version control and shared document storage can be had, and little damage has been done. However, if the developer has just made a lot of mistakes, it might be quicker to either screenshot the page and mark up all the issues with circles, or walk them through each issue before leaving them to fix, check and redeploy. Taking this approach helps the Developer learn to check their own work and take a little more accountability for quality, while leaving you, the Tester, to focus on code that is actually ready to be tested.

Happy path and Acceptance Criteria bugs: Testing usually starts with the ‘happy path’, you’ll likely need to set up some test data before you get down to the trickier tests, so you try to use the new functionality as it is intended, nothing clever. From there you would do a quick run through of the Acceptance Criteria — just to make sure that all passes. When these tests fail you can be sure the Developer did not Dev Test their own work, or there has been an issue with the deployment. These types of bugs are usually showstoppers. It is worth checking with the Developer whether the function worked for them to help isolate the problem. If it did not, I will hand the code back asking them to Dev Test before handing over to the test team. If it did, we can check what happened with the deployment, get it fixed, and we are unblocked without losing too much time.

Negative test, and complex bugs: I am truly in awe of the amount of knowledge Developers have, they are very clever people. I do not expect them to be thinking of all the scenarios and all the permutations the software needs to respond to. That is my job. So, finding these bugs is satisfying. It shows how deep the Tester understands the use of the software and how thoroughly we put a system through its paces. I happily log these bugs.

“What the Heck Happened there?” bugs: Sometimes Testers end up raising a bug for some very quirky system behaviour, or a defect that only shows up intermittently or with very specific criteria. We find these using a special skill only Testers have, I call it “Tester’s Luck”. There could be a list of a hundred names and a Tester could click on the only one that is not set up correctly, and break the software. It’s a gift. But the beauty of these bugs is finding out what is causing the issue. Most of the bugs discussed in the sections above have obvious causes, but if the Tester does not understand the cause, it is very important to learn what happened to successfully retest the fix. Besides this, you start learning where to look for these bugs in the future. Here are a couple of examples I have run into, multiple times.

Race Conditions: An intermittent bug can be caused by race conditions — which is when two or more back end tasks are set to run at the same time, but need to run in a specific sequence. Sometimes the order is correct and the function works, sometimes it is not, and it does not. The code will need to be fixed to define a set order to avoid this occurring.

List limits: When importing batch data/during migrations there can be unseen list limits (i.e. the amount of data might be capped) between systems. Losing apparently random data during a migration or upload can be very hard to diagnose, but if you’ve hit a list limit (e.g. SharePoint used to throttle at 5,000 list items) in the past, it gives you a starting diagnosis the next time you encounter it.

When the tester encounters these quirks, they should add new tests to their suite to deliberately check for these kinds of defects in the future.

Bugs have a bad reputation, but when found, the whole team has an opportunity learn and grow from what bugs teach us. Use that opportunity.