Exploratory Testing in an
Agile Context
A guide to using Exploratory Testing on Agile soft waredevelopment teams.

Elisabeth Hendrickson

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

2

What is Exploratory Testing Anyway?
Exploratory Testing involves simultaneously learning about theExploratory Testing.

software under test while designing and executing tests, using

So you bang on thefeedback from the last test to inform the next.

keyboard randomly,
right?

Um no,
not really.

Cem Kaner coined the term in the 1980’s, though the practiceundoubtedly started much earlier.
Because there is a common misunderstanding that Exploratory
Testing means simply “do random stuff and see what happens,” it’s
important to emphasize that this is a rigorous investigative practice.
We use the same kinds of test design analysis techniques andheuristics in Exploratory Testing that we do in traditional test
design, but we execute the tests immediately. The test design andexecution becomes inseparable, a single activity.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

3Tests as Experiments
When we design tests while exploring, we have a hypothesis about
how the software will behave: perhaps we suspect the software will
exhibit a particular type of error, or perhaps we are seeking toconfirm how the software works.
Either way, we think of a little experiment, and then perform itimmediately. Our experiments teach us about the system. We learn
how it works, what makes it tick, how it’s organized. We discover
not just what works and what doesn’t, but also general patterns ofvulnerabilities.
The more we learn about how the system works, and thecircumstances under which it misbehaves, the better we are atdesigning good experiments that yield useful information.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

4TestingTool?

Really? No Keyboard Banging?
Sometimes we do things when exploring that seem odd to an
outside observer.
James Bach talks about his “Shoe Test” in which he places a shoe on
the keyboard.
But James doesn’t put a shoe on the keyboard because he’s trying tocome up with wacky random stuff. He does it because he has
noticed that some software exhibits bad behavior when it receives
too many key inputs at one time. Placing a cat on the keyboard, orhanding the keyboard to a 2-year-old, might result in similar
behavior. But a shoe or a book is usually more handy.
So yes, there might be keyboard banging involved in Exploratory
Testing. But it’s keyboard banging with a theory of potential error,

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

5not just random wacky stuff because we can’t think of anything
better to do.

Exploratory Testing and Agile
Exploratory Testing is a core testing practice for Agile teams.
In an Agile context, scripted regression tests are typicallyautomated. The Continuous Integration (CI) system executes thesetests with every build. Such automation is essential to obtaining thefast feedback that Agile teams need to deliver frequently at asustainable pace.
However, all that automation is not sufficient to ensure that thesystem does everything we expect and nothing we don’t. Someone,
somewhere, has to actually use the system.
And yet writing detailed test scripts for a manual test effort doesn’t
make sense in an Agile context. If it’s a test that’s important enough
DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

6
to script, and execute repeatably, it’s important enough toautomate. 1
So Agile teams still need manual testing, but don’t need repeatablefollow-a-script testing. Instead, Agile teams need a manual testing
approach that’s adaptable (because the software under test changes

1

Discussing automation strategies is outside the scope of this book. But yes, I do thinkthat if you can write a manual script for a test, you can automate it. If you can give ahuman repeatable step-by-step instructions for executing a test, you can write anautomated test to do the same thing. And if you’re going to argue “but you can’t write
software to unplug a network cable” then I’m going to tell you that you can. Or more
specifically, the programmers who write the network error handling logic can make itpossible by writing the code such that a simulator can fake pulling the network cable.
You’ll still need to do manual exploration, of course, but having a small number ofrepeatable automated regression tests around network cable unplugging will save theteam a world of grief on the day that someone accidentally checks in code that breaks
that error handling.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

7
very quickly on Agile projects) and that produces large amounts ofinformation quickly (because Agile teams thrive on fast feedback).
Exploratory Testing is a perfect fit.
Instead of documenting step-by-step instructions, we capture justsimple charter statements that represent questions we want testing
to be able to answer.
Then using charters to focus our explorations and heuristics toguide us, we move through the software rapidly, poking andprodding to reveal unintended consequences of design decisions
and risks we didn’t consider in advance. In doing so, we gather large
amounts of information very quickly.
Further, because Exploratory Testing involves using the results fromthe last test to inform the next, we can adapt our testing approachquickly if we notice something that might indicate a vulnerability orDRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

8risk. We can tailor our investigations to what’s important right now,
without being constrained to follow a test script that we thought
was important some number of months ago.
This book covers the essential elements of Exploratory Testing:learning the system from the outside in; designing tests using
heuristics; executing tests and observing results closely; andintegrating Exploratory Testing into story development within aSprint or Iteration.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

9Learning.
In which we discuss learning about thesystem and capturing our understanding forlater use in designing tests.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

10Simple Scenarios
What is the software intended to do?
This is where you start learning the system: with the simplest casethat should work. You’re not testing the software yet, but you are
testing your understanding.
Are you exploring a shopping cart? Add an item. Check out. Testinga word processor? Write some text. Add an image. Save it. Testing a
DNS server? Query it to get an IP address for a domain. Whatever itis that you’re testing, start by doing basic user actions to see what
the system does.
As you go, start taking notes about...

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

11Nouns and Verbs
The nouns of the system are the things that it manipulates,manages, and interacts with. Typical nouns for a shopping cart
would include things like: carts, registered users, items, quantities,
prices, discounts, coupons, payment methods, and addresses.
The verbs of the system are the actions you can take using thesystem, or that the system can take. In the shopping cart example
verbs could include: checkout, update, remove, abandon, save,login, logout.
As you collect a list of nouns and verbs for the system, you’recapturing a language that describes the system. This will come inhandy for designing tests, as well as for further explorations tolearn more about the shape of the system.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

12Alternate Paths
As you explore the system noting nouns and verbs, you’re likely to
notice that there is more than one way to do things.
Perhaps you can login by bringing up a Login page, and also by
filling in your login information into username/password fields thatappear at the top of each page. Perhaps there’s a “remember me”
option that allows the system to keep you logged in for days orweeks.
Perhaps you can delete an item from the shopping cart by changing
the quantity to 0, or by clicking a Delete button, or by emptying
your shopping cart.
Whenever you notice that there are multiple ways to get to the same
result, make a note. You’ll want to vary these paths as you explore.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

13Context Diagrams

PaymentGateway

Catalog

Software lives within an entire ecosystem. It runs on an operating
system, stores data on the file system or in a database, andintegrates with other software, like payment gateways for credit

Shopping
Cart

card processing, SMTP servers for email, and internal systems likesales reporting, inventory control, and warehouse/shipping

WarehouseSMTP
Server(Email)applications.
As you begin exploring, create a simple context diagram that shows
external resources or dependencies, like the one to the left. Overtime, when you learn more about the system and the context within
which it lives, add more details.
Every dependent system provides additional opportunities forexploration.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

14

Variables
If you’re a programmer, a variable is a named location in memory.
You declare variables with statements like “int foo;”That’s not the kind of variable we’re talking about here.
Rather, we’re talking about “variable” in the sense of “things you canvary.” More specifically, a variable in testing is anything that you canchange, or cause to be changed, via external interfaces (like the UIor the file system), that might affect the behavior of the system.
Sometimes variables are obviously changeable things like the value
in a field on a form. Sometimes they’re obvious, but not intended tobe changed directly, like the key/value pairs in a URL string for a
web-based application. Sometimes they’re subtle things that canonly be controlled indirectly, like the number of users logged in at
any given time or the number of results returned by a search.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

15
Subtle variables are the ones we often miss when analyzing thesoftware to design tests. Consider for example the Therac-25 casethat Nancy Leveson describes in her book Safeware.
The Therac-25 was a radiation therapy machine that overdosed
patients with radiation, killing them. The story dates back to the1980’s, but it’s still relevant today.
According to Nancy Leveson, in at least one case, the malfunctionthat caused a death could be traced back to the medical technician’sentering and then editing the treatment data in under 8 seconds.
That’s the time it took the magnetic locks to engage. Notice two key
subtle variables, or things that a user could change: the speed of
input and the sequence of actions.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

16
Further, Leveson found that every 256th time the setup routine ran,it bypassed an important safety check. That’s yet another subtle
variable: the number of times the setup routine ran.
As you begin to learn the system, make note of things you noticethat you can change, or cause to be changed, that are subtle orotherwise interesting.
The more you look for variables, the more you’ll find. They’reeverywhere.
Then, as you explore, use the Heuristics described in the Test
Design section to suggest interesting variations.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

17Designing.
In which we discuss using heuristics todesign tests on the fly.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

18Using Heuristics
You’ve started exploring the system. You have the general shape ofit. You understand the context in which it lives, the kind of data it
manipulates, the actions you can take. You may have some ideas
about sequences and configurations that might be interesting.
This section lays out a selection of heuristics: test ideas that are
general enough to be applicable across a wide range of domains
and technologies.
Use this section as a reference when you run out of new ideas forthings to try.
[note: each heuristic will get about a page worth of explanation/examples.]

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

19File System Interactions
Net work Variations
CRUD: Create, Read, Update, Delete
Position: Beginning, Middle, End
Count: 0, 1, ManyInterruptions
States and Transitions
Time: Before, During, After
Combining Heuristics
Randomizing Noun and Verb Combinations
Following the Data
Personas

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

20

Executing.
In which we discuss observing, note-taking,and defect isolation.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

21Mechanics.
In which we discuss how to structure yourexplorations within a Sprint or Iteration inan Agile process.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

22Charters
Traditional scripted testing is documentation-centric with written
Test Plans, Test Strategies, Test Cases, and Test Procedures.
Exploratory Testing involves far less documentation. We don’t
document each and every test case. Instead, we write charters:
simple statements of the information that we hope to discoverthrough exploration.
One way of expressing charters is with the simple template:
Explore area
With resources, constraints, tools, etc.
To discover information
Some charters are broad: “Explore the system with typical usage
scenarios to discover how it works.”
DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

23
Some are narrow: “Explore the File Import feature with various
invalid file formats to discover if there are any circumstances underwhich the error handling does not give a reasonable response to aninvalid file.”
Some are about valid usage scenarios with representative user
personas: “Explore browsing and shopping with a non-technical user
perspective to discover how easy or hard it is to buy items.”
Others are about misusage scenarios: “Explore the shopping cartfeature with a tool to perform http POST requests to discover if
there’s any way to get stuff for free.”
We might express charters a little differently, and that’s fine. The
purpose of the charter is to provide a focus to our explorations.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

24

Sessions
During a session, if we notice that we’re going off on a tangent, weuse the charter to remind us what we’re supposed to beinvestigating. We might make a note of another charter toinvestigate later, but we make a point of staying on charter for now.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

25

Frequently Asked
Questions.
In which we tackle questions that seem tocome up over and over (and over) again.

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

26

References
Bach, James. “What Is Exploratory Testing?”
Bach, Jonathan. “Session-Based Test Management”
Kohl, Jonathan. “Exploratory Testing on Agile Teams”
Kohl, Jonathan, “User Profiles and Exploratory Testing”
Marick, Brian. “A Survey of Exploratory Testing”
Tinkham, Andy and Kaner, Cem. “Exploring Exploratory Testing”

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.

27

If you liked this book...
You’ll love our classes...
[blatant promotional text goes here.]

DRAFT. Do not distribute. v 060509
Copyright © 2009 Quality Tree Software, Inc.