What Is An Innovation?
•innovation (ĭn'ə-vā'shən)
1. Something new or different
2. Something newly introducedor adopted
3. A creation (a new device orprocess) resulting from study andexperimentation

Lee Copeland
lee@sqe.com© SQE 20112Are All Innovations?
•For some of you – Yes•

For others -- No

Trends and Innovations•

Process•

From Agile•

Education•

Technology and Tools•

Process Improvement

34Trends and Innovations•

Context-Driven SchoolProcess•

Testing groups exist to provide
testing-related services. They donot run the development project;they serve•

It is entirely proper for different test groups tohave different missions. A core practice in theservice of one mission might be irrelevant or
counter-productive in the service of another•

There are NO BEST PRACTICES – there are,however, good practices in specific contexts− Context-driven School− Checking vs. Testing− Crowdsourced Testing− The Weekend Testers− Exploratory Testing561

Trends and Innovations•Checking vs. Testing

Process•

Checking is something that we dowith the motivation of confirming
existing beliefs•Checking is a process ofMichael Bolton
confirmation, verification,and validation. When we already believe
something to be true, we verify our belief bychecking•

Checking is a highly automatable process− Context-driven School− Checking vs. Testing− Crowdsourced Testing− The Weekend Testers− Exploratory Testing

http://www.developsense.com/blog/2009/08/testing-vs-checking/
78Trends and Innovations

Checking vs. Testing•

Testing is something that we do with themotivation of finding new information•

Testing is a process of exploration, discovery,investigation, and learning. When weconfigure, operate, and observe a productwith the intention of evaluating it, or with theintention of recognizing a problem that wehadn’t anticipated, we’re testing••

Process− Context-driven School− Checking vs. Testing− Crowdsourced Testing− The Weekend Testers− Exploratory Testing

Testing is not automatable. It’s using our brain
in real time, and it’s really fun
910

Crow dsourced Testing•If virtualization makes sense for hardware,
why not for test teams?

•Crowd testing is using a virtual
test team rather than, or inaddition to, an organization’sdedicated team•

Crow dsourced TestingYour organization defines a set of test
requirements (types of testing, scenarios,environments, platforms, etc.)

11•

A crowd test vendor (utest.com,mob4hire.com, J) identifies a pool of
qualified testers, creates a test project,assigns the work to testers, monitors theprogress, and reports defects•

The crowd test vendor compensates testers
and evaluates their performance•

Testers receive “Pay Per Bug” and also “PayPer Valuable Feedback”

122Trends and Innovations•

The Weekend TestersProcess•

The Weekend Testers began with a group ofpeople in India who wanted to learn to testbetter•

A typical weekend session
consists of registration,facilitation, a testing session,and a follow-up discussion•

It’s all done through a Gmail chat group. The
facilitator provides product download details,
test mission, and one hour time limit− Context-driven School− Checking vs. Testing− Crowdsourced Testing− The Weekend Testers− Exploratory Testing1314

The Weekend Testers

Trends and Innovations•At the end of the time limit, testers participate
in a group discussion for the next hour•

They share experiences, challenges, defects
found, traps they got caught in, and otherideas− Checking vs. Testing•

Testers not only hone their skills, but shareand learn new test ideas, strategies, and tools− The Weekend Testers•

Want to join, or form your own group?

− Exploratory Testing•

Process− Context-driven School− Crowdsourced Testingwww.weekendtesting.com
15

Exploratory Testing

16

Exploratory Testing

Learning“The classical approach to test
design is like playing ‘20 Questions’by writing out all the questions inadvance.”
simultaneous

Test Execution- James Bach

Test Design17183

Exploratory Testing

Exploratory Testing“Exploratory Testing, as I practice it, usuallyproceeds according to a conscious plan. But not arigorous plan J it is not scripted in detail.”• Exploratory testing can be concurrent withproduct development and test execution“To the extent that the next test we do is influenced
by the result of the last test we did, we are doing
exploratory testing. We become more exploratorywhen we can’t tell what tests should be run inadvance of the test cycle.”• Based on implicit as well as explicit (if theyexist) specifications as well as the “as built”
product• Starts with a conjecture as to correct behavior,followed by exploration for evidence that it
works/does not work

- James Bach
1920

Trends and Innovations•

Test-First Development

From Agile
Developers “refuse to add
even a single line of code
until a test exists for it.”− Test-first Development− Acceptance Test-driven Development- Scott Ambler(ATDD)Then,

TDD = TFD + Refactoring

21

22

Test-First Development•Trends and Innovations

Goals−Accurate specifications−Useful specifications−Focus the mind−

Write clean code−

Prevent the creation of an “inventory” of things
that don’t work•

From Agile− Test-first Development− Acceptance Test-driven Development(ATDD)

23

244

ATDD
•

Acceptance Test Driven Development (ATDD)
is the equivalent of TDD but at the functional
level rather than the unit level•

Acceptance tests are often
written by users (with theguidance of professionaltesters) using frameworks
such as FitNesse

ATDD•Like TDD tests, ATDD tests are createdand implement before code is written•Like TDD tests, ATDD tests can take theplace of pages of written requirements

25

26

Trends and Innovations•

Really Good BooksEducation•

A Practitioner's Guide to Software
Test Design – Copeland•

Secrets of a Buccaneer Scholar – Bach•

Systematic Software Testing ─ Craig andJaskiel•

Lessons Learned in Software Testing ─ Kaner,Bach, and Pettichord− Really Good Books− (Big) Testing Conferences− (Small) Testing Workshops− Freedom of the Press

27

28

Really Good Books•

Testing Computer Software ─ Kaner•

How to Break Software ─ Whittaker•

The Art of Software Testing ─ Myers•

The Craft of Software Testing ─ Marick•Managing the Testing Process ─ BlackTrends and Innovations•Education− Really Good Books− (Big) Testing Conferences− (Small) Testing Workshops− Freedom of the Press

29305

(Big) Testing Conferences•

STAREAST and STARWEST•EUROSTAR•(Big) Testing Conferences•STANZ•

STC – QAI in India

•

JaSST

Generalized focus and Lecture style−Attendance solicited throughout the community−

Unlimited participants−Focus on a large set of topics−Participation is not generally required−Presentations are collected, published, andavailable to all participants

3132

Trends and Innovations•(Small) Testing Workshops

Education•

Specialized focus and Participatory style− Really Good Books−

Attendance by application
and/or invitation− (Big) Testing Conferences−

Limited to 15-20 participants− (Small) Testing Workshops−

Focus on a small set of topics−Participation is required; challenging presenters
is encouraged−

Learnings are collected, published, and available
to all participants− Freedom of the Press

33

34(Small) Testing Workshops•Trends and Innovations

Examples are:

•Education−Austin Workshop on Test Automation− Really Good Books−

London Exploratory Workshop in Testing
Los Altos Workshop on Software Testing− (Big) Testing Conferences−−

Workshop on Heuristic and ExploratoryTechniques−

Workshop on Open Certification of software
Testers−

Workshop On Performance and Reliability−

Workshop on Teaching Software Testing− (Small) Testing Workshops− Freedom of the Press

35

36

6

Freedom of the Press•

Freedom of the Press“Freedom of the press is limited to those whoown one.”•─ A.J. Liebling(American Journalist)•Blogs–James Bachwww.satisfice.com/blog–Michael Hunter

blogs.msdn.com/michael–

Jonathan Kohl–

Antony Marcano–

Brian Marick–Elisabeth Hendrickson
www.testobsessed.com/category/ruminations–

many others J

Today’s “press”(and lots of people own these)www.kohl.ca/blog
www.testingreflections.com
www.exampler.com/blog

37

38

Freedom of the Press•

Trends and Innovations

Open Source Training–•

Black Box Software Testing (Kaner & Bach)Technology and Tools− Open-source tools− Virtualization− Testing in the Cloud–

http://www.testingeducation.org/BBST/
39

40

Open Source Tools

Trends and InnovationsDevelopment•Eclipse

Technology and Tools− Open-source toolsUnit Testing

System TestingxUnit
Cobertura
NCover

FIT
FitNesse
Watir
Selenium− Virtualization− Testing in the Cloud

Performance Testing
OpenSTAJMeter
41

42

7

Virtualization••

Virtualization

Definition•Benefits−Rapid provisioning (30%-50% of testing isenvironmental setup time)−

System state capture (capture/replay in its finestform)−

Reduced costs (purchase, maintenance, space,power)

Vendors include Microsoft, Surgient, VMLogix,
and VMWare
4344

Trends and Innovations•Testing in the Cloud

Technology and Tools•− Open-source tools

Rent as many servers as youlike for as long as you like–

Need a thousand servers forperformance load generation?

–

Need multiple servers, eachwith a different configuration
to simultaneously test yourproduct?

− Virtualization− Testing in the Cloud

45

46

Testing in the Cloud•

Trends and Innovations

Usage rates (US$):–$0.10/hour – small machine–$1.00/hour – extra large machine–$3.10 /canat the hotel•Improvement− TPI− TMMI

47

48

8

TPI•

TPI

Created by Tim Koomen and Martin
Pol based on their experience at theDutch software company IQUIP.

•The TPI model was defined in 1997
and a book was published in Dutch
that same year.

•The English translation was published in 1998.

Key Areas

TestMaturity
MatrixLevels

Checkpoints

Improvement Suggestions

4950

TPITPIKEY AREA•

Twenty key areas cover the testing process:––––––––––

Test strategy
Life-cycle model
Moment of involvement
Estimating and planning
Test specification techniques
Static test techniques
Metrics
Test tools
Test environment
Office environment––––––––––

Test strategy
Life-cycle model
Moment of involvement
Estimating and planning
Test specification techniques
Static test techniques
Metrics
Test tools
Test environment
Office environment
Commitment and motivation
Test functions and training
Scope of methodology
Communication
ReportingDefect management
Testware management
Test process management
Evaluation
Low-level testing

Commitment and motivation
Test functions and training
Scope of methodology
Communication
ReportingDefect management
Testware management
Test process management
Evaluation
Low-level testing012
3
4
Controlled

AA

567
8
9
Efficient

AC

B

C
B

A
B

A

A

11 12 13Optimizing
DA

B

CB
B

CC

B
A
B
B

AC
B

CC
DCBB

A

DCB
A

A
A

DBA

A
A
A

A

10B
BCC

B

DCA

ABB
C

51

52

Trends and Innovations•

TMMiImprovement•The Test Maturity Model Integration has beendeveloped to complement the existing CMMI
framework•It provides a structured presentation ofmaturity levels, allowing for standard TMMi
assessments and certification, enabling aconsistent deployment of the standards andthe collection of industry metrics− TPI− TMMI

5354

9TMMi•TMMI
Level 5: Optimization• Defect Prevention• Test Process Optimization• Quality Control

TMMi has a rapidly growing uptake acrossEurope, Asia and the USA and owes its
popularity to being the only independent test
process measurement method

Level 4: Management & Measurement• Test Measurement• Software Quality Evaluation• Advanced Peer Reviews
Level 3: Defined• Test Organization• Test Training Program• Test Life Cycle and Integration• Non-functional Testing• Peer Reviews

www.tmmifoundation.org

Level 2: Managed• Test Policy and Strategy• Test Planning• Test Monitoring and Control• Test Design and Execution• Test Environment
Level 1: Initial55

Future Innovations?

Trends and Innovations•Process•

From Agile•

Education•

Technology and Tools•

Process Improvement

56?
57

•58

Future Innovations?

Future Innovations?

My personal event horizon is very short“The best way to predict thefuture is to invent it.”– Alan Kaycirca 1969
59

60

10

Thank You•

Thanks for joining with me today. I greatly
appreciate the invitation to speak at JaSST•

If I can be of assistance, or if you’d just like tochat, please contact me atlee@sqe.com•And remember, J keep on innovating
My thanks to:
James Bach, Jon Bach, Rex Black, Bob Galen,
Andy Glover, Dorothy Graham, Cem Kaner,
Harry Robinson, Rob Sabourin, and James
Whittaker

6111