AI-DRIVEN TESTINGKNA NEW ERA OF TEST AUTOMATIONIGMAR272019AGENDA
ABOUT ME
MOTIVATION: REDEFINING TEST AUTOMATIONBACKGROUND: AI, MACHINE LEARNING, TESTING
AI-DRIVEN TESTING: A NEW ERA OF TEST AUTOMATIONCONCLUSION
2ELLENCE
HEAD, QUALITY CENTER OF EXCER SCIENCEADJUNCT PROFESSOR, COMPUT
33AUTOMATIONMACHINES WORKING WITHOUT DIRECT HUMAN CONTROL

au·to·ma·tion/ˌôdəˈmāSH(ə)n/ noun

41.

Capable of starting, operating, and completing independently.

2.

Use of various control systems for operating machinery andequipment with minimal human intervention.

5678

Artificial Intelligence for Software Testing (AIST)
is an emerging field aimed at the developmentof AI systems to test software, methods to test
AI systems, and ultimately designing softwarethat is capable of self-testing and self-healing.

9AI-DRIVENTESTINGMETHODSTOOLSDESIGNS10

SELF-TESTINGSYSTEMSTESTING AISYSTEMS

AI-DRIVENTESTINGMETHODSTOOLSDESIGNS1112

SELF-TESTINGSYSTEMSTESTING AISYSTEMSartificial intelligence:

13141: a branch of computer science dealing with thesimulation of intelligent behavior in computers

2: the capability of a machine to imitate intelligent
human behavior

INTELLIGENT AGENTAN AUTONOMOUS ENTITY THAT CAN…

Perceive and act in its environment using sensors
and actuators, respectively.
Direct its activity toward goals and use knowledge
and learning in achieving those goals.
Analyze itself in terms of behavior, error, and successand adapt online, possibly in real time.
Be organized with other agents, typically in a hierarchy
to tackle problems at different abstraction levels.
Typically referred to as bots or AI.
1516

machine learning:

the science of getting computers to act without being explicitly programmed

17

If a program can improve how it performs a certain
task based on previous experience then you cansay the program has learnt.

18

If the performance P of a computer program atcompleting a task T improves with experience E,
the program is said to learn from experience E.

19

Input Layer

Hidden Layers

Output Layer

REWARDPUNISH20

software testing:

dynamic program verification using finite tests from an infinite execution domain

21

Program(Function)

Input22Output

Domain(Input)Program(Function)

Range(Output)

OK (d)df(d)¬OK (d)Is f (d) acceptable?
23

Input Layer

Hidden Layers

Output Layer

REWARDPUNISH24

Program

DomainRange(Function)(Input)(Output)

T D

f(t1)

t1

f(t2)

t2

f(t3)

t5

t3

OK (t1)
OK (t2)
OK (t3)OK (t4)
OK (t5)

f(t4)

t4

f(t5)

T is successful iff∀t ∈ $ %&(t)25Program

DomainRange(Function)(Input)(Output)

T D

f(d1)
f(d8)

f(d2)

t1
t2

f(d7)
f(d3)

t5t3

f(d4)

t4

f(d6)

f(d5)

T is an ideal test iff

26∀t ∈ $ %&(t) → ∀d ∈ + %&(d)

f(d9)
f(d∞)

OK (d1)
OK (d2)
OK (d3)OK (d4)
OK (d5)
OK (d6)
OK (d7)
OK (d8)OK (d9):OK (d∞)Complexity / Coverage

SoftwareComplexityCoverage
Gap
Conventional
Test Coverage

Time
Source: AI for Software Testing, Jason Arbon, CEO, test.ai
https:/www.linkedin.com/pulse/ai-software-testing-jason-arbon

27

Complexity / Coverage

SoftwareComplexityAI-Driven
Test Coverage
Conventional
Test Coverage

Time28Source: AI for Software Testing, Jason Arbon, CEO, test.ai
https:/www.linkedin.com/pulse/ai-software-testing-jason-arbon

29

AI-DRIVEN TESTING
FAD, FICTION, OR FUTURE?

testrigor
30

AI-DRIVEN TESTING
A NEW ERA OF TEST AUTOMATIONAPPROACH

HOW BOTS LEARN TO TEST SOFTWARE

MODEL

INTERACT

Keep track of the applicationand environment state tohelp deal with uncertainty
and partial observability.

Generate actions to navigate
through the application UI,
input data, and verify
observable outcomes.
AIDT

EXPLORE
Perceive the environment
including the application UI
structure, behavior and any
differences over builds.

32

LEARN
Direct feedback on the qualityof an agents actions to help
improve future actions.

APPROACH

HOW BOTS LEARN TO TEST SOFTWARE

cart

home

PINCH

TAP…

ML

PunishRewardRewriteReinforce

33

ARCHITECTURE

DEMYSTIFYING AI-DRIVEN TESTING BOTS

AI-DRIVEN TEST BOT
Analyze

Plan

MonitorExecuteKnowledge
Machine Learning

34

MAPE + KML
SEEDED BASE KNOWLEDGEACQUIRES NEW KNOWLEDGE
MAINTAINS EXISTING KNOWLEDGESHARES ITS KNOWLEDGE

TEST AGENT<<Coordinator>>

TEST RESULTS AND HISTORY

Analyze

AI-GENERATED TESTSPlanMonitorX

ExecuteX

DEDUPE AND DISTRIBUTE

EXPLORERUN GENERATEDAND GENERATETEST

EXPLORE AND GENERATEEXPLORERUN GENERATEDAND GENERATETESTEXECUTIONQUEUE

EXECUTIONQUEUE

EXECUTION
QUEUE

35PROTOTYPES

AI-GENERATION AND EXPLORATION IN TEST

http://github.com/UltimateSoftware/AGENT

36/AGENT-X

3738

39

FROM TEXT GENERATION TO TEST GENERATIONA LESSON LEARNED FROM ALICE IN WONDERLAND~150,000 Characters
Total in the book

47 Distinct Characters
When converted to lowercase

100-Length Subsequence Training
N-Length Subsequence → Character Output

Example (N=5)
CHAPT → EHAPTE → R40be no mistake about it: it wasneither more nor less than a pig,and she felt that it would be quite aelin that she was a little want oetoiet ano a grtpersent to the tas alittle war th tee the tase oa teettee thehad been tinhgtt a little toiee at thecadl in a long tuiee aedun thet sheer
was a little tare gereen to be a gentleof the tabdit soenee the gad ouw ie
the tay a tirt of toiet at the was alittle anonersen, and thiu had beenwoite io a lott of tueh a tiie andtaede bot her aeain she cere thth thebene tith the tere bane to tee toaete to
tee the harter was a little tire theherself lying on the bank, with herhead in the lap of her sister, who was
gently brushing away so siee, andshe sabbit said to herself and thesabbit said to herself and the sood
way of the was a little that she wasa little lad good to the garden, andthe sood of the mock turtle said toherself, 'it was a little that the mockturtle said to see it said to sea it
said to sea it say it the marge hard
sat hn a little that she was so
sereated to herself, and she sabbit
said to herself, 'it was a little little
shated of the sooe of the coomouse it
was a little lad good to the little

same oare cade an anl ano the garee
and the was so seat the was a little
gareen and the sabdit, and thewhite rabbit wese tilel an the caoe
and the sabbit se teeteer, and thewhite rabbit wese tilel an the cade ina lonk tfne the sabdi ano aroing totea the was sf teet whitg the was alittle tane oo thete the sabeit she
was a little tartig to the tar tf tee
the tame of the cagd, and the whiterabbit was a little toiee to be anle
tite thete ofs and the tabdit was thewiite rabbit, andgooder head. and said to herself, ‘it
was a little little shated of the mouseof the good of the courte, and it wasa little little shated in a little thatthe was a little little shated of thethmee said to see it was a little
book of the was a little that she wasso sereated to hare a little the begansitee of the was of the was a little
that she was so seally and thesabbit was a little lad good to thelittle gooder head of the gad searedto see it was a little lad good to thelittle good

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERS

ABSTRACT TESTLANGUAGE

43

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERSFirst Name*

Last Name*Company

Observe Required TextBox FirstName

44

Input BLANK FirstName Invoke Submit

A Model-Based AI-Driven Test Generation System, Dionny Santiago.
Masters Thesis Proposal, Florida International University (2018).

Observe ErrorMessage

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERSFirst Name*

Last Name*

Company

Observe Required TextBox FirstName

Input BLANK FirstName Invoke Submit

Observe ErrorMessage

IF

Perceived Conditions
A Model-Based AI-Driven Test Generation System, Dionny Santiago.
Masters Thesis Proposal, Florida International University (2018).

45

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERSFirst Name*

Last Name*

Company

Observe Required TextBox FirstName

Input BLANK FirstName Invoke Submit

IF

WHEN

Perceived Conditions

46

Input Actions

A Model-Based AI-Driven Test Generation System, Dionny Santiago.
Masters Thesis Proposal, Florida International University (2018).

Observe ErrorMessage

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERS
Last Name* Please enter last name.

First Name* Please enter first name.

Company

Observe Required TextBox FirstName

Input BLANK FirstName Invoke Submit

IF

WHEN

Perceived Conditions

Input Actions

Observe ErrorMessage

THEN

Required Observations

A Model-Based AI-Driven Test Generation System, Dionny Santiago.
Masters Thesis Proposal, Florida International University (2018).

47

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERS

48

Input SentencePredicted Next WordObserve

RequiredObserve Required

TextboxObserve Required TextboxFirstName

Observe Required Textbox FirstName

InputObserve Required Textbox FirstName Try

VALID

Observe Required TextBox FirstName Try VALID FirstName

InvokeObserve Required TextBox FirstName Try VALID FirstName ClickSubmit

Observe Required TextBox FirstName Try VALID FirstName Click Commit

NotObserve

Observe Required Textbox FirstName Try VALID FirstName Click Commit NotObserve

ErrorMessage

AI-DRIVEN TEST GENERATION
KEY ADVANTAGES

50

GENERAL PURPOSEPESTICIDE FREE

LARGE SCALE

Different Testing Types
Any Application or Domain

Generates New Tests Each TimeEliminates Pesticide Paradox

Deep Learning
Cloud Infrastructure

LIMITATIONS?
51

INPUTPROGRAMOUTPUTINPUT

MACHINE LEARNINGOUTPUT(Domain)(Function)(Range)53(Domain)

54(Function)(Range)SELF-TESTING
AI-DRIVEN TESTING FROM THE INSIDE-OUT

55

AUTONOMIC SELF-TESTINGTESTING DYNAMICALLY ADAPTIVE SYSTEMS

Autonomic Test Manager

SelfConfigure

Analyze

SelfHeal
SelfTest

SelfOptimize

PlanMonitor

ExecuteKnowledge

SelfProtect(Brain)Software Component

56

A Self-Testing Approach for Autonomic Software, Tariq M. King.
ProQuest Electronic Theses and Dissertations (2009).

AI-DRIVENTESTINGMETHODSTOOLSTESTING AISYSTEMSDESIGNSSELF-TESTINGSYSTEMS

57

AI-DRIVENTESTINGMETHODSTOOLSDESIGNS58

SELF-TESTINGSYSTEMSTESTING AISYSTEMSTHANK YOU
ACKNOWLEDGEMENTS

DIONNY SANTIAGODAVID ADAMOJUSTIN PHILLIPSPATRICK ALT
KEITH BRIGGSNICOLETTE CELLIPETER J. CLARKEJASON ARBON
59

REFERENCES
1. Test Machina: Demystifying AI-Driven Testing Bots, Tariq M. King, Automation Guild, Feb 2019.
2. AI-Driven Testing: A New Era of Test Automation, Tariq M. King, CIO Review Software Testing Special, Dec 2018.
3. A Model-Based AI-Driven Test Generation System, Dionny Santiago, ProQuest Electronic Theses andDissertations, Dec 2018.
4. Abstract Flow Learning for Web Application Test Generation, Dionny Santiago, Peter J. Clarke, Patrick Alt, andTariq M. King, European Software Engineering Conference/Foundations of Software Engineering International
Workshop on Automating TEST Case Design, Selection, and Evaluation, Nov 2018.
5. AI and Machine Learning Skills for the Testing World, Tariq M. King and Jason Arbon, STAREAST, May 2018.
6. Text Generation with LSTM Recurrent Neural Networks in Python with Keras, Jason Brownlee, Aug 2016.
7. A Self-Testing Approach to Autonomic Software, Tariq M. King, ProQuest Electronic Theses and Dissertations,Jul 2009.
8. Towards a Theory of Test Data Selection, John B. Goodenough and Susan L. Gerhart, International Conference on
Reliable Software, Jun 1975.
60