Visualizations to Support
Interactive Goal Model Analysis
Jennifer Horkoff1
Eric Yu2
Department of Computer Science1
Faculty of Information2
j h k
jenhork@cs.utoronto.ca

yu@ischool.utoronto.cai h l

University of Toronto

September 28, 2010
REV’10
REV’Challenges in Goal Models
 Goal Models can be used as part of an RE processCapturep
and visuallyy present:p
Stakeholders
 Their needs (goals)
 Relationships between goals, and Stakeholder interdependencies
Show alternative ways to achieve goals (design alternatives) Analyze the effects of design alternatives
 However…

Goal models are often complex and difficult to read
 Challenging to following the reasoning through complex paths inthe model Example:

inflo Case Study

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

2

Example: inflo Case Study

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

3

Challenges
g in Interactive Goal Model Analysisy Interactive goal model analysis:

Used to supplementpp
the incompletepnature of the model withstakeholder domain knowledge
 Encourages stakeholder participation in modeling and evaluation
 Recent studies testing the utility of such evaluation

revealed several usability issues (PoEM’10):
Users have difficulty choosing starting points for analysis (initialvalues)l ) Users have difficulty understanding conflicts in analysis results These difficulties can be alleviated with visualization

techniques

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

4

Background: Goal Models


We use i* as an example goal modeling framework

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

5Interactive Forward Satisfaction AnalysisyA question/scenario/alternative isplaced on the model and its affects
are propagatedd “forward”“fd” throughhhmodel links
 Interactive: user input (humanj djudgment)t) is
i usedd tto decided id onpartial or conflicting evidence“What is the resulting value?” Publications:
P bli ti

CAiSE’09 (short paper)
PoEM’09,
IJISMD

What if…?

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

6Interactive Forward Satisfaction AnalysisyA question/scenario/alternative isplaced on the model and its affects
are propagatedd “forward”“fd” throughhhmodel links
 Interactive: user input (humanj djudgment)t) is
i usedd tto decided id onpartial or conflicting evidence“What is the resulting value?” Publications:
P bli tiHuman JJudgment
g

CAiSE’09 (short paper)
PoEM’09,
IJISMD

What if…?

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

7

Interactive Forward Satisfaction AnalysisyA question/scenario/alternative isplaced on the model and its affects
are propagatedd “forward”“fd” throughhhmodel links
 Interactive: user input (humanj djudgment)t) is
i usedd tto decided id onpartial or conflicting evidence“What is the resulting value?” Publications:
P bli tiHuman Judgment

CAiSE’09 (short paper)
PoEM’09,
IJISMD

What if…?

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

8Interactive Forward Satisfaction AnalysisyA question/scenario/alternative isplaced on the model and its affects
are propagatedd “forward”“fd” throughhhmodel links
 Interactive: user input (humanj djudgment)t) is
i usedd tto decided id onpartial or conflicting evidence“What is the resulting value?” Publications:
P bli ti

CAiSE’09 (short paper)
PoEM’09,
IJISMD

What if…?

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

9Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

10Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links

Human Judgment

Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

11

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

12

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver




H
Human
JJudgmentdtPublications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

13

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Conflict

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

14

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

B kt kBacktrack

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

15

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links

Backtrack

Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

16

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

17

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver




H
Human
JJudgmentdtPublications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

18

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

19

Interactive Backward Satisfaction Analysisy

A question/scenario/constraints
are placed on the model and its
affectsff
are propagatedd “b“backward”
kd”
through model links
Asks for human judgment “Whatincoming values could produce thetarget value?” Model is iteratively encoded in
CNF form and ppassed to a SAT
solver

Publications:istar’08,
ER’10

Is this possible…?
Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

20

Qualitative Studies of Interactive Goal Model Analysis


11 studies were conducted in previous work to the utility of interactive
goal model analysis
Ten studies had individuals analyzing models for up to 2 hours
 1 study involved a group implementing the inflo “back‐of‐the‐envelope”
calculation modeling tool Results described in PoEM
PoEM’1010Observations pointed to several analysis usability issues
In this work we focus on two issues: initial value selection and conflictcomprehension

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

21Issue #1

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

22

Issue: Starting Points for Analysis
Users had difficulties knowing how or where to start analysis
When given an analysis question: problems finding intentions to placei i i l labels
initiall b l
 When coming up with their own analysis question: difficulties knowing
how to start analysis
S
Suggestedt d analysis
l i methodology
th d lStart forward analysis by identifying leaf intentions
 Start backward analysis by identifying root intentions
 All participants
i i
using
i the
h methodology
h d l
h
had
d diffidifficulties
l i fi
finding
di roots orleaves i* models are not like regular tree‐shaped graphs:
 Some links do not have an obvious direction
 Easy to ignore links across actor boundaries
 Cycles leads to non‐conventional layout

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

23

Challenge:g Where are the Leaves and Roots?
Example from individual study: conference sustainability PC andPublicity Chair Leaf: an intention that has no “incoming” links
 Root: an intention that has no “outgoing” links


Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

24

Visual Intervention: Automatic Leaf and Rootg ggIntention Highlighting
OpenOME implementation has “Mark Model Leaves” (green) or“Mark Model Roots” (blue) options

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

25

Issue #2

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

26

Issue: Understanding Conflicts
Conflict: the case where the SAT solver used in the backward
analysis procedure cannot find a solution over a CNF model
encodingdiFor one or more intentions, i, both v(i) and not v(i) hold, where v
is an analysis value, e.g. S(i) and not S(i)“Conflict” in goal modeling is an overloaded term

There is a conflict label, meaning roughly equal amounts ofpositive and negative evidence

Two alternatives can “conflict” in relation to one goal

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

27

Issue: Understanding Conflicts

We can use the “UNSAT core” to help understand conflicts
UNSAT core: an unsatisfiable subset of clauses in a CNF
representing
i theh modeld l
 Helps to isolate the intentions involved in the conflict Previously, we presented the UNSAT core to the user in text form

Users hU
hadd difficultydiffi l understandingddi theh conflict
fli andd relatingl i theh
UNSAT core to the model

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

28

Challenge:g Where are the Conflicts?

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

29

Challenge:g Where are the Conflicts?
The following intention clauses are conflicting:not PS(Simple functionality) OR PD(Type checking
for consistency)
not PS(Flexibility) OR PD(Type checking andconversion)
not PS(use inflo) OR PS(Graphing)
not PS(Graphing) OR PS(Be inflo)
not PS(Be inflo) OR PS(Create graphs)
not PS(Type checking and conversion) OR PS(Nodecreated automatically)not PS(Type checking and conversion) OR PS(Define
types)
not PD(Type checking and conversion) OR PD(Nodecreated automatically) OR PD(Define types)
not PS(Create graphs) OR PS(Dimensional analysis)
not PS(Dimensional analysis) OR PS(Type checking
for consistenc
consistency)) OR PS(T
PS(Typepe checking andconversion)
S(use inflo)
PS(Usability for graph creation)
not S(use inflo) OR PS(use inflo)
not PD(Define types) OR not PS(Define types)
not PD(Type(checking for
f consistency)) O
OR notPS(Type checking for consistency)
not PD(Node created automatically) OR not PS(Nodecreated automatically)not PS(Usability for graph creation) OR PS(Simplicity
of inflo)
not PS(Usability for graph creation) OR PS(Flexibility)
not PS(Simplicity of inflo) OR PS(Simply functionality)

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

30

Visual Intervention: Conflict Highlightingg ggAutomatically find all intentions involved in clauses in the UNSAT
core




Find the “logical sources of the conflict”, i.e. the intentions for whichv(i) is true and not trueHighlight intentions orange in the model

Highlight intentions red in the model

Users are presented with a list of intentions involved in the conflictThe assigned analysis value in the conflicting situation is displayed

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

31

Visual Intervention: Conflict Highlighting

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

32

Visual Intervention: Conflict Highlighting

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

33

Visual Intervention: Conflict Highlighting

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

34

Visualizations Applied
Five follow‐up studies tested new visualizations
 Participants were asked about the new interventions:Do the leaves/roots highlighted in the model make sense?
 Can you understand why there is a conflict?


Q
Qualitative
analysisy of results (no statistical significance)g Several threats to validity described in paper

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

35

Results
Leaf and Root Intention Highlighting: A few roots and leaves were initially surprising, but upon examination
the links became clear Difference between global vs. local leaves and roots became apparent
Algorithm finds global leaves and roots Participantspoften focused on local leaves and roots ‐> Future improvements could highlight both types


Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

36

Results
Conflict Highlighting: A considerable amount of knowledge of i* and the analysis procedure wasneeded to understand the causes of the conflict
 Highlighting helped the study facilitators explain the conflicts toparticipants‐> i* analysis
y experiencepor the presencepof an experiencedpfacilitator isneeded to gain the full benefits of interactive analysis
 Similar conclusions as in the initial studies (PoEM’10)


Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

37

Conclusions and Future Work

Although goal model analysis can be helpful, comprehensiondifficulties exist
Applied two visual interventions to help alleviate these difficulties

Future work should look for additional visual enhancements formodel analysis:Highlighting areas needing human judgment (done) Highlighting areas affected by changes in the model (in progress)

Future work could take an action research approachCase studies in industrial settings

Visualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

38

Thank you jenhork@cs.utoronto.ca
 www.cs.utoronto.ca/~jenhork
www cs utoronto ca/ jenhork yu@ischool.utoronto.ca@i h l tt www.cs.utoronto.ca/~eric OpenOME:
 https://se.cs.toronto.edu/trac/omehdVisualizations to Support Interactive Goal Model Analysis ‐ Horkoff, Yu

39