Evaluation methods forcollaborative multi-touch
supportITS 2011 tutorial
G. Coppin1, 2, C. Corré-Nicolas1, J.M. Diverrez1, 2, F.
Legras3
Nov 2011, Kobe
gilles.coppin@telecom-bretagne.eu
1, 2 Telecom Bretagne
2 Lab-STICC, CNRS UMR 3192
3 Deev Interaction

Elements of context• Where do I come from?
• Which research are we doing at our lab?
• Why a tutorial on evaluation of collaborative workon tabletops?

Western coast of FranceResearch Lab• 350 researchers (full time professors, on contract
researchers and PhD students)• 9 departments of research• 4 Affiliations to national research laboratories• Foton (CNRS) for optical communications• Latim (INSERM) for medical information processing• Lab-STICC (CNRS) for communications systems
development and information processing• Marsouin for internet applications and analyses of use• Marconi Prize 2005 (turbocodes)• SPIE Award 2006 (optics)• Inserm Prize 2006 (Medical Imagery)• One representative at the French Académie des sciences

Research lab

ATOL (Aeronautics technico-operational lab)
THALES, Telecom Bretagne, French Navy School

Why a tutorial?
• Achievement of several research projects
contributing to the elicitation of a generic methodof cooperation analysis and evaluation• Method robust and generic enough to be sharedand challenged with specialists (please interact!)

Tutorial
1. Main goals of evaluation
2. Main elements of methodology
3. Tools
4. Cooperation analysis / Use case
5. Results

Main	  goals	  of	  Evalua.on

Maritime patrol example

Maritime patrol example
Consoles individuelles

Vue radarMaritime patrol example
Consoles individuelles

Vue radar

Vue radarMaritime patrol example• Interactive tabletop / multi-users• No explicit constraints on workflows• One expert role by user

FLIRISARESMMaritime patrol example• Interactive tabletop / multi-users• No explicit constraints on workflows• One expert role by user

FLIRISARESM

Where is the problem (ifany)?
Consoles individuelles

Would this one be more appropriate
for the cooperative task ?

How to deal with a newcooperation process or tobenchmark an existing one?

Designing cooperative
applications on TT

Exis+ng	  Applica+on

Transposi+on	  on	  tabletopUsability

Designing cooperative
applications on TT
Exis+ng	  Applica+onTransposi+on	  on	  tabletop

Utilisabilité
ility

UsabSimply transposing an application
on an interactive surface
is obviously not enoughbilityDesigning cooperative
applications on TT
Exis+ng	  Applica+on

Transposi+on	  on	  tabletop

Utilisabilité
ility

Usabbility• Facilitating cooperative work on cooperative work on
interactive tabletop:• while modeling cooperation on the tabletop in order to better
understand the phenomenas• while inferring recommendations for the design of suchinterfaces

Designing cooperative
applications on TT

Exis+ng	  applica+on

Observa+onsExis+ng	  applica+on

Analysis	  of	  collabora+on

Transposingon	  surface

Identification of
behaviours

Usabilitiy
Utilisabilité

bilit

RecommendationsyDesigning cooperative
applications on TT
Coop.	  Applica+on	  design	  on	  tabletop

Exis+ng	  applica+on

Observa+ons

Exis+ng	  applica+on

Analysis	  of	  collabora+on

Transposingon	  surface

Usability

Identification of
behaviours

Usabilitiy
Utilisabilité

bilit

RecommendationsyErgonomics forcooperation• How to caracterize cooperation?
• How to describe cooperative / teambehaviors around a tabletop?

Aiming atdesigning
re-designing
adapting (on-line analysis)•••

Theore.cal	  backgroundCooperation• Cooperation is a task by itself (Klein 2003,Hoc 2001)• Need for understanding and adapting to“cooperation patterns”• Need for identifying “cooperative modes” todiscriminate and assist teams of different kindSurface (Microsoft)

Tabletops andcooperation
DiamondTouch (MERL)

Intuiface (Intuilab)• Large surfaces or multiple surfaces naturally
support cooperative tasks• Cooperation around collocated surfaces isnot as standardized and mediated as most of
CSCW approaches

How to represent
cooperative behaviors(around a tabletop)?

Space• Surface, space and topology
Montferrat 2009

S.Scott 2004

... but is it enough?

Multiple spaces ofrepresentation• Surface, space and topology• Acts of cooperation• Intentions• Tasks• Gestures

Acts of cooperation
EXT-­‐maintainance

EXT-­‐elabora.on

INT-­‐goal-­‐maintainance

INT-­‐goal-­‐elabora.on

INT-­‐plan-­‐maintainance

INT-­‐plan-­‐elabora.on

Maintaining	  representa.on	  of	  external	   Elabora.ng	  representa.on	  of	  external	  elements	  (situa.on,	  context,	  elements	  (situa.on,	  context,	  environment)
environment)

Maintaining	  internal	  representa.on	  of	   Elabora.ng	  internal	  representa.ons	  of	  team	  goals
team	  goals

Maintaining	  internal	  representa.on	  of	  team	  plans

Elabora.ng	  internal	  representa.on	  of	  team	  plans

INT-­‐role-­‐alloc-­‐maintainance

INT-­‐role-­‐alloc-­‐elabora.on

Maintaining	  internal	  representa.on	  on	   Elabora.ng	  internal	  representa.on	  on	  roles	  alloca.on
roles	  alloca.on(Hoc, 2001)

Acts of cooperation
EXT-­‐elabora.on
I check you haveEXT-­‐maintainancethe sameMaintaining	  representa.on	  of	  external	   Elabora.ng	  representa.on	  of	  external	  representationof external
elements	  (situa.on,	  context,	  situationelements	  as I have(situa.on,	  context,	  environment)
environment)
INT-­‐goal-­‐maintainance

INT-­‐goal-­‐elabora.on

INT-­‐plan-­‐maintainance

INT-­‐plan-­‐elabora.on

Maintaining	  internal	  representa.on	  of	   Elabora.ng	  internal	  representa.ons	  of	  team	  goals
team	  goals

Maintaining	  internal	  representa.on	  of	  team	  plans

Elabora.ng	  internal	  representa.on	  of	  team	  plans

INT-­‐role-­‐alloc-­‐maintainance

INT-­‐role-­‐alloc-­‐elabora.on

Maintaining	  internal	  representa.on	  on	   Elabora.ng	  internal	  representa.on	  on	  roles	  alloca.on
roles	  alloca.on

COFOR (Hoc, 2001)

Acts of cooperation
EXT-­‐elabora.on

EXT-­‐maintainance

Maintaining	  representa.on	  of	  external	   Elabora.ng	  representa.on	  of	  external	  elements	  (situa.on,	  context,	  elements	  (situa.on,	  context,	  environment)
environment)
I check you INT-­‐goal-­‐maintainanceshare aINT-­‐goal-­‐elabora.on
common representationof
ourMaintaining	  internal	  representa.on	  of	   Elabora.ng	  internal	  representa.ons	  of	  team goals
team	  goals
team	  goals

INT-­‐plan-­‐maintainance

INT-­‐plan-­‐elabora.on

Maintaining	  internal	  representa.on	  of	  team	  plans

Elabora.ng	  internal	  representa.on	  of	  team	  plans

INT-­‐role-­‐alloc-­‐maintainance

INT-­‐role-­‐alloc-­‐elabora.on

Maintaining	  internal	  representa.on	  on	   Elabora.ng	  internal	  representa.on	  on	  roles	  alloca.on
roles	  alloca.on

COFOR (Hoc, 2001)

Acts of cooperation
EXT-­‐maintainance

EXT-­‐elabora.on

INT-­‐goal-­‐maintainance

INT-­‐goal-­‐elabora.on

Maintaining	  representa.on	  of	  external	   Elabora.ng	  representa.on	  of	  external	  elements	  (situa.on,	  context,	  elements	  (situa.on,	  context,	  environment)
environment)

Maintaining	  internal	  representa.on	  of	   Elabora.ng	  internal	  representa.ons	  of	  team	  goals
team	  goals
I check you shareaINT-­‐plan-­‐maintainancecommon representation
ofinternal	  our representa.on	  of	  Maintaining	  team plans
team	  plans

INT-­‐role-­‐alloc-­‐maintainance

INT-­‐plan-­‐elabora.on

Elabora.ng	  internal	  representa.on	  of	  team	  plans

INT-­‐role-­‐alloc-­‐elabora.on

Maintaining	  internal	  representa.on	  on	   Elabora.ng	  internal	  representa.on	  on	  roles	  alloca.on
roles	  alloca.on

COFOR (Hoc, 2001)

Acts of cooperation
EXT-­‐maintainance

EXT-­‐elabora.on

INT-­‐goal-­‐maintainance

INT-­‐goal-­‐elabora.on

INT-­‐plan-­‐maintainance

INT-­‐plan-­‐elabora.on

Maintaining	  representa.on	  of	  external	   Elabora.ng	  representa.on	  of	  external	  elements	  (situa.on,	  context,	  elements	  (situa.on,	  context,	  environment)
environment)

Maintaining	  internal	  representa.on	  of	   Elabora.ng	  internal	  representa.ons	  of	  team	  goals
team	  goals

Maintaining	  internal	  representa.on	  of	  team	  plans

Elabora.ng	  internal	  representa.on	  of	  team	  plans

I check
you share aINT-­‐role-­‐alloc-­‐maintainancecommonINT-­‐role-­‐alloc-­‐elabora.on
representation
Maintaining	  of ourinternal	  representa.on	  on	   Elabora.ng	  internal	  representa.on	  on	  respective roles in theroles	  alloca.on
roles	  alloca.on
team (or task)

COFOR (Hoc, 2001)

Acts of cooperation
I elaborate andEXT-­‐elabora.on

EXT-­‐maintainanceproposea commonof	  external	  Maintaining	  representa.on	  of	  external	   Elabora.ng	  representa.on	  representation
elements	  (situa.on,	  context,	   of theelements	  (situa.on,	  context,	  external situationenvironment)
environment)

INT-­‐goal-­‐maintainance

INT-­‐goal-­‐elabora.on

INT-­‐plan-­‐maintainance

INT-­‐plan-­‐elabora.on

Maintaining	  internal	  representa.on	  of	   Elabora.ng	  internal	  representa.ons	  of	  team	  goals
team	  goals

Maintaining	  internal	  representa.on	  of	  team	  plans

Elabora.ng	  internal	  representa.on	  of	  team	  plans

INT-­‐role-­‐alloc-­‐maintainance

INT-­‐role-­‐alloc-­‐elabora.on

Maintaining	  internal	  representa.on	  on	   Elabora.ng	  internal	  representa.on	  on	  roles	  alloca.on
roles	  alloca.on

COFOR (Hoc, 2001)

Intentions andcommunication
1

Give suggestions, ideas while respecting others’ ones

2

Give opinions, evaluates, express wishes and feelings

3

Give orientations, information, clarify, elicit, confirm

4

Ask for orientations, informations, look for confirmations

5

Ask for opinions, expect for others’s evaluations to analyse them

6

Ask for ideas and suggestions, directions and possible actions
Bales 72

Intentions andcommunication• Information• Order• Suggestion• Explanation• Attention• Acknowledge(Chéné & al., to appear)Task modelTaskscreatedrag
identifInteraction
model

Logsswitch
kill
Piste 26b

Piste 51

FLIR :ISAR :

createdragcreatedragswitch

dragidentifkillSpace and topology

Gestures as non verbal
communication• Co-verbal (deictic)• Quasi-linguistic• Synchronization• Manipulation(McNeill 2005, Cosnier 96)

Back to our firstexample ...

Taskscreatedrag
identifInteraction
model

Logsswitch
kill
Piste 26
Piste 51

FLIR :ISAR :

createdragcreatedragswitch

dragidentifkill

SpaceIntentionGive
1 suggestions,Give
2 ideaswhileopinions,Giverespecting
3 others’evaluates,orientations,Ask forones
express
4 orientations,information,
Ask forwishes
andclarify,
5Ask
informations,opinions,forelicit,
ideas
feelings
confirm
look
6 expectforandforBack to our firstexample ...
• Extraction of patterns of cooperation• Proposal of recommendations
State of availability of operators
Visual reification ofresponsibilities and actions
done in different spaces
Area dedicated to common
reflection

ToolsObservation methods• Observe to understand ...
• ... Understand to enhance• No intrusion (or as few as possible)• Performance is not always the key

Observation tools
The Observer XT is the professional and user-friendly event
logging software for the collection, analysis, and presentationof observational data.

The Observer
Setup

Observation

Analyze

The Observer XTFirst step: setup 1/4

The Observer
XT
Synchronization of information sources

The Observer XTFirst step: setup 2/4
Coding scheme

Subject
Behaviour
Modifier

The Observer XTFirst step: setup 3/4
SubjectBehaviours

The Observer XTFirst step: setup 4/4
ModifierCoopera.on	  analysis
Use	  Case

Methodology
Our method of evaluation of cooperation relies on theidentification and measurement of activity along 4 main
dimensions involved in cooperative work around tabletops.
From these measures, we can get a representation ofcooperation and determine whether it is efficient andsatisfying. If not, recommendation on system andorganization are proposed.

The 4 dimensions are :Shared mental representation within the teamTeam behavior and cohesiveness
Relevance of intra team verbal exchanges
User experience of cooperation

Presentation of theCollagis use caseCollagis1 project is a research project aiming atdeveloping a generic software frameworksu p p or t i n g an i n t e r ac t i ve dev i c e forcooperative work around geographic data.

• Multiple users• GIS (geographic information system)
1 Collagis was funded by French MOD on the eriod 2009 - 2011Collagis configuration
The application is a serious game simulating a context andprocess of decision making. Two operators cooperate in a
collocated dual screen system to decide about the location of a
new healthcare center, depending on the existing ones andsources of risk (polluting / Seveso factories and sites) present
on the area.

Collagis configuraion
4 cameras (1 camera/operator, 1global, 1 backup)
Acquisition of operators on thescreen (logs)
1 hour experiment
Questionnaire pre (profile) and post(cooperation evaluation)
experimentCollagis functions
Map

Retrieving information
on sites (factories /hospitals)

Collagis functions
Risk propagation area / intervention

LayersCollagis functions
Locating the new center ofhealthcare« In the area from Nice to Marseille, the experts have detected an overpassing number ofdangerous factories when compared with the existing generic or specialized healthcare
centers. It is urgent to build a new one to fill this gap. Your objective is to find the optimal
location while taking into account the existing.»Shared mental
representation
Representation of external situation (situation, context,environment)
Representation of teams goals
Representation of teams plans
Representation of role allocation
Example of elaboration :Shared mental
representation
Maintenance or elaboration
Maintenance of common ground is defined as aunique communication followed by anacknowledge. Transmitted information is inaffordance with receiver’s mental model andhe/she integrates it immediately.
While elaboration is a sequence of alternatedtalks with surprises and conflicts between
operators, that ends with a final agreement(Hoc 2001 - acts of cooperation categories)

Loops and common
groundWhen coding dialog between
operators, we acquire start timeand end time of informationexchanges. This corresponds to a
maintenance or elaboration loop.

Otherwise we consider the loop is
closed and this is a success(Chauvin & al. 2010)

If there is no observation of anending loop, it remains open and it
is considered as a failure in setting
a common ground between thetwo operators.
Example of “open loop”

Loops duration
We measure the duration of closed maintenance loops orelaboration (success loops). In this case, we consider thatthe shorter the loops will be the more efficient they will
be for common ground setting. Operators reach thetarget with a minimal effort and a minimal duration (ISO9241-11)

Common groundmaintenance
Giving acknowledges indicates
maintenance of the common ground. It
avoids all misunderstanding during
i n fo r m a t i o n e x c h a n g e s b e t we e noperators. On the contrary, asking for
acknowledge is a sign of the emitter
doubting about the sharing of his/her
representation with his/her receiver.
Consequently, common ground can besupposed to be degraded (Chauvin & al.
2010)

Team behavior andanticipation
Two behavioral profiles can beidentified: reactive behaviors andproactive behaviors. A proactive
behavior supposes a good knowledgeof operators about the task to beachieved, the work organization andteam and mutual competencies. This
allows to bring the necessary andrelevant information to the otheroperator before he/she asks for.

Team behavior andanticipation
With a reactive behavior, an
operator asks to his/her partner
information he/she needs to achieve
the task. This implies an extra delay
in task achievement and/or a
cognitive overload.

Hierarchical links andoperators roles
To check about the actual hierarchical links between operators,
we rely on several criteria that are coded within Noldus:
emission or requests for orders, emission or requests fosuggestion, frequency of intervention and decision making.
We thus evaluate hierarchical links thanks to the orders andsuggestions. Emission of orders is a direct sign of authority andleadership. Asking for directions and orders represents theopposite.

Hierarchical links andoperators roles (II)Suggestions gather communications related to opinions,recommendations and advices. Emitting suggestions is considered as asupportive behavior, while asking for suggestion reveals doubts and needfor help on the emitter side (Chauvin & al. 2010)

Example of emission of orderAccuracy of exchanges
We gather in explanations (requested or given) all
communication acts such as clarification, tuning, repetititon,correction and justification. We consider explanations to becorrelated to the level of soudness and accuracy of exchanges.
Requests for explanation express a problem of mutual
understanding between operators (Chauvin et al, 2010).
ExempleAttention and focus
Need for (or lack of) collaboration / cooperation can be detected
through requests for attention. We suppose that without such
requests, attention can be considered as satisfying (use of eyetrakers information should allow to refine this approach) (Chauvinet al, 2010).

Gesture and gaze
We focus on gaze and pointing gestures (Ekman et Friesen (1969),McNeill (1992)). Gazes can be considered as regulating gestures / acts.
Gazes and gestures (e.g. nodding) accompany the interaction (asmumblng can accompany dialog).
Pointing gestures are deictic and indicate somehing in the environment.

Content of exchanges

The set of concepts addressed by the exchanges between operators isgathered and analyzed. This allows to deal with the “What?”. These concepts
may be analyzed while making reference to the domain knowledge of experts
or attached to the current task.

To summarize
CommongroundLoops
management

Need forcooperation
Attention andfocus

Nonverbalregulation
Gazes
Pointing

User experience
Cooperationquestionnaire

Cooperation
AnalysisTeam
cohesiveness
Proactionand respect ofroles

Relevance
Content analysis

Post experiment group
evaluation
Questionnaire about:• Information sharing• Coordination amongst actors• Group communication• Individual engagement• Team cohesion(qualita.ve)	  resultsTime of scenario
execution
Time ofscenario execution

Cooperative “mode”
Frequency of elaboration andmaintenance acts

LoopsMean time ofclosed loops

Number of open and closed loops

Group 2

Group 4

Group 5

8,6

9,0

15,0

Mean time ofmaintenance closed loops

Orders, suggestions,precisionsuggestion

precisionordersGaze and gesturenature of pointing
gaze and pointing

nature of gaze

Group profiles
group 2group 4group5anticip.

proactive

proactive

proactive

hierarc.

no link / factory
decideno link / factory
decideno link / healthdecideprecision

good level ofunderstanding

averagealteredneed collab.

satisfying

satisfying

satisfying

gesturesfewfewgazes >> point.

User experience
Sharing information

No cross marker!!

User experience
Coordination

No structure forthe project
Only oral
exchanges onstate of progress

User experience
Communication

No space dedicated tochat!

Usr experience
Cohesiveness and Engagement• Nothing to declareThanks for attention!