-----------------------------------------------------------------------

  -----------------------------------------------------------------------

+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
| Methodology for\                 |   |                                  |
| Evaluation of Collaboration      |   |                                  |
| Systems                          |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
| Version 1                        |   |                                  |
+----------------------------------+---+----------------------------------+
| John Cugini                      |   |                                  |
|                                  |   |                                  |
| Laurie Damianos                  |   |                                  |
|                                  |   |                                  |
| Lynette Hirschman                |   |                                  |
|                                  |   |                                  |
| Robyn Kozierok                   |   |                                  |
|                                  |   |                                  |
| Jeff Kurtz                       |   |                                  |
|                                  |   |                                  |
| Sharon Laskowski                 |   |                                  |
|                                  |   |                                  |
| Jean Scholtz                     |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
| *The Evaluation Working Group    |   |                                  |
| of\                              |   |                                  |
| The DARPA Intelligent            |   |                                  |
| Collaboration and Visualization  |   |                                  |
| Program\                         |   |                                  |
| Revision 4.0: 12 April 1999\     |   |                                  |
| Submit or view comments at\      |   |                                  |
| http://www.ant                   |   |                                  |
| d.nist.gov/\~icv-ewg/forum.html* |   |                                  |
+----------------------------------+---+----------------------------------+
| Version 4.0, Revised July 15,    |   |                                  |
| 1999                             |   |                                  |
|                                  |   |                                  |
| Jill Drury                       |   |                                  |
|                                  |   |                                  |
| Laurie Damianos                  |   |                                  |
|                                  |   |                                  |
| Tari Fanderclai                  |   |                                  |
|                                  |   |                                  |
| Jeff Kurtz                       |   |                                  |
|                                  |   |                                  |
| Lynette Hirschman                |   |                                  |
|                                  |   |                                  |
| Frank Linton                     |   |                                  |
|                                  |   |                                  |
| *The MITRE Corporation, Bedford, |   |                                  |
| MA*                              |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+
|                                  |   |                                  |
+----------------------------------+---+----------------------------------+

# Table of Contents {#table-of-contents .Front-Matter-Header}

Section Page

1\.

Introduction [1](#introduction)

1.1 Background: The Defense Advanced Research Project Agency (DARPA)
Intelligent Collaboration and Visualization (IC&V) Program
[1](#number-range-level2number-range-level3number-range-level4-background-the-defense-advanced-research-project-agency-darpa-intelligent-collaboration-and-visualization-icv-program)

1.2 Background: The Evaluation Working Group and Its Aims
[2](#number-range-level2number-range-level3number-range-level4-background-the-evaluation-working-group-and-its-aims)

1.3 The Scope and Structure of this Document
[3](#number-range-level2number-range-level3number-range-level4-the-scope-and-structure-of-this-document)

2\. Context and Definitions [5](#context-and-definitions)

2.1 Introduction
[5](#number-range-level2number-range-level3number-range-level4-introduction)

2.2 Methods of Interface Evaluation
[5](#number-range-level2number-range-level3number-range-level4-methods-of-interface-evaluation)

2.3 Use of Scenarios
[6](#number-range-level2number-range-level3number-range-level4-use-of-scenarios)

2.4 Definition of Terms
[7](#number-range-level2number-range-level3number-range-level4-definition-of-terms)

2.4.1 Collaborative Environments
[7](#number-range-level3number-range-level4-collaborative-environments)

2.4.2 Tasks or Collaborative Activities
[8](#number-range-level3number-range-level4-tasks-or-collaborative-activities)

2.4.3 Scenarios [8](#number-range-level3number-range-level4-scenarios)

3\. A Framework for Collaborative Systems
[9](#a-framework-for-collaborative-systems)

3.1 Introduction
[9](#number-range-level2number-range-level3number-range-level4-introduction-1)

3.2 Overview of the Collaborative Framework
[9](#number-range-level2number-range-level3number-range-level4-overview-of-the-collaborative-framework)

3.3 Using the Framework to Begin Evaluating a CSCW System
[11](#number-range-level2number-range-level3number-range-level4-using-the-framework-to-begin-evaluating-a-cscw-system)

3.3.1 General Evaluation Approaches Using the Framework
[11](#number-range-level3number-range-level4-general-evaluation-approaches-using-the-framework)

3.3.2 Using the Framework at Each Level
[12](#number-range-level3number-range-level4-using-the-framework-at-each-level)

3.4 Detailed Framework Description
[12](#number-range-level2number-range-level3number-range-level4-detailed-framework-description)

3.4.1 Requirement Level
[12](#number-range-level3number-range-level4-requirement-level)

3.4.1.1 Work Tasks [12](#number-range-level4-work-tasks)

3.4.1.2 Transition Tasks [13](#number-range-level4-transition-tasks)

3.4.1.3 Social Protocols [13](#number-range-level4-social-protocols)

3.4.1.4 Group Characteristics
[15](#number-range-level4-group-characteristics)

3.4.1.5 Requirements Level Measures
[16](#number-range-level4-requirements-level-measures)

3.4.2 Capability Level
[16](#number-range-level3number-range-level4-capability-level)

3.4.3 Service Level
[20](#number-range-level3number-range-level4-service-level)

3.4.4 Technology Level
[21](#number-range-level3number-range-level4-technology-level)

3.5 Collaborative Tasks
[22](#number-range-level2number-range-level3number-range-level4-collaborative-tasks)

3.6 Summary of Collaborative Framework
[23](#number-range-level2number-range-level3number-range-level4-summary-of-collaborative-framework)

4\. Scenarios for Evaluation of Collaboration Tools
[31](#scenarios-for-evaluation-of-collaboration-tools)

4.1 Introduction
[31](#number-range-level2number-range-level3number-range-level4-introduction-2)

4.2 Constructing Scenarios
[32](#number-range-level2number-range-level3number-range-level4-constructing-scenarios)

4.3 Choosing Scenarios
[33](#number-range-level2number-range-level3number-range-level4-choosing-scenarios)

4.4 Using Scenarios for Evaluation
[33](#number-range-level2number-range-level3number-range-level4-using-scenarios-for-evaluation)

4.4.1 Using Scenarios to Iteratively Evaluate a Single System
[33](#number-range-level3number-range-level4-using-scenarios-to-iteratively-evaluate-a-single-system)

4.4.2 Using Scenarios to Evaluate a System's Appropriateness for Your
Requirements
[34](#number-range-level3number-range-level4-using-scenarios-to-evaluate-a-systems-appropriateness-for-your-requirements)

4.4.3 Using Scenarios to Compare Systems
[34](#number-range-level3number-range-level4-using-scenarios-to-compare-systems)

5\. Metrics and Measures [35](#metrics-and-measures)

5.1 Introduction
[35](#number-range-level2number-range-level3number-range-level4-introduction-3)

5.2 Metrics
[37](#number-range-level2number-range-level3number-range-level4-metrics)

5.2.1 Data Collection Methods
[42](#number-range-level3number-range-level4-data-collection-methods)

5.3 Measures
[43](#number-range-level2number-range-level3number-range-level4-measures)

5.4 Data Collection Methods: Logging
[48](#number-range-level2number-range-level3number-range-level4-data-collection-methods-logging)

6\. Using the Methodology to Design an Experiment
[51](#using-the-methodology-to-design-an-experiment)

6.1 Introduction
[51](#number-range-level2number-range-level3number-range-level4-introduction-4)

6.2 Designing an Experiment
[51](#number-range-level2number-range-level3number-range-level4-designing-an-experiment)

6.3 An Example: The Map Navigation Experiment
[53](#number-range-level2number-range-level3number-range-level4-an-example-the-map-navigation-experiment)

# Section 1Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-1number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading}

# Introduction {#introduction .Section-Header}

This document outlines a two-part methodology for evaluating
collaborative computing systems. In the first part of the methodology,
the researcher uses our framework to analyze and describe a given
collaborative computing system in terms that reveal the capabilities of
the system and allow preliminary judgments about the kinds of work tasks
the system might best support. In the second part, the researcher asks
subjects to use the system being evaluated to run scenarios representing
the kinds of work tasks highlighted by the initial analysis, and/or the
kinds of work tasks a group of potential users need support for. These
scenarios help the researcher evaluate how well the system actually
supports the work tasks in question.

This methodology was developed to provide a reliable but inexpensive
means of evaluating collaborative software tools. At a relatively low
cost, researchers in the collaborative computing research community can
evaluate their own or others' collaborative tools, and user groups can
determine their requirements and ascertain how well a given
collaborative system supports their work.

## 1.1Number range Level2Number range Level3Number range Level4 Background: The Defense Advanced Research Project Agency (DARPA) Intelligent Collaboration and Visualization (IC&V) Program

The DARPA Intelligent Collaboration and Visualization program (IC&V) has
the goal of developing the generation-after-next collaboration
middleware and tools that will enable military components and joint
staff groups to enhance the effectiveness of collaborations by:

-   Gathering collaborators together across time and space for rapid
    response in time-critical situations

-   Bringing appropriate information resources together across time and
    space within the context of a task

The IC&V program has funded a number of groups to develop collaborative
technologies to address these objectives; it has also devoted funds to
establishing evaluation metrics, methodologies and tools. The IC&V
program objectives are:

1.  Enable access to collaborative systems via diverse portals, from
    hand-held through room-sized.

2.  Enable interoperability across systems using diverse encoding
    formats, coordination and consistency protocols, and real-time
    services.

3.  Scale collaborations to 10 active contributors, 100 questioners, and
    1000 observers.

4.  Reduce by an order of magnitude the time needed to generate
    collaborative applications.

5.  Enable real-time discovery of relevant collaborators and information
    within task context.

6.  Reduce by an order of magnitude the time required to establish
    collaborative sessions across heterogeneous environments.

7.  Reduce by an order of magnitude the time required to review
    collaborative sessions.

8.  Improve task-based performance of collaborators by two orders of
    magnitude.

The effectiveness of the overall IC&V program will be evaluated with
respect to these high-level objectives. The Evaluation Working Group
(EWG) of the IC&V program was established to support implementation of
the evaluation of collaborative tools developed under IC&V. The EWG will
develop the evaluation metrics and methodology, and will develop, or
guide the development of, specific tests and tools for achieving
effective and economical evaluation of the collaborative technologies
that make up the IC&V program.

## 1.2Number range Level2Number range Level3Number range Level4 Background: The Evaluation Working Group and Its Aims

The original Evaluation Working Group included researchers with diverse
backgrounds and interests from several sites: Carnegie Mellon University
(CMU), The MITRE Corporation, National Imagery and Mapping Agency
(NIMA), National Institute of Standards and Technology (NIST), and
Amerind.[^1] The EWG's primary task is to define and validate low-cost
methods of evaluating collaborative environments, so that researchers
can use these methods to evaluate research products and users can use
these methods to choose collaborative systems that will best suit their
needs. This objective is further refined into a set of goals as follows:

1.  To develop, evaluate, and validate metrics and methodology for
    evaluating collaborative tools.

2.  To provide reusable evaluation technology, such that research groups
    can assess their own progress.

3.  To provide evaluation methods that are cheap relative to the
    requirements.

4.  To apply DOD-relevant criteria when evaluating collaborative systems
    relevant to areas such as:

    1.  Planning/design/analysis domains

    2.  C2 environments to capture planning/reaction/replanning cycle

    3.  Disaster relief exercises

    4.  Collaborative information analysis activities

5.  To define an application vision that will drive collaborative
    computing research.

The technologies supported under the IC&V program range from
infrastructure technologies at the level of networking and bus
protocols, to middleware for providing easy interoperability, to
user-oriented collaborative tools. Given this wide range of technologies
and the background of the EWG members, the EWG has decided to focus on
the user-oriented end of the spectrum. In addition, specific interests
of various EWG members (NIST, in particular) may lead to subgroups
working in the area of infrastructure technology evaluation, especially
as these areas affect the user level (e.g., sensitivity to network load
may limit number of participants in a collaborative session). Currently,
there are no plans for the EWG to provide evaluation metrics aimed at
the software infrastructure; e.g., how easy it is to make a new
application collaborative, or how a given layer of middleware might
enhance interoperability. These are clearly important issues that will
affect the long-term success of the program, but they lie outside the
scope of the EWG as it is currently constituted.

## 1.3Number range Level2Number range Level3Number range Level4 The Scope and Structure of this Document

This document was developed to encode agreements of the IC&V Evaluation
Working Group as we develop a framework and methodology for evaluation
of the IC&V technologies.

The IC&V program is not targeted at a specific collaboration problem.
Rather, the challenge for the EWG is to provide an evaluation
methodology that can be applied across the diverse IC&V research
projects and approaches to collaboration. Researchers need tools to
measure the incremental progress towards developing useful collaborative
systems, as well as methods to evaluate the impact of specific
technologies on the effectiveness of collaboration. Users need ways in
which to determine which collaborative software systems could meet their
needs.

We present a scenario-based approach to evaluation. The long-term goal
of the EWG is to develop a repository of scenarios that are scripted for
a collaborative community and enacted using the technologies under
evaluation. Since the technologies are diverse, the scenarios must be
generic enough to provide meaningful evaluation across multiple research
projects. Enacting the scenarios will provide data for the functional
evaluation and also provide exercise tools developed for the technology
evaluation. Different scenarios will exercise different aspects of
collaborative work, such as number of participants, kind of shared
objects, and ways participants need to interact with each other and with
the shared objects.

The remaining sections of this document are structured as follows.
Section 2 situates this methodology in the context of current evaluation
approaches from human-computer interface (HCI) and computer-supported
cooperative work (CSCW) research, and discusses the rationale for
scenario-based evaluation. It also defines critical terminology for use
in the remainder of the document.

Section 3 presents a framework that defines the design and
implementation space for collaborative systems. It includes a set of
generic task types that can be used to construct scenarios.

Section 4 discusses the concept of a scenario as a vehicle for
simulating a collaborative activity for purposes of evaluation. Our
approach to exercise and evaluate specific collaborative technologies
requires selection of appropriate scenarios. Section 4 describes methods
for using scenarios for purposes such as iterative evaluation,
assessment of system appropriateness, and comparison of systems.

Section 5 discusses a range of suggested metrics and measures for
evaluating collaborative technologies at various levels and illustrates
these with several examples.

Section 6 includes a discussion of how the methodology can be used to
design an experiment.

# Section 2Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-2number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading}

# Context and Definitions {#context-and-definitions .Section-Header}

## 2.1Number range Level2Number range Level3Number range Level4 Introduction

This section provides additional background for the remainder of this
document. The first subsection looks at related evaluation efforts from
the HCI and CSCW communities. The second subsection introduces the
scenario-based approach, and the third subsection defines our
terminology.

## 2.2Number range Level2Number range Level3Number range Level4 Methods of Interface Evaluation

Evaluations of human-computer interaction have traditionally been done
by a number of methods, including field studies, laboratory experiments,
and inspections. Each method assesses different aspects of the
interfaces and places different demands on the developer, user, and
evaluator.

Evaluations of collaborative technology are done best through field
evaluations because they can, among other things, be used to assess
social-psychological and anthropological effects of the technology
(Grudin 1988). Field studies unfortunately require substantial funding,
a robust system, and an environment that can accommodate experimental
technology. These three conditions are difficult to satisfy, and so
researchers turn to less onerous means, such as inspection methods and
laboratory exercises.

System inspections, such as cognitive walk-through (Polson et. al.
1992), heuristic evaluation (Nielsen and Molich 1990), and standards
inspection (for example, Open Software Foundation Motif inspection
checklists), employ a set of usability guidelines written by usability
experts. There exists a fairly large set of guidelines for user
interface design and single user applications, but few guidelines are
available for multi-user applications or collaborative technology. Also,
these methods are largely qualitative, not quantitative, and require HCI
expertise that may not always be available.

This leaves laboratory experiments, or empirical testing, which as an
evaluation technique more closely relates to field studies than
inspection techniques. Experiments are very appealing for new and
rapidly evolving technology and are potentially less expensive than
field studies. However, since they are conducted in controlled
environments with time restrictions they less accurately identify
dynamic issues such as embedding into existing environments, learning
curves, and acculturation. Watts et al. (1996) recommend compensating
for this flaw by performing ethnographic studies followed by laboratory
experiments. The ethnographic study helps evaluators understand the
"work context" which influences what is measured and observed in the
laboratory.

A generic process cannot presuppose a specific work context. Rather, we
have chosen to develop scenarios and measures based on high-level
collaborative system capabilities, to provide broad applicability across
a range of applications. Ethnographic studies related to specific work
contexts could provide a useful tool for validating the measures,
because some measures may not be appropriate or of interest in certain
contexts.

## 2.3Number range Level2Number range Level3Number range Level4 Use of Scenarios

A scenario is an instantiation of a generic task type, or a series of
generic tasks linked by transitions. It specifies the characteristics of
the group that should carry it out, and the social protocols that should
be in place. It describes what the users should (try to) do, but not
usually how they should do it. (Note that scenarios can be scripted in
various degrees of detail and thus could constrain evaluator's choices
for how they accomplish tasks; scripts will be discussed later in this
document.)

Scenarios are used in laboratory experiments to direct system usage.
They are versatile tools that can be used for many development
activities including design, evaluation, demonstration, and testing.
When used for evaluation, scenarios exercise a set of system features or
capabilities.

We would like to define general, reusable scenarios for collaborative
technologies. This is a challenge, requiring consideration of a large
set of technologies, but we can build on earlier work in this area.

In 1995, researchers met at the European Human-Computer Interaction
(EHCI) conference to develop scenarios that could be used to design,
evaluate, illustrate, and compare CSCW systems (Bass 1996). Their
approach was to define generic tasks that would be used to construct
technology specific scenarios. The tasks they described were mostly
single user activities such as joining a conference, setting up a new
group, and integrating a system with a collaborative tool.

Our approach begins by defining collaborative system capabilities or
functional requirements. Our "capabilities" are defined at a higher
level than the tasks defined by the EHCI researchers. The tasks we use
to evaluate the capabilities are combined to build scenarios that are
much more general than those described by the EHCI group. Consequently,
the scenarios are appropriate vehicles for cross-technology comparisons.
Many of the scenarios can be segmented if the complete scenario is too
large to use for comparisons. Also, general scenarios can be readily
adapted for any system that supports the capabilities required by the
scenario.

Nardi (1995), who has extensively studied the use of scenarios for
interface design, has argued for a provision to create a library of
reusable scenarios. We will begin to populate such a library with
scenarios that are technology-independent. Technology-specific scenarios
could be added when scenarios are specialized for real systems.

## 2.4Number range Level2Number range Level3Number range Level4 Definition of Terms

This evaluation program is concerned with the three principle variables
of participants, collaborative environments and collaborative
activities. It is easy to say that participants are actors who engage in
collaborative activities. Classifying collaborative environments and
activities in meaningful ways takes a bit more work.

### 2.4.1Number range Level3Number range Level4 Collaborative Environments

A *collaborative* *environment* is a system that supports users in
performing tasks collaboratively. It may be a particular piece of
software, such as Lotus Notes or MITRE\'s Collaborative Virtual
Workspace (CVW), or it may be a collection of components used together
to enable collaboration.

We are charged with the task of providing an evaluation methodology for
collaborative computing systems, present and especially future. Part of
our approach involves examining the types of things an environment
allows one to do in support of collaboration. To describe them, we must
define these terms: *requirement*, *capability*, *service*, and
*technology*.

*Requirements* for collaborative systems refer to the high level goals
that a group needs to accomplish. For example, "I need to keep my
colleagues informed of my progress on this project."

Collaborative *capabilities* are relatively high-level functions that
support users in performing particular collaborative tasks. Examples are
concepts such as synchronous human communication, persistent shared
object manipulation, archival of collaborative activity, etc.

The term *service* is used to describe the means by which one or more
collaborative environments achieve a given capability, and *technology*
is used to describe the particular hardware and/or software
implementation of a service. For example, a service is email, and a
technology is Eudora.

To tie together the four components of collaborative environments,
consider the following. To satisfy the *requirement* of sharing
information with colleagues, a group could use the collaborative
*capability* of synchronous human communication. One *service* that may
be used to achieve the goal in a variety of collaborative environments
is audio conferencing. One *technology* for audio conferencing is
Lawrence Berkeley Laboratory's Visual Audio Tool.

Examining which requirements and capabilities a collaborative
environment supports, and the services and specific technologies it uses
to do so, is one way of generating a functional categorization of the
collaborative environment. The categorization can be used to help
determine which collaborative systems may be best suited for the
proposed activities and which scenarios can be used to evaluate those
systems.

### 2.4.2Number range Level3Number range Level4 Tasks or Collaborative Activities

*Tasks* or *collaborative* *activities* are what people do, or wish to
do, together.

We use the term *task* synonymously with *collaborative activity*. Task
is a term that transcends the level of detail at which the activity is
described; *task* may refer to anything from a real activity that people
are engaging in to a highly scripted mock-up of such an activity
intended strictly for evaluation purposes.

A *general work task* (hereafter referred to as simply a "work task") is
a particular objective of a group, such as making a decision,
disseminating information, or planning for the next phase of a project.
A work task decomposition based on McGrath's categorization (McGrath
1984) is discussed in Section 3 to aid in generating specific measures
and anticipated needed capabilities and services. A *transition task* is
the activity necessary to move from one objective to another. For
example, starting up a session, summarizing results of a decision,
reporting the decision to absent colleagues, and assigning action items
constitute transition tasks. *Social protocols* constitute attributes of
work tasks, and are those activities that directly facilitate the
interpersonal management of joint work, such as floor control and access
control mechanisms. *Group characteristics* such as size, homogeneity,
and collocation versus non-collocation affect how tasks can be
performed.

### 2.4.3Number range Level3Number range Level4 Scenarios

Several different types of descriptions of mocked-up activity can be
used during evaluations. At the highest level we have the *scenario*.
This is a high-level description of a collaborative task that subjects
might be asked to engage in to test collaborative systems. A *fully*
*specified scenario* will include the background documentation,
instructions, etc. required to have a subject or subjects enact the
scenario. Scenarios are often broken down into their constituent tasks.
In some cases it will be possible to do evaluations based upon only a
subset of the tasks for a given scenario.

Some evaluation goals are better met by precisely repeatable scenarios,
perhaps even by automated actors. For this purpose there are *scripted
scenarios,* which are procedural descriptions of scenarios. Scripts can
be written at levels corresponding to technologies, services, or
capabilities.

# Section 3Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-3number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading}

# A Framework for Collaborative Systems {#a-framework-for-collaborative-systems .Section-Header}

## 3.1Number range Level2Number range Level3Number range Level4 Introduction

The framework outlined in this section provides a structured way of
thinking about collaborative systems and the evaluation of those
systems. The framework can aid the researcher in making some preliminary
judgments about a system or its usefulness to a particular group; for
example, the framework can identify systems that are likely or unlikely
to support a group's work, thus narrowing down the number of systems to
be further investigated using the scenario-based portion of the
methodology. The framework can also aid in choosing scenarios for a
scenario-based evaluation of a system.

This framework builds on one devised by Pinsonneault and Kraemer (1989)
to analyze the impact of technology on group process while controlling
for the effect of other contextual variables. We have merged the work of
McGrath (1984) into our expanded framework to enable us to classify
tasks that groups perform.

This section includes an overview of the framework and a description of
how the framework can be used to describe and evaluate CSCW systems. We
also provide detailed descriptions of each level of the framework.

## 3.2Number range Level2Number range Level3Number range Level4 Overview of the Collaborative Framework

The framework is divided into four levels: requirement, capability,
service, and technology.

The *requirement level* of the collaborative framework is composed of
requirements generated from the tasks being performed by the group and
the support necessitated by the characteristics of the group.
Requirements for supporting different types of groups include support
for the social interactions of the group as well as the requirements due
to group size, location, computer platforms, etc. The requirement level
includes both work tasks and transition tasks (described below).

The *capability level* of the framework describes functionality that is
needed to support the different requirements. The functionality
described in capabilities can be obtained from different services. For
example, the capability to synchronously communicate with another
meeting participant during an electronic meeting could be accomplished
by a text chat service or by telephone service. Certain capabilities may
be necessary or recommended to support work and transition tasks, social
protocols, and group characteristics in the requirement level.

The *service level* describes services such as e-mail, audio, video,
application sharing, and networking services, that can be used to
support the capabilities needed in CSCW systems. Different types of
services can be used to provide the same capabilities that support
specific requirements. For example, a need for synchronous,
non-collocated communications could be satisfied by text chat or video
teleconferences. Comparisons and tradeoffs of performance and cost can
be made at this level.

The *technology level* describes specific implementations of services.
This level could be considered to be the set of possible components
needed to build a given CSCW system, as well as their integration and
interfaces. For example, different e-mail systems would exist at this
level, as would the numerous ways to implement floor control, the
various algorithms to control documentation locking and requesting, and
the various networking services such as ATM. Specific implementations
can be compared with respect to performance, cost, functionality, and
usability.

## .3Number range Level2Number range Level3Number range Level4 Using the Framework to Begin Evaluating a CSCW System

### 3.3.1Number range Level3Number range Level4 General Evaluation Approaches Using the Framework

The framework can be used for evaluation in a top-down (requirement to
technology level) approach, a bottom-up (technology to requirement
level) approach, or a "middle out" approach.

Depending on the approach taken, the framework may, for example, help an
evaluator select a subset of the evaluation tools a group needs to chose
from; systems that do not meet or exceed acceptable levels for the
measures available at a given level can be eliminated from further
consideration. Or the framework may help a researcher determine whether
a particular system could support a particular group adequately, or
understand whether and how to implement missing functionality in a
system.

A top-down approach allows evaluators to match requirements of the group
to the tools needed to support collaborative efforts. To perform a
top-down evaluation, users would begin with a top-level requirement such
as "we need to work together even though half our group is in Toronto
and half is in Amsterdam." The users would then decide which
capabilities they need to support their group work, such as synchronous
non-collocated meeting support and asynchronous document transfer. They
would then look at services to provide these two types of capabilities,
such as text chat and email. Finally, the users would evaluate at the
technology level whether systems such as MITRE's CVW or America On Line
(AOL) Instant Messenger would provide the necessary text chat
performance and features, and similarly evaluate systems such as
Netscape Mail and Eudora for email needs.

A bottom-up approach allows evaluators to determine the types of
collaboration requirements that a given system can support best. For
example, a researcher may need to investigate a newly developed
tool---which exists at the technology level---to find out whether it
answers the need it was developed to meet, and whether it answers other
requirements as well. The researcher would move up through the
collaborative framework, first deciding what services the tool provides,
then abstracting to the more general capabilities, and finally
determining the top-level requirements that the tool satisfies.

A "middle-out" evaluation begins either at the capabilities or services
level. Such an evaluation could be used whenever there are questions
regarding whether missing functionality should be included in a system,
or how some capabilities could be used. For example, researchers who
would like to know the effectiveness of incorporating a new pointing
feature in a shared whiteboard might use a middle-out evaluation. The
"middle-up" portion of the evaluation would involve determining what
requirements such a new feature would satisfy. The "middle-down" portion
of the evaluation would result in guidance for how to implement such a
feature at the detailed technology level.

### 3.3.2Number range Level3Number range Level4 Using the Framework at Each Level

At the requirement level we can evaluate how well a CSCW system supports
the work tasks, the transition tasks, the social protocols, and the
characteristics of a specific group in general. We can also evaluate the
artifacts produced as well as the process the group used to produce an
artifact. The detailed framework description (Section 3.4) describes the
measures for task outcome for each task type. It is important to note
that the variables being measured differ depending on the type of task
the group is doing. For example, different measures are needed to
evaluate the outcome of a brainstorming task than to evaluate the
outcome of a planning task. (See Section 5 for a more thorough
discussion of measures.)

At the capability level, researchers and potential users of a system may
evaluate the appropriateness of specific capabilities to support the
work tasks, transition tasks, social protocols, and characteristics of
the group. To obtain answers to these questions, measures such as time
on task, awareness questions, amount of setup time for equipment and
configuration are used.

At the service level, framework users will examine the functionality of
various types of services to understand how a given capability would be
supported using a particular type of service. This allows users to
compare services in order to determine which service seems most
appropriate for the requirements. Performance thresholds, including
robustness, can be examined at this level. Tests such as the ability to
view an image with a certain amount of discrimination and the
acceptability of audio or video can be conducted at this level.

At the technology level, specific implementations exist. Therefore,
actual usability measures can be examined for an implementation. The
implementations of the services can be evaluated to make sure they meet
threshold values. Performance comparisons can be made at this level
between different implementations.

## 3.4Number range Level2Number range Level3Number range Level4 Detailed Framework Description

### 3.4.1Number range Level3Number range Level4 Requirement Level

The following requirement level description is based on its contents:
work tasks, transition tasks, social protocols, and group
characteristics.

#### 3.4.1.1Number range Level4 Work Tasks

Work tasks are the heart of a collaboration---the work that people need
to do to meet their collaborative goals.

Work tasks include activities such as solving a problem, developing a
plan, disseminating information, negotiating, and reaching consensus.

#### 3.4.1.2Number range Level4 Transition Tasks

Transition tasks are tasks used to move between work tasks. They may
include summarizing the outcome of the last task, assigning action items
to members of the group, and noting dates for expected completion of
assignments. The beginning and ending of any group collaboration involve
transition tasks such as taking roll, requesting changes to an agenda,
and locating missing meeting participants. Transition tasks also apply
to asynchronous collaborations. A group member may suggest that the
e-mail discussions about a particular subject end and volunteer to
summarize the discussion and distribute this to group members; or a new
person may join the meeting and need to get "caught up."

A transition task may occur formally or informally, depending on the
social protocol that the group is using. Transitions to the next work
task occur formally if the chair of the group either moves the group to
the next agenda item or the group votes to move to the next item.
Informal transitions to the next work task occur when the group moves
the discussion rather naturally to another topic or starts another group
activity.

#### 3.4.1.3Number range Level4 Social Protocols

Social protocols define the ways in which collaborative sessions are
conducted. Collaborative sessions may vary from informal sessions to
very formal sessions with a chair, an agenda that is strictly followed,
and rules of order. In the context of meeting support, for example,
social protocols support role management, floor control, and other basic
meeting conduct activities. Table 1 lists example parameters that social
protocol requirements support for meetings.

####### Table 1. Example Parameters for Social Protocols During Meetings

  --------------- -------------------------------------------------------
  Meeting         Parameters
  Component       

  Chair           None, loose control, tight control

  Agenda          None, modifiable, non-modifiable (strict)

  Rules of order  Used, not used

  Titles          Yes, no, anonymous

  Floor control   Dictated by agenda, directed by chair, informal
                  turn-taking, free-for-all

  Hierarchy       Voting, contributing-restricted, contributing-free
  support         access, observing only

  Communication   Private or public, 1-way or n-way
  support         

  Security        From none to highly classified (e.g., top secret
                  special compartmented information)
  --------------- -------------------------------------------------------

Social protocols may also support awareness of other group members'
presence, activities, locations, temporality, and motivations. There are
several different ways to organize awareness components; the approach
used by Villegas and Williams (1997) is used in Table 2.

####### Table 2. Example Awareness Components and Questions for Social Control

  ----------------------- -----------------------------------------------------
  Awareness Component     Awareness: Top-Level and Subordinate Questions

  Presence: Who?          Who else is in the workspace?

                          Can the user tell who else is logged into the
                          session?

                          Can the user tell whether anyone else is working on
                          the collaborative task?

                          Can the user tell the identity of other people
                          working on the collaborative task?

  Actions: What?          What are other participants doing?

                          Can the user tell what tasks the other participants
                          are working on?

                          Can the user tell what tools or objects the other
                          participants are using or manipulating?

                          Can the user tell what changes the other participants
                          are making to objects in the shared workspace?

                          Can the user tell what changes he/she and others are
                          allowed to make?

                          Can the user tell what the relative activity levels
                          of the other participants are?

                          Can the user tell whether the other participants are
                          willing to be interrupted?

  Location: Where?        Where are the other participants working?

                          Can the user tell where in the shared workspace the
                          other participants are working?

                          Can the user tell what the other participants can
                          see?

                          Can the user tell where the other participants are
                          focused?

  Time: When?             When do changes made by the other participants take
                          place?

                          Can changes made by the other participants be shown
                          to the user in real time?

                          Can past elements be replayed?

                          Can the user find out when a particular past event
                          happened?

  Motivation/Intention:   Why are the other participants doing what they are
  Why?                    doing?

                          Can the user tell what the other participants'
                          immediate intentions are?

                          Can the user tell what the other participants' goals
                          are?
  ----------------------- -----------------------------------------------------

#### 3.4.1.4Number range Level4 Group Characteristics

Group characteristics are attributes that determine how a group can work
together. Groups have different requirements depending on the makeup of
the group, the social relations (peer-to-peer versus boss-employee),
formality, the location of the group members, and the time requirements
for collaborative sessions. Examples of these characteristics are
included in Table 3. In addition to considering the current dimensions
of the group, system requirements should also take into account
anticipated changes in these dimensions. For example, all members of a
task force might start by being collocated but with the knowledge that,
in two weeks, half of the group will be remotely located. Group
characteristics affect how all of the different types of tasks are
performed.

####### Table 3. Example Group Characteristics

  -------------- ------------------------------ -----------------------------
  **Category**   **Characteristics**            **Potential Values**

  Group type     Number of members              Number

                 Group location                 Same for all, various
                                                locations

                 Homogeneity                    Gender diversity, peers only
                                                or multiple levels of
                                                corporate hierarchy,
                                                differences in computer
                                                experience, cultural
                                                diversity

                 Stage of development           Newly formed to
                                                well-established group

                 Motivation of group members    Very low to very high

  Group's time   Duration of collaboration      Number of hours to number of
  constraints    sessions                       days

                 Synchronicity of collaboration Synchronous or asynchronous
                 sessions                       

                 Length of time over which      Number of days to indefinite
                 collaboration will take place  

  Group's        Hardware, software             Platforms, software needed
                 requirements                   

  computer       Training expectations          Walk-up-and-use to formal
  requirements                                  classes

                 Computer expertise of group    Novice to expert
                 members                        
  -------------- ------------------------------ -----------------------------

#### 3.4.1.5Number range Level4 Requirements Level Measures

The primary measures taken at the requirements level are time for task
completion, similarity of participants' perceptions of the outcome,
quality of work produced, and satisfaction of users. The secondary
measures include agreement of participants about what will be done next,
types of conflicts occurring in turn-taking, and number of awareness
breakdowns.

### 3.4.2Number range Level3Number range Level4 Capability Level

Collaborative capabilities provide a means of matching tasks with
services. This matching process must take into account how well the
service supports the capabilities and whether this level of support is
acceptable given the high level requirements.

The capability level can be divided into those capabilities that support
the different requirement drivers (i.e., work, transition, and social
protocol). By explicitly understanding which types of tasks a system
supports well, potential users can better weight an evaluation to choose
the system that best supports their highest priority types of tasks.
Examples of the capabilities that support work and transition tasks can
be seen in Table 4.

####### Table 4. Examples of Capabilities that Support Work and Transition Tasks

+--------+-------------------------------------------------------------+
| Task   | Example Capabilities/Subcapabilities                        |
+--------+-------------------------------------------------------------+
| Work   | -- Shared workspace                                         |
|        |                                                             |
|        | -- Full access to all objects                               |
|        |                                                             |
|        | -- Restricted access                                        |
|        |                                                             |
|        | -- Anonymous contributions                                  |
|        |                                                             |
|        | -- Communication                                            |
|        |                                                             |
|        | -- Anonymous communication                                  |
|        |                                                             |
|        | -- Side chats and private communication                     |
|        |                                                             |
|        | -- Message passing                                          |
|        |                                                             |
|        | -- Message leaving                                          |
|        |                                                             |
|        | -- N way communication                                      |
|        |                                                             |
|        | -- 1 way communication                                      |
|        |                                                             |
|        | -- Gesturing, pointing, agreeing, disagreeing               |
|        |                                                             |
|        | -- Feedback channel                                         |
|        |                                                             |
|        | -- Private communication                                    |
|        |                                                             |
|        | -- Secure communication                                     |
|        |                                                             |
|        | -- Private workspace                                        |
|        |                                                             |
|        | -- Support for object types                                 |
|        |                                                             |
|        | -- Object visualization                                     |
|        |                                                             |
|        | -- Object manipulation                                      |
|        |                                                             |
|        | -- Object management                                        |
+--------+-------------------------------------------------------------+
| Tran   | -- Collaboration coordination capabilities                  |
| sition |                                                             |
|        | -- Summarization capabilities                               |
|        |                                                             |
|        | -- Playback facility                                        |
|        |                                                             |
|        | -- Distribution of objects                                  |
|        |                                                             |
|        | -- Translation of objects between modalities                |
|        |                                                             |
|        | -- Collaboration planning capabilities                      |
|        |                                                             |
|        | -- Agenda support                                           |
|        |                                                             |
|        | -- Calendar support                                         |
|        |                                                             |
|        | -- Meeting notification                                     |
|        |                                                             |
|        | -- Voting                                                   |
|        |                                                             |
|        | -- Locator capabilities                                     |
|        |                                                             |
|        | -- Locate possible collaborators                            |
|        |                                                             |
|        | -- Locate group members                                     |
|        |                                                             |
|        | -- Locate objects                                           |
+--------+-------------------------------------------------------------+

At the capability level, measures can be taken of the various times
spent using objects, broken down by modality of use. Times spent using
capabilities for transition tasks and social protocols can be noted as
well. Collisions in turn taking and questions about awareness can also
be measured.

Part of the evaluation process could involve understanding whether
competing systems have the same set of capabilities. Table 5 shows a
part of an example checklist that includes a number of capabilities with
corresponding boxes that are filled in if the capability is provided, or
provided partially, by the subject system, or left blank if the system
does not support that capability. Five sample systems are identified as
"A" through "E."

####### Table 5. Sample Checklist of Collaborative Capabilities

####### ![](media/image3.wmf)

### 3.4.3Number range Level3Number range Level4 Service Level

The service level provides mechanisms to meet the user's need for
specific capabilities. It includes different types of services that can
be used in developing CSCW systems. In the future, this level could be
expanded to include pointers to threshold values a given service should
meet to provide adequate support for various capabilities. A list of
basic characteristics of the various services could also be included at
this level.

Examples of services include:

> -- Email
>
> -- Chat facility
>
> -- Internet connections
>
> -- Telephone conversation
>
> -- Multicast audio
>
> -- Multicast video
>
> -- Half duplex audio
>
> -- Full duplex audio
>
> -- Whiteboard
>
> -- Shared workspace
>
> -- Shared applications
>
> -- Encryption
>
> -- Recording
>
> -- History mechanism
>
> -- Lists of objects, participants, possible collaborators
>
> -- Version control
>
> -- Simultaneous sessions
>
> -- Collaborative space management
>
> -- Collaborative space navigation
>
> -- Object repository
>
> -- Object control
>
> -- Import/export facilities

Service level evaluations are of two types: comparisons to thresholds
and feature sets and comparisons between services (of the same type or
of different types). Service level measures such as quality of imported
image (readability, discernability, contrast), audio quality (clarity,
echo), and video quality (contrast, color tone) could also be used to
compare services.

A template/checklist can be provided to facilitate the research in
determining whether services are provided. An example of a partial
checklist can be seen in Table 6 below. This table lists the services
provided by sample systems, denoted system A through E. Each cell in the
table is filled in if the service is present and left white if it is
absent.

### 3.4.4Number range Level3Number range Level4 Technology Level

The technology level refers to the specific implementation of a system.

User interface components and the elements needed to integrate
technology building blocks into a unified CSCW system exist at this
level. Usability evaluations are best conducted at the technology level
within the context of group work tasks, although simple evaluations of
performance and quality can be made outside of a group work context.
Performance and quality measures can be compared to threshold measures
at the service level to ensure that the implementation being evaluated
meets or exceeds those values.

####### Table 6. Sample Checklist of Collaborative Services

![](media/image4.wmf)

## 3.5Number range Level2Number range Level3Number range Level4 Collaborative Tasks

The types of tasks a group needs to accomplish can determine many of
their specific work and transition task requirements. A listing of
collaborative task types may be used as a starting point in analyzing a
group\'s requirements in a top-down evaluation. In a bottom-up
evaluation, systems can be analyzed to determine which tasks and
requirements they can support.

Our task descriptions are based mainly upon tasks described in McGrath's
group research (1984). The tasks described by McGrath should be thought
of as a continuum. They are numbered from 1 to 8, with successive tasks
related to each other. The tasks are summarized in Table 7.

Note that a portion of McGrath task 6 is not included in Table 7 because
it is not applicable to collaborative work. Slight differences in other
tasks are also noted in Table 7. In addition, we have added a task type
(task 9) for "information dissemination." Information dissemination
refers to activities such as participating in classes or sharing news of
corporate restructuring. Such tasks are not covered by the other McGrath
tasks, and so the addition and definition of a ninth task type was
necessary to form a comprehensive collection of current work practices.
We refer to these nine types of tasks as "work tasks" throughout the
rest of this paper. Collectively, the work task descriptions and
examples should enable users of the methodology to identify the task
types that reflect the tasks performed by a particular group that is in
need of collaborative computing support.

In Table 7, each task type has suggested measures of task outcome. In
addition, for many task types, research has uncovered problems that may
occur with groups performing this type of task. In some instances,
computer-mediated processes or other group processes may be able to
alleviate these problems. We have listed known problems so that
comparison measures can be made to see if there is any effect. Where
there is research about computer-mediated work or group interactions and
the effect on the task outcome, we have included it under the heading
\"known research.\" An example of each task is also provided to help in
understanding how the generic task maps to a real world task. Under
\"suggested capabilities\" are those capabilities that the research
suggests may be valuable in carrying out that particular generic task.

## 3.6Number range Level2Number range Level3Number range Level4 Summary of Collaborative Framework

The collaborative framework is intended to benefit both researchers and
developers of collaborative systems as well as purchasers and users of
collaborative systems. The framework is useful in identifying scenarios
for a group to use in evaluating a system or in comparing several
systems. Over time, we expect that guidelines will appear at the various
levels of the framework. These guidelines will suggest capabilities for
work tasks and transition tasks; services best suited to certain
capabilities; and the appropriate usability, performance, and features
thresholds needed for specific services.

+----+----------+----------+---------------+-----------+--------------+
| *  |          |          |               |           |              |
| *T |          |          |               |           |              |
| ab |          |          |               |           |              |
| le |          |          |               |           |              |
| 7. |          |          |               |           |              |
| M  |          |          |               |           |              |
| cG |          |          |               |           |              |
| ra |          |          |               |           |              |
| th |          |          |               |           |              |
| T  |          |          |               |           |              |
| as |          |          |               |           |              |
| ks |          |          |               |           |              |
| Wi |          |          |               |           |              |
| th |          |          |               |           |              |
| Va |          |          |               |           |              |
| ri |          |          |               |           |              |
| at |          |          |               |           |              |
| io |          |          |               |           |              |
| ns |          |          |               |           |              |
| (" |          |          |               |           |              |
| Wo |          |          |               |           |              |
| rk |          |          |               |           |              |
| T  |          |          |               |           |              |
| as |          |          |               |           |              |
| ks |          |          |               |           |              |
| ") |          |          |               |           |              |
| ** |          |          |               |           |              |
+====+==========+==========+===============+===========+==============+
| ** | **Defi   | **       | **Known       | **        | **Example**  |
| Ty | nition** | Specific | Research**    | Suggested |              |
| pe |          | me       |               | Capab     |              |
| ** |          | asures** |               | ilities** |              |
+----+----------+----------+---------------+-----------+--------------+
| 1  | P        | --       | -- Social     | --        | The group    |
|    | lanning. | Amount   | relations     | Calendar  | has to       |
|    | Group    | of time  | hinder task   | support   | produce an   |
|    | members  | per      | efforts       |           | advertising  |
|    | are      | par      |               | -- Text   | campaign for |
|    | given a  | ticipant | -- There can  | object    | a new        |
|    | goal and |          | be a strong   | creation, | account by   |
|    | asked to | --       | effect on the | editing,  | the end of   |
|    | develop  | Amount   | group due to  | di        | the month.   |
|    | a        | of       | social        | splaying, | They have a  |
|    | written  | calendar | influence and | a         | meeting to   |
|    | plan for | time     | conformity    | rranging, | plan the     |
|    | carrying |          |               | storing   | different    |
|    | out the  | --       | -- Groups     |           | tasks that   |
|    | steps    | Prac     | often have    |           | each member  |
|    | needed   | ticality | trouble       |           | will carry   |
|    | to reach | of plan  | seeing        |           | out,         |
|    | that     | (quality | alternatives; |           | complete     |
|    | goal.    | of task  | tend to focus |           | with time    |
|    | The      | outcome) | on only a few |           | lines for    |
|    | written  |          | alternatives  |           | doing so and |
|    | plan     |          |               |           | different    |
|    | should   |          | --            |           | coordination |
|    | include  |          | Participation |           | points.      |
|    | alt      |          | can be very   |           |              |
|    | ernative |          | unequal; this |           |              |
|    | paths or |          | increases as  |           |              |
|    | actions. |          | group size    |           |              |
|    |          |          | grows         |           |              |
|    |          |          |               |           |              |
|    |          |          | -- Groups     |           |              |
|    |          |          | tend to avoid |           |              |
|    |          |          | conflict and  |           |              |
|    |          |          | spend more    |           |              |
|    |          |          | time on       |           |              |
|    |          |          | non-          |           |              |
|    |          |          | controversial |           |              |
|    |          |          | issues.       |           |              |
|    |          |          | Controversial |           |              |
|    |          |          | issues tend   |           |              |
|    |          |          | to become     |           |              |
|    |          |          | personalized  |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 2  | Brain    | --       | -- Creativity | --        | The group    |
|    | storming | Number   | of            | Anonymous | has a goal   |
|    | and      | of ideas | individuals   | comm      | to raise     |
|    | group    |          | is stifled by | unication | \$200,000 to |
|    | cre      | --       | social        |           | build a new  |
|    | ativity. | Ori      | influence of  | --        | community    |
|    | Members  | ginality | group         | Sy        | center. They |
|    | of a     | of ideas |               | nchronous | generate     |
|    | group    |          | --            | comm      | ideas for    |
|    | are      |          | Individuals   | unication | funding      |
|    | given a  |          | are able to   |           | raising      |
|    | pa       |          | take          | -- N way  | events,      |
|    | rticular |          | advantage of  | comm      | people to    |
|    | topic    |          | creativ       | unication | ask for      |
|    | area and |          | ity-enhancing |           | c            |
|    | asked to |          | forces in     | -- Shared | ontributions |
|    | br       |          | group -       | workspace | and          |
|    | ainstorm |          | social        |           | p            |
|    | on       |          | support,      |           | ossibilities |
|    | ideas.   |          | cross         |           | for loans or |
|    |          |          | stimulation   |           | selling      |
|    |          |          |               |           | \"shares\"   |
|    |          |          |               |           | to the       |
|    |          |          |               |           | community    |
|    |          |          |               |           | members to   |
|    |          |          |               |           | raise this   |
|    |          |          |               |           | money.       |
+----+----------+----------+---------------+-----------+--------------+
| 3  | Intel    | --       | -- Written    | -- Shared | A logical    |
|    | lective. | Number   | media is      | workspace | reasoning    |
|    | The      | of       | slower to     |           | problem such |
|    | group is | trials   | arrive at a   | --        | as the       |
|    | asked to | to       | solution than | G         | cannibal and |
|    | solve a  | solution | voice media   | esturing, | missionary   |
|    | problem  |          | is. But voice | pointing, | problem.     |
|    | for      | --       | media uses    | agreeing, |              |
|    | which    | Solution | more messages | di        | Three        |
|    | there is | (quality | than written. | sagreeing | cannibals    |
|    | a        | of task  |               |           | and three    |
|    | re       | outcome) | -- Audio only | -- N way  | missionaries |
|    | cognized |          | does not      | comm      | are on a     |
|    | s        | --       | differ        | unication | riverbank.   |
|    | olution. | Errors   | significantly |           | They need to |
|    | The      |          | from face to  | --        | cross the    |
|    | group is | --       | face (and     | Private   | river. There |
|    | asked to | Inferred | hence,        | group     | is a rowboat |
|    | d        | st       | probably      | commu     | that can     |
|    | etermine | rategies | video)        | nications | hold only    |
|    | a        |          |               |           | two people.  |
|    | concept, |          | --            |           | Only         |
|    | given    |          | Interacting   |           | missionaries |
|    | i        |          | groups are    |           | know how to  |
|    | nstances |          | almost always |           | row. At no   |
|    | of the   |          | more accurate |           | time can     |
|    | concept. |          | than their    |           | there be     |
|    |          |          | average       |           | more         |
|    |          |          | member        |           | cannibals    |
|    |          |          |               |           | than         |
|    |          |          | -- Groups     |           | missionaries |
|    |          |          | seldom do as  |           | on either    |
|    |          |          | well as their |           | side of the  |
|    |          |          | best members  |           | river. What  |
|    |          |          |               |           | is the       |
|    |          |          |               |           | minimum      |
|    |          |          |               |           | number of    |
|    |          |          |               |           | trips that   |
|    |          |          |               |           | can be made  |
|    |          |          |               |           | to transport |
|    |          |          |               |           | all six to   |
|    |          |          |               |           | the other    |
|    |          |          |               |           | side?        |
+----+----------+----------+---------------+-----------+--------------+
| 4  | Decision | How far  | -- Groups may | -- Shared | The group    |
|    | -making. | and in   | not use their | workspace | must decide  |
|    | Group    | what     | collective    |           | which of     |
|    | members  | d        | knowledge     | -- N way  | three job    |
|    | are      | irection | fully or      | comm      | candidates   |
|    | asked to | (if      | efficiently   | unication | should be    |
|    | develop  | any),    |               |           | hired. All   |
|    | c        | the      | -- Some       | -- Side   | candidates   |
|    | onsensus | group as | members may   | chats     | have the     |
|    | on       | a group  | have more     |           | same degree  |
|    | issues   | and the  | influence     |           | and          |
|    | that do  | ind      | than others;  |           | specialty    |
|    | not have | ividuals | the influence |           | type, but    |
|    | correct  | in the   | may not be    |           | different    |
|    | answers. | group    | based on      |           | work         |
|    |          | shift.   | competency    |           | experiences. |
|    |          | (A       |               |           | The group    |
|    |          | ttitudes | -- May be     |           | must decide  |
|    |          | are      | pressure      |           | which        |
|    |          | measured | towards       |           | candidate to |
|    |          | before   | quick, rather |           | hire.        |
|    |          | and      | than good,    |           |              |
|    |          | after    | decisions     |           |              |
|    |          | group    |               |           |              |
|    |          | disc     | -- Diversity  |           |              |
|    |          | ussion.) | of views and  |           |              |
|    |          |          | values may    |           |              |
|    |          |          | make          |           |              |
|    |          |          | r             |           |              |
|    |          |          | econciliation |           |              |
|    |          |          | difficult     |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 5  | C        | --       | -- Verbal     | -- Shared | The group is |
|    | ognitive | A        | interactions  | workspace | hiring a     |
|    | conflict | greement | can lead to   |           | designer.    |
|    | tasks.   | among    | clarification | -- N way  | Three        |
|    | Members  | members  | of why group  | comm      | candidates   |
|    | of the   |          | members are   | unication | are at the   |
|    | group    | --       | consistently  |           | top of the   |
|    | hold     | Inter    | using         |           | list. Each   |
|    | d        | personal | different     |           | has a        |
|    | ifferent | trust    | policies. But |           | different    |
|    | vie      |          | if policies   |           | degree type. |
|    | wpoints. | --       | are used      |           | The group is |
|    | The      | Changes  | in            |           | divided      |
|    | group is | in       | consistently, |           | about the    |
|    | asked to | m        | this leads to |           | type of      |
|    | make a   | ember\'s | a distrust of |           | experience   |
|    | series   | views    | and a         |           | best suited  |
|    | of       |          | reduction in  |           | to this      |
|    | d        |          | understanding |           | position.    |
|    | ecisions |          | of the other. |           | The group is |
|    | from     |          |               |           | inter        |
|    | a        |          | -- Group      |           | disciplinary |
|    | vailable |          | members may   |           | and each     |
|    | inf      |          | change policy |           | tends to     |
|    | ormation |          | to increase   |           | favor hiring |
|    | that is  |          | accuracy.     |           | the          |
|    | imp      |          |               |           | candidate    |
|    | erfectly |          |               |           | most closely |
|    | co       |          |               |           | aligned with |
|    | rrelated |          |               |           | their        |
|    | with     |          |               |           | discipline.  |
|    | cr       |          |               |           |              |
|    | iterion. |          |               |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 6  | Mixed    | Not      | Not           | Not       | Not          |
|    | motive   | app      | applicable.   | ap        | applicable.  |
|    | tasks. A | licable. |               | plicable. |              |
|    | range of |          | \*Note that   |           |              |
|    | tasks,   |          | McGrath also  |           |              |
|    | differe  |          | includes      |           |              |
|    | n-tiated |          | dilemma tasks |           |              |
|    | by the   |          | in this       |           |              |
|    | degree   |          | category.     |           |              |
|    | to which |          | However,      |           |              |
|    | a group  |          | since the     |           |              |
|    | m        |          | dilemma       |           |              |
|    | ember\'s |          | decisions are |           |              |
|    | outcome  |          | made          |           |              |
|    | is       |          | i             |           |              |
|    | affected |          | ndependently, |           |              |
|    | by a     |          | no            |           |              |
|    | com      |          | collaboration |           |              |
|    | bination |          | occurs.       |           |              |
|    | of his   |          | Therefore, we |           |              |
|    | own      |          | have not      |           |              |
|    | actions  |          | included      |           |              |
|    | and the  |          | dilemma tasks |           |              |
|    | group\'s |          | in our        |           |              |
|    | ou       |          | framework.    |           |              |
|    | tcome.\* |          |               |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 6A | Neg      | --       | > --          | -- N way  | Company A    |
|    | otiation | Quality  | > Negotiators | comm      | and Company  |
|    | task.    | of       | > are more    | unication | B are        |
|    | The      | solution | > competitive |           | negotiating  |
|    | group is | as       | > when any of | -- Group  | the sale of  |
|    | divided  | e        | > these       | private   | supplies     |
|    | into x   | valuated | > conditions  | comm      | from A to B. |
|    | s        | by each  | > hold:       | unication | Company A    |
|    | ubgroups | subgroup | >             |           | wants to     |
|    | with a   |          | > -- They     | -- Shared | sell more of |
|    | ne       | -- Time  | > think       | workspace | the supplies |
|    | gotiator | to       | >             |           | at a lower   |
|    | elected  | solution |  constituents | --        | price to     |
|    | for      |          | > distrust    | Private   | company B,   |
|    | s        | --       | > them        | workspace | but this     |
|    | ubgroup. | At       | >             |           | means that   |
|    | The      | tainment | > -- They     |           | company B,   |
|    | d        | of       | > were        |           | while saving |
|    | ifferent | solution | > elected     |           | money on the |
|    | s        | (task    | >             |           | sale, will   |
|    | ubgroups | com      | > -- They are |           | have to      |
|    | d        | pletion) | > being       |           | arrange      |
|    | isagree; |          | > observed    |           | financing    |
|    | t        | --       | >             |           | with company |
|    | radeoffs | Eva      | > -- They     |           | A.           |
|    | have to  | luations | > have a      |           |              |
|    | be made  | of       | > prior       |           |              |
|    | in       | neg      | > commitment  |           |              |
|    | multiple | otiators | >             |           |              |
|    | dim      | by group | > -- Their    |           |              |
|    | ensions. |          | >             |           |              |
|    | It is    | --       |  constituents |           |              |
|    | not      | Inter    | > belong to a |           |              |
|    | nec      | personal | > highly      |           |              |
|    | essarily | r        | > cohesive    |           |              |
|    | a        | elations | > group       |           |              |
|    | zero-sum | between  | >             |           |              |
|    | problem  | group    | >             |           |              |
|    |          | members  | --Negotiators |           |              |
|    |          |          | > who do not  |           |              |
|    |          |          | > belong to   |           |              |
|    |          |          | > the group   |           |              |
|    |          |          | > feel freer  |           |              |
|    |          |          | > in the      |           |              |
|    |          |          | > negotiation |           |              |
|    |          |          | > process but |           |              |
|    |          |          | > are less    |           |              |
|    |          |          | > supported   |           |              |
|    |          |          | > by the      |           |              |
|    |          |          | > group.      |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 6B | Ba       | > --     | See under     | -- N way  | There has    |
|    | rgaining | > F      | task 6A.      | comm      | been a price |
|    | task. A  | requency |               | unication | set by       |
|    | conflict | > of     |               |           | Company A    |
|    | of       | > no-    |               | -- Group  | for a        |
|    | interest | solution |               | private   | machine that |
|    | must be  | > trials |               | comm      | company B    |
|    | resolved | >        |               | unication | wishes to    |
|    | between  | > --     |               |           | purchase.    |
|    | two      | >        |               | -- Shared | Company B    |
|    | indi     | Absolute |               | workspace | does not     |
|    | viduals, | > and    |               |           | feel that    |
|    | but in   | >        |               | --        | the price is |
|    | this     | relative |               | Private   | low enough.  |
|    | case     | > gains  |               | workspace | Company A is |
|    | whatever | > of the |               |           | trying to    |
|    | one      | > two    |               | -- Text   | maximize     |
|    | in       | > ind    |               | object    | profit as    |
|    | dividual | ividuals |               | man       | the company  |
|    | gains    | > over   |               | ipulation | is having    |
|    | results  | > series |               |           | cash flow    |
|    | in a     | > of     |               |           | problems,    |
|    | loss for | > trials |               |           | but loosing  |
|    | the      | >        |               |           | the sale     |
|    | other    | > --     |               |           | will be a    |
|    | ind      | > R      |               |           | problem      |
|    | ividual. | esponses |               |           | also.        |
|    | The      | > to     |               |           |              |
|    | t        | > d      |               |           |              |
|    | rade-off | ifferent |               |           |              |
|    | is made  | > st     |               |           |              |
|    | on a     | rategies |               |           |              |
|    | single   | >        |               |           |              |
|    | di       | > --     |               |           |              |
|    | mension: | > Op     |               |           |              |
|    | what one | ponents' |               |           |              |
|    | party    | >        |               |           |              |
|    | gains,   |  ratings |               |           |              |
|    | the      | > of     |               |           |              |
|    | other    | > each   |               |           |              |
|    | party    | >        |               |           |              |
|    | loses.   | other\'s |               |           |              |
|    |          | > ba     |               |           |              |
|    |          | rgaining |               |           |              |
|    |          | > st     |               |           |              |
|    |          | rategies |               |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 6C | Winning  | > --     | -- Strong     | -- N way  | A variant of |
|    | c        | > Which  | tendency      | comm      | Parcheesi is |
|    | oalition | > co     | towards the   | unication | played in    |
|    | tasks.   | alitions | formation of  |           | which a      |
|    | Subsets  | > form   | coalitions of | -- Side   | player\'s    |
|    | of       | >        | minimum       | chats     | piece is     |
|    | members  | > -- The | winning       |           | given a      |
|    | make     | >        | resources     | -- Shared | weight.      |
|    | agr      | division |               | workspace | Moves are    |
|    | eements. | > of     | -- For groups |           | based on     |
|    | The      | >        | larger than   | --        | product of   |
|    | subset   | outcomes | 3, there is a | Private   | the weight   |
|    | that     | >        | tendency      | workspace | of the piece |
|    | wins     | > -- Any | towards       |           | and the roll |
|    | then     | > shift  | coalitions    | --        | of a pair of |
|    | a        | > over   | with minimum  | G         | dice.        |
|    | llocates | > time   | numbers of    | esturing, | Players can  |
|    | the      |          | players       | pointing, | play alone   |
|    | r        |          |               | agreeing, | or can       |
|    | esources |          | -- Females    | di        | combine with |
|    | among    |          | play the      | sagreeing | one other    |
|    | the      |          | coalition     |           | player by    |
|    | group    |          | game          | --        | adding their |
|    | members. |          | acc           | Support   | moves        |
|    | The two  |          | ommodatingly; | for       | together.    |
|    | research |          | males play    | comp      | They must    |
|    | q        |          | e             | utational | then agree   |
|    | uestions |          | xploitatively | object    | on how to    |
|    | are the  |          |               |           | divide the   |
|    | f        |          |               | -- 2D     | payoff       |
|    | ormation |          |               | object    | assuming     |
|    | of the   |          |               | man       | they win.    |
|    | co       |          |               | ipulation |              |
|    | alition, |          |               |           |              |
|    | and the  |          |               |           |              |
|    | al       |          |               |           |              |
|    | location |          |               |           |              |
|    | of the   |          |               |           |              |
|    | re       |          |               |           |              |
|    | sources. |          |               |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 7  | Com      | > --     | --            | -- N way  | A focal      |
|    | petitive | > Team   | Inter-group   | comm      | group        |
|    | perfo    | > per    | competition   | unication | competes     |
|    | rmances. | formance | increases     |           | with an      |
|    | Groups   | >        | within-group  | -- Side   | opposing     |
|    | are      | (quality | cohesion.     | chats     | team that    |
|    | c        | > of     |               |           | has a series |
|    | ompeting | > task   | -- Success in | --        | of           |
|    | against  | >        | a competitive | Private   | preplanned,  |
|    | each     | outcome) | task also     | comm      | semi-        |
|    | other    | >        | increases     | unication | standardized |
|    | with no  | > --     | within-group  |           | patterns.    |
|    | re       | > In     | cohesion.     | -- Secure | The          |
|    | solution | dividual |               | comm      | responses to |
|    | of       | > per    | -- Groups do  | unication | the focal    |
|    | conflict | formance | not always    |           | group\'s     |
|    | e        | >        | distinguish   | --        | moves are    |
|    | xpected. | (quality | between good  | Private   | based on     |
|    | The goal | > of     | group         | (to       | pre-planned  |
|    | of each  | > task   | performance   | group)    | strategies.  |
|    | group is | >        | and winning.  | workspace | A            |
|    | to win   | outcome) |               |           | re           |
|    | over the | >        |               |           | connaissance |
|    | other    | > -- The |               |           | patrol or    |
|    | group.   | >        |               |           | defense of a |
|    | S        |  overall |               |           | position is  |
|    | ubgroups | > winner |               |           | an example   |
|    | are      |          |               |           | of such an   |
|    | paired   |          |               |           | activity.    |
|    | against  |          |               |           |              |
|    | each     |          |               |           |              |
|    | other an |          |               |           |              |
|    | equal    |          |               |           |              |
|    | number   |          |               |           |              |
|    | of       |          |               |           |              |
|    | times,   |          |               |           |              |
|    | under an |          |               |           |              |
|    | equal    |          |               |           |              |
|    | pattern  |          |               |           |              |
|    | of       |          |               |           |              |
|    | sit      |          |               |           |              |
|    | uations. |          |               |           |              |
|    | In the   |          |               |           |              |
|    | original |          |               |           |              |
|    | McGrath  |          |               |           |              |
|    | work,    |          |               |           |              |
|    | these    |          |               |           |              |
|    | perf     |          |               |           |              |
|    | ormances |          |               |           |              |
|    | are      |          |               |           |              |
|    | p        |          |               |           |              |
|    | hysical. |          |               |           |              |
|    | Here,    |          |               |           |              |
|    | these    |          |               |           |              |
|    | types of |          |               |           |              |
|    | tasks    |          |               |           |              |
|    | may be   |          |               |           |              |
|    | physical |          |               |           |              |
|    | or       |          |               |           |              |
|    | nonp     |          |               |           |              |
|    | hysical. |          |               |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 8  | Non-com  | -- Cost  | -- Increased  | -- Shared | A survival   |
|    | petitive | and      | interpersonal | workspace | task or      |
|    | c        | ef       | interaction   |           | rescue task  |
|    | ontests. | ficiency | does not      | -- N way  | in which a   |
|    | Groups   | of       | always lead   | comm      | group has to |
|    | perform  | perfo    | to higher     | unication | perform to   |
|    | some     | rmance - | productivity  |           | achieve a    |
|    | sort of  | speed    |               |           | goal:        |
|    | complex  | and      | -- Groups     |           | getting back |
|    | group    | errors   | influence     |           | to base or   |
|    | task.    |          | their members |           | saving       |
|    | The plan | --       | toward        |           | individuals. |
|    | for the  | Evidence | conformity    |           |              |
|    | task has | of       | with the      |           |              |
|    | already  | per      | group\'s      |           |              |
|    | been     | formance | standards -   |           |              |
|    | decided  | level    | this may      |           |              |
|    | upon. In | changes  | increase or   |           |              |
|    | this     | over     | decrease      |           |              |
|    | type of  | time     | productivity  |           |              |
|    | task,    | (quality |               |           |              |
|    | the      | of task  |               |           |              |
|    | group is | outcome  |               |           |              |
|    | merely   | but can  |               |           |              |
|    | e        | only be  |               |           |              |
|    | xecuting | measured |               |           |              |
|    | the      | for      |               |           |              |
|    | plan.    | rep      |               |           |              |
|    |          | etitions |               |           |              |
|    |          | of task) |               |           |              |
|    |          |          |               |           |              |
|    |          | --       |               |           |              |
|    |          | Member   |               |           |              |
|    |          | satis    |               |           |              |
|    |          | -faction |               |           |              |
|    |          | with the |               |           |              |
|    |          | task,    |               |           |              |
|    |          | the      |               |           |              |
|    |          | group,   |               |           |              |
|    |          | their    |               |           |              |
|    |          | own      |               |           |              |
|    |          | roles    |               |           |              |
+----+----------+----------+---------------+-----------+--------------+
| 9  | Non-     | --       |               | -- 1 way  | A corporate  |
|    | McGrath. | Shared   |               | comm      | officer      |
|    | Disse    | under    |               | unication | gives a      |
|    | mination | standing |               |           | briefing to  |
|    | of       | of       |               | --        | a division   |
|    | info     | inf      |               | Feedback  | about the    |
|    | rmation. | ormation |               | channel   | new sales    |
|    | The task |          |               |           | strategy.    |
|    | is to    | -- Time  |               | -- Object |              |
|    | di       | for      |               | d         | A professor  |
|    | stribute | dist     |               | isplaying | gives a      |
|    | inf      | ribution |               |           | lecture to a |
|    | ormation |          |               | --        | class.       |
|    | among    | --       |               | Summ      |              |
|    | members  | Audience |               | arization |              |
|    | of the   | reached  |               | cap       |              |
|    | group.   |          |               | abilities |              |
|    | Group    |          |               |           |              |
|    | members  |          |               |           |              |
|    | may      |          |               |           |              |
|    | share    |          |               |           |              |
|    | inf      |          |               |           |              |
|    | ormation |          |               |           |              |
|    | with     |          |               |           |              |
|    | each     |          |               |           |              |
|    | other,   |          |               |           |              |
|    | or a     |          |               |           |              |
|    | superior |          |               |           |              |
|    | may      |          |               |           |              |
|    | dis      |          |               |           |              |
|    | seminate |          |               |           |              |
|    | the      |          |               |           |              |
|    | inf      |          |               |           |              |
|    | ormation |          |               |           |              |
|    | to the   |          |               |           |              |
|    | group.   |          |               |           |              |
+----+----------+----------+---------------+-----------+--------------+

# Section 4Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-4number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading}

# Scenarios for Evaluation of Collaboration Tools {#scenarios-for-evaluation-of-collaboration-tools .Section-Header}

## 4.1Number range Level2Number range Level3Number range Level4 Introduction

Having defined a framework for describing collaborative systems and
generic task types, we can re-examine what a scenario is, how it can be
constructed or chosen from a library, and how we can use it for
evaluating collaborative systems.

Scripted and unscripted scenarios each serve different, but valuable
purposes.

An unscripted scenario allows the most natural interaction, because it
does not constrain the user's actions. It can also allow the
experimenters to determine whether the best ways of doing things are
apparent or intuitive to the users.

A scripted scenario allows a scenario to be repeated much more exactly.
Thus it allows much more accurate comparisons of measurements across
systems or even across multiple implementations of the same system.

A scripted scenario at the capability level gives the detailed
instructions in terms of the capabilities that the script uses. This
script does not require the use of any particular services, so it can be
run on any system that provides some means of instantiating the required
capabilities. Scripts defined at the capability level can often be used
across platforms that provide radically different sets of services in
support of the same basic capabilities.

A scripted scenario at the service level specifies the steps in the
scenario in terms of the services to be used, without committing itself
to a particular technological implementation of the service. It may, for
example, state that a particular discussion is to take place via video
conferencing, without specifying exactly how that will be achieved in
any particular system. A service-level script might be useful in cases
where one wants to compare different implementations of the same basic
services.

A scripted scenario at the technology-level is the most detailed. It
tells the users what buttons to press and exactly how to carry out the
scenario on a given set of technologies. It provides a detailed,
standardized, repeatable sequence of actions. This script can be carried
out by even the most inexperienced users, including those who lack the
domain knowledge required to run an unscripted version of the scenario,
and those lacking training on the technology being assessed. It could
also serve as training material for new users of the technology. Such a
script could even be carried out by participatons, removing the
reproducibility problems the human in the loop necessarily introduces.
It can be used for some types of user interface analysis, such as formal
dialog modeling techniques (Card, Moran and Newell 1983), but of course
it cannot be used to assess issues such as intuitiveness, since the
users are guided through every step of the interaction.

Since it can be useful to have both scripted and unscripted scenarios at
various levels for the same collaborative activity, it is instructive to
consider how one might generate scripts at various levels of detail for
a particular collaborative activity starting from a scenario for that
activity. It would be possible to start with the script and have real
people (with appropriate expertise, if necessary) run through it on a
particular set of technologies, logging the interactions. One could then
take the logs, edit out any undesirable actions (such as wrong paths or
puzzling over the interface) and create a detailed technology-level
script from it. By generalizing the script first to generic services and
then to the capabilities supported by those services, the scenario
developer could then generate higher-level scripts to allow scripted
evaluation across a greater variety of collaborative systems.

## 4.2Number range Level2Number range Level3Number range Level4 Constructing Scenarios

As an example of a means of constructing scenarios, consider an
evaluation at the requirements level to compare several systems. The
methodology aids in achieving repeatability across trials using the
different systems.

Scenarios are constructed by putting together tasks based on the
collaborative task types described in section 3. A group selects the
various McGrath task types for which its members will use the
collaborative system. They also consider the social protocols and group
characteristics appropriate for the group. The collaborative task types
are generic and thus, to construct scenarios, each group will choose
specific tasks that are instantiations of generic task types.

For example, a group might need to plan an activity and, in doing so,
solve a problem. These activities can be represented by McGrath task
type 1 (planning) and task type 9 (information dissemination). Here is a
scenario that instantiates these two task types:

A frantic call comes in. Colleague #1 is late for an important meeting
across town. You know about a few obstacles such as road construction
blockages between Colleague #1 and her meeting location, but Colleague
#2 has the latest report from the local news radio. Together, you and
Colleague #2 must devise the quickest route to the meeting. You must
agree on the route and inform Colleague #1.

Experimenters may find it useful to code their scenarios using
consistent terminology so that they can be more quickly compared to each
other. For example, in the following description Ti refers to a type of
transition task i. Task Type \# refers to a work task type of that
number. Pj refers to a collection j of attributes for a social protocol.
Using the collaborative framework to describe this scenario, we would
say:

> The group starts with a transition task to begin (Tstart). They
> perform a planning task (Task Type 1) using an appropriate social
> protocol (Ppeer). They summarize the decision in the transition task
> (Tsummary) and move to the information dissemination task (Task Type
> 9), followed by the ending transition task (Tend).

## 4.3Number range Level2Number range Level3Number range Level4 Choosing Scenarios

Ideally, a set of scenarios developed over time could be shared among
experimenters so that scenarios need not always be developed from
scratch. The Evaluation Working Group (EWG) has created some sample
unscripted and scripted scenarios that may be used for evaluation. They
may be found at http://www.antd.nist.gov/\~
icv-ewg/pages/scenarios.html. We encourage others to contribute
scenarios or scripts that they have written to help develop this
repository. A scenario template to complete may be found at
http://zing.ncsl.nist.gov/\~cugini/icv/domain-template.html.

Each example clearly sets out the types of tasks according to the
extended work task (McGrath-like) categorization in Section 3; and the
transitions, social protocols and group transitions that it exercises
(or aims to exercise). It also lists the capabilities and/or services
required or recommended to complete the scenario. This information will
be helpful for users in choosing appropriate scenarios to use for their
evaluation, as discussed below.

## 4.4Number range Level2Number range Level3Number range Level4 Using Scenarios for Evaluation

The utility of any given scenario is obviously much greater if it is
chosen to have the property defined as "salience" (Potts 1995). Potts
characterizes salient scenarios as those that are pertinent to
collaboration goals and model expected obstacles to task completion. The
evaluator should begin by identifying the goals addressed by the system
under evaluation and the obstacles to meeting those goals. Explicitly
identifying goals and obstacles can help the evaluator select the
appropriate scenario.

### 4.4.1Number range Level3Number range Level4 Using Scenarios to Iteratively Evaluate a Single System

One major goal of the Evaluation Working Group is to support system
developers in their need to do frequent evaluations of the system to
validate the theoretical improvements offered by new versions. This is a
bottom-up evaluation, beginning with the technology (the system being
evaluated) and moving up to the requirements that the system supports.

Developers should attempt to choose one or more scripted or unscripted
scenarios that exercise the types of capabilities and services their
system provides and/or involves the types of tasks, transitions and
social protocols that they hope to support.

Developers who devise their own scripted or unscripted scenarios (see
section 4.3) to exercise other sets of capabilities are encouraged to
contribute them to the scenario repository that the EWG has started.

### 4.4.2Number range Level3Number range Level4 Using Scenarios to Evaluate a System's Appropriateness for Your Requirements

For potential users looking to select a collaborative system to suit
their needs, the scenarios can serve a different purpose. Instead of
choosing scenarios based upon a particular set of capabilities or
services, they can instead choose scenarios that highlight the types of
tasks that they need to perform.

### 4.4.3Number range Level3Number range Level4 Using Scenarios to Compare Systems

If multiple systems are to be compared using scenarios, several
scenarios should be chosen for the evaluation. A complete comparison of
the systems should compare the performance of the systems on the
different scenarios. The set of scenarios in which each system performs
well or poorly highlights the requirements that each system does or does
not meet. When it is not possible to say which system is best overall,
it should be possible to determine which system is best for a particular
set of requirements, or which provides the greatest breadth of
requirement support even if it is not the best system for any particular
set of requirements. The results must be interpreted in terms of the
goals of the experiment.

# Section 5Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-5number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading}

# Metrics and Measures {#metrics-and-measures .Section-Header}

## 5.1Number range Level2Number range Level3Number range Level4 Introduction

In the preceding sections, we presented a framework for classifying
collaborative systems and an approach to evaluation that employs
scenarios. Here we enumerate the metrics and measures introduced in
Section 3 and discuss techniques for gathering the metrics.

Metrics are indicators of system, user, and group performance that can
be observed, singly or collectively, while executing scenarios. Metrics
-- such as time, length of turn, and other countable events -- are
directly measurable and can often be collected automatically.

Measures can be taken at each of the four levels of the collaborative
framework. Measures are derived from interpretations of one or more
metrics. For example, task completion time (a requirement-level measure)
is based on start and end time metrics. For asynchronous tasks, it may
be useful to distinguish time on task from elapsed time. A measure can
also be a combination of interpreted metrics and other measures. A
complicated measure, like efficiency, is partially derived from the
interpretation of metrics such as time, as well as user ratings and tool
usage. In addition, measures of system breakdown (taken at the service
or technology levels) contribute to efficiency.

A simple way to distinguish between metrics and measures is by the
following statement: a metric is an observable value, while a measure
associates meaning to that value by applying human judgment.

As another example of the metric and measure relationship, consider the
turn overlap metric. Overlap of speaker A by speaker B can be counted if
the start time of speaker B is greater than the start time of speaker A
but less than the end time of speaker A. Further interpretation is
required to determine if a particular occurrence of turn overlap is an
attempt to gain the floor (interruption in communication) or a
back-channel response indicating attentiveness (support, grounding).
This might be an interesting measure at the capability level, to
understand the ease and effectiveness of multi-person conversational
interaction.

The metrics and measures described here are relevant to laboratory
experiments. The goal of the Evaluation Working Group is to define
inexpensive evaluation technology that can be reproduced in a
laboratory. Hence, the measures do not address organizational impact and
other measures that require fielding production-quality systems.

Experiments involving human subjects, or sets of subjects, are expensive
and time consuming, both in terms of collection and analysis. In many
cases, the measures must be developed and validated before they can be
applied with confidence. Despite these difficulties, we describe options
for evaluating collaborative systems at all four levels of the framework
discussed in Section 3.

We begin with an overview of the methods, metrics and measures, and then
present the measures found at each level of the framework. Metrics are
introduced first because they are components of measures, including
methods used to collect metrics. Next we discuss measures, before
turning to automated logging techniques that may aid data collection and
analysis.

Figure 2 illustrates the relationship of data collection methods,
metrics, measures, and human judgment. The diagram also shows how the
measures of the four levels of the collaborative framework are nested,
emphasizing the mappings between these levels. As illustrated at the
bottom of the diagram, we use data collection methods, such as a logging
tool or video taping, to gather metrics on a system (the next level up).
Metrics provide the raw input necessary for refining measures for
requirements, capabilities, services, and technologies. Human
judgements, from both experts and users are associated with all four
levels.

![](media/image5.wmf)

###### Figure 2. Overview of the Levels of Measures

Metrics based on human judgment also support measures for making
judgments about the system. Questions to both experts and users of the
system can provide valuable data points in the evaluation of a system.

## 5.2Number range Level2Number range Level3Number range Level4 Metrics

The following table of metrics comprises the observable data elements
that were identified as useful indicators of system and group
performance. A single metric is just a number (e.g., number of turns) or
a set of numbers (e.g., start time and end time). Metrics can also be
used in combinations (e.g., the repair activities metric is partially
built up from the number of undos and the number of cancels and is the
deviation from some established 'right path'). For each metric, we
present in Table 8 definitions and examples of ways to capture the
metric. Where applicable, the metrics are enumerated or are broken down
into finer granularity.

We attempt to provide some guidelines for gathering each metric. It is
important for the evaluator to record how each metric was observed so
that comparisons can be made with data collected from multiple
experiments. Methods used to gather each metric are listed in
Section 5.2.1.

+---------+-------------------+---------------------------------------+
| **Table |                   |                                       |
| 8.      |                   |                                       |
| Me      |                   |                                       |
| trics** |                   |                                       |
+=========+===================+=======================================+
| Metric  | Definition        | Examples                              |
+---------+-------------------+---------------------------------------+
| Cou     | Any items that    | • Turns (spoken utterances, e-mail    |
| ntables | can be counted,   | messages, typed lines, etc.)          |
|         | once a clear      |                                       |
|         | definition has    | • Moves (a subtype of turn, e.g.,     |
|         | been developed.   | turns taken in a game)                |
|         | The difficulty    |                                       |
|         | with countables   | • Steps (number of high-level steps   |
|         | is in ensuring    | to produce result; number of mouse    |
|         | that the          | clicks or button presses towards      |
|         | definitions are   | accomplishing a task)                 |
|         | consistent and    |                                       |
|         | that items are    | • Trials (number of aborted attempts  |
|         | counted in the    | and successful attempts to reach      |
|         | same manner.      | goal)                                 |
|         |                   |                                       |
|         |                   | • Ideas generated                     |
|         |                   |                                       |
|         |                   | • Responses (replies to e-mail        |
|         |                   | message; responses to a question)     |
|         |                   |                                       |
|         |                   | • Cancel button presses (Note that    |
|         |                   | there could be different reasons for  |
|         |                   | this: 1) the user made a mistake, or  |
|         |                   | 2) the user changed his/her mind)     |
+---------+-------------------+---------------------------------------+
| Length  | The length or     | • Length of spoken utterance = end    |
| of turn | size of any type  | time (of utterance) - start time      |
|         | of turn. A turn   |                                       |
|         | is a single unit  | • Length of e-mail message = number   |
|         | of communication  | of lines                              |
|         | or activity.      |                                       |
|         |                   | \- or - = number of sentences         |
|         |                   |                                       |
|         |                   | \- or - = number of words             |
|         |                   |                                       |
|         |                   | \- or - = size in bytes               |
|         |                   |                                       |
|         |                   | • Length of turn in chat session =    |
|         |                   | number of sentences                   |
|         |                   |                                       |
|         |                   | \- or - = number of words             |
|         |                   |                                       |
|         |                   | • Length of move (subtype of turn) =  |
|         |                   | time at end of move --                |
|         |                   |                                       |
|         |                   | time at start of move                 |
+---------+-------------------+---------------------------------------+
| Task    | Whether or not a  | Yes or no (completion or              |
| com     | task is           | non-completion). In some cases, the   |
| pletion | completed.        | degree of completion can be measured. |
+---------+-------------------+---------------------------------------+
| Time    | Time, as a        | • Overall task execution time: the    |
|         | metric, supports  | interval from task beginning to task  |
|         | a number of       | end. This is the total time it takes  |
|         | measures. Base    | the collective group to complete      |
|         | metrics for start | execution of the task.                |
|         | and end time can  |                                       |
|         | be interpreted to | • Task time: the time spent on the    |
|         | support various   | actual task. This does not include    |
|         | measures. Time    | transition time or time spent doing   |
|         | can be measured   | non-task-related activities.          |
|         | with respect to   |                                       |
|         | an individual, as | • Transition time: the amount of time |
|         | a sum of all      | it takes the group to transition from |
|         | individual times, | one task to another. This includes    |
|         | and/or as the     | set up time.                          |
|         | longest           |                                       |
|         | individual time   | • Other time: determined by the       |
|         | in the group.     | following formula: other time =       |
|         |                   | overall time - (task time +           |
|         |                   | transition time)                      |
|         |                   |                                       |
|         |                   | • Repair activities time: the time    |
|         |                   | spent going down a wrong path in      |
|         |                   | addition to the time spent repairing  |
|         |                   | those actions. (See the definition of |
|         |                   | repair activities below. In           |
|         |                   | asynchronous tasks, repair activities |
|         |                   | time = overall time -- task time --   |
|         |                   | transition time -- idle time.         |
+---------+-------------------+---------------------------------------+
| Prep    | These metrics     | • Dollar amount.                      |
| aration | include the       |                                       |
| cost    | monetary amount   | • Learning time.                      |
| metrics | of a system and   |                                       |
|         | the learning time | • Learning cost = labor cost \* time. |
|         | for individual    |                                       |
|         | users.            |                                       |
+---------+-------------------+---------------------------------------+
| Expert  | Questions posed   | Yes/no/to what degree/quality         |
| ju      | to experts        | questions relating to:                |
| dgments |                   |                                       |
|         |                   | • Scalability                         |
|         |                   |                                       |
|         |                   | • Security                            |
|         |                   |                                       |
|         |                   | • Interoperability                    |
|         |                   |                                       |
|         |                   | • Collaboration management measures   |
|         |                   |                                       |
|         |                   | • Communication                       |
|         |                   |                                       |
|         |                   | • Collaborative object support        |
|         |                   | measures                              |
|         |                   |                                       |
|         |                   | • Transition measure                  |
|         |                   |                                       |
|         |                   | usability                             |
|         |                   |                                       |
|         |                   | product quality                       |
|         |                   |                                       |
|         |                   | task outcome                          |
|         |                   |                                       |
|         |                   | the set of tools used to accomplish   |
|         |                   | the task (the efficiency of the tools |
|         |                   | used)                                 |
+---------+-------------------+---------------------------------------+
| User    | Questions posed   | • Task outcome                        |
| ratings | to users          |                                       |
|         |                   | -- Product quality                    |
|         |                   |                                       |
|         |                   | • User satisfaction                   |
|         |                   |                                       |
|         |                   | -- Satisfaction with the group        |
|         |                   | process                               |
|         |                   |                                       |
|         |                   | -- Satisfaction with task outcome or  |
|         |                   | final solution                        |
|         |                   |                                       |
|         |                   | -- Satisfaction with an individual\'s |
|         |                   | participation                         |
|         |                   |                                       |
|         |                   | -- Satisfaction with the group        |
|         |                   | participation                         |
|         |                   |                                       |
|         |                   | • Participation of:                   |
|         |                   |                                       |
|         |                   | -- An individual                      |
|         |                   |                                       |
|         |                   | -- The group                          |
|         |                   |                                       |
|         |                   | • Efficiency                          |
|         |                   |                                       |
|         |                   | -- Efficiency of the system           |
|         |                   |                                       |
|         |                   | • Consensus on:                       |
|         |                   |                                       |
|         |                   | -- The solution                       |
|         |                   |                                       |
|         |                   | -- The task outcome                   |
|         |                   |                                       |
|         |                   | • Awareness of:                       |
|         |                   |                                       |
|         |                   | -- Other participants                 |
|         |                   |                                       |
|         |                   | -- Objects                            |
|         |                   |                                       |
|         |                   | -- Actions                            |
|         |                   |                                       |
|         |                   | • Communication                       |
|         |                   |                                       |
|         |                   | -- Whether communication was possible |
|         |                   |                                       |
|         |                   | -- The goodness of the communication  |
|         |                   |                                       |
|         |                   | -- Ability to get floor control       |
|         |                   |                                       |
|         |                   | -- Ability to ask a question / make a |
|         |                   | response                              |
|         |                   |                                       |
|         |                   | • Grounding                           |
|         |                   |                                       |
|         |                   | -- Establishing common understanding  |
|         |                   | with other participants               |
|         |                   |                                       |
|         |                   | -- Understanding what other           |
|         |                   | participants were talking about       |
|         |                   |                                       |
|         |                   | • Transition                          |
|         |                   |                                       |
|         |                   | -- Smoothness of the transition       |
|         |                   |                                       |
|         |                   | • Usability                           |
|         |                   |                                       |
|         |                   | -- Standard user interface evaluation |
|         |                   | and usability questions               |
+---------+-------------------+---------------------------------------+
| Tool    | The frequency and | • Can be measured as the deviation    |
| usage   | distribution of   | from a pre-determined "correct" tool  |
|         | tools used to     | usage established by experts. Experts |
|         | accomplish a      | can rate the use of sets of tools for |
|         | particular task.  | each subtask, and tool usage can be   |
|         | Also the way in   | measured as the deviation from those  |
|         | which tools are   | ratings.                              |
|         | used to           |                                       |
|         | accomplish a      |                                       |
|         | particular task.  |                                       |
+---------+-------------------+---------------------------------------+
| Turn    | The overlap in    | • Occurs when the start time of a     |
| overlap | communication by  | turn happens before the end time of a |
|         | two or more       | previous turn. Turn overlaps can be   |
|         | people.           | counted and categorized into          |
|         |                   | interruptions, backchannel            |
|         |                   | communications, etc.                  |
+---------+-------------------+---------------------------------------+
| Repair  | Include all       | • The repair activities metric is     |
| act     | errors, following | made up of the number of undos and    |
| ivities | down a wrong      | the number of cancels, as well as     |
|         | path, and all     | repetitive or useless keystrokes.     |
|         | actions performed |                                       |
|         | to correct those  |                                       |
|         | errors. Repair    |                                       |
|         | activities also   |                                       |
|         | include not       |                                       |
|         | knowing what to   |                                       |
|         | do and seeking    |                                       |
|         | help. In order to |                                       |
|         | have this metric, |                                       |
|         | there must be an  |                                       |
|         | established       |                                       |
|         | 'right path', or  |                                       |
|         | a list of         |                                       |
|         | possible 'right   |                                       |
|         | paths'. Experts   |                                       |
|         | determine the     |                                       |
|         | 'right paths'.    |                                       |
+---------+-------------------+---------------------------------------+
| C       | A general         | Given topic segmentation, we can      |
| onversa | category of       | measure:                              |
| -tional | metrics that      |                                       |
| con     | includes semantic | • Topic mention -- the number of      |
| structs | and grammatical   | times a topic is mentioned            |
|         | content of        |                                       |
|         | communication.    | • Distance of topic from a given turn |
|         | Topic             |                                       |
|         | segmentation and  | • Number of supportive responses      |
|         | labeling is the   | (backchannelling, explicit agreement, |
|         | ability to        | etc.)                                 |
|         | segment dialog    |                                       |
|         | into multiple,    | References can be counted by their    |
|         | distinct topics   | occurrence, type, and distribution in |
|         | with each segment | a transcript. The use of pronouns can |
|         | labeled as to its | indicate group inclusion (you, we) or |
|         | topic. A          | exclusion (I).                        |
|         | reference is the  |                                       |
|         | use of a          |                                       |
|         | grammatical       |                                       |
|         | construction      |                                       |
|         | (e.g., a definite |                                       |
|         | noun phrase or a  |                                       |
|         | pronoun) to refer |                                       |
|         | to a preceding    |                                       |
|         | word or group of  |                                       |
|         | words.            |                                       |
+---------+-------------------+---------------------------------------+

### 5.2.1Number range Level3Number range Level4 Data Collection Methods

The following is a list of methods, or data-collection instruments, used
for gathering the metrics described above:

-   Logs

-   Direct observation

-   Questionnaires/interviews/rating scales (open-ended or closed/fixed
    alternatives)

-   Tape recorder/transcription

-   Video recorder/annotation

Many of the metrics can be gathered by a variety of tools or methods.
For example, countables can be obtained by studying logs, observations,
audio transcripts, and video recordings. (Automated methods of
collecting data are discussed in Section 5.4.) Each method offers
potential opportunities not available by other means but also has
inherent limitations. For this reason, we suggest using multiple methods
to obtain each metric. The possibilities are enumerated in Table 9
below.

  ----------------------- ---------- -------------- ----------------- ----------- -----------
  **Table 9. Table of                                                             
  Metrics vs. Data                                                                
  Collection Methods**                                                            

  **Metrics**             **Logs**   Observations   Question-naires   **Audio**   **Video**

  Countables              x          x                                x           x

  Expert judgments                   x              x                             x

  Length of turns         x          x                                x           x

  Turn overlap            x                                           x           

  Resource costs          x          x              x                 x           x

  Task completion         x          x                                            

  Time                    x          x                                x           x

  Tool usage              x          x              x                             x

  User ratings                                      x                             

  Conversational          x          x                                x           x
  constructs                                                                      

  Repair activities       x          x              x                 x           x
  ----------------------- ---------- -------------- ----------------- ----------- -----------

## 5.3Number range Level2Number range Level3Number range Level4 Measures

The framework described in Section 3 of this document divides the
collaborative system design space into four levels: requirement,
capability, service, and technology. Measures are associated with each
level. For example, participation is a requirement level measure
correlated with various metrics: the total number of turns per
participant, the total number of turns per group, and user ratings. By
contrast, usability is a technology level measure.

For each measure, we present:

-   A definition

-   The "building blocks": metrics and other measures

-   Associated task types

The "building blocks" of a measure are metrics and other measures. A
building block can be a combination of metrics or arithmetic formulas
involving metrics. Note that many of the components of each measure are
examples; exact components may vary depending upon the specific
situation. Where available, we have included references to supporting
research. For some measures, we have also included metrics and measures
that might show some indirect relevance.

In general, all measures can be applied to all tasks. However, some
measures may have little or no relevance for a particular instantiation
of a task type. For example, measuring participation is usually not
important to a dissemination of information task (Type 9), but is often
very important in a brainstorming and creativity task (Type 2). With
each measure, we include a list of what we believe to be the applicable
task types that the measure helps to evaluate. The task types are
enumerated and explained in Section 3 of this document.

The requirement level measures are summarized in Table 10 and include
definitions and components. Capability level measures evaluate general
capabilities of collaborative systems (see Section 3), and are
summarized in Table 11. Service level measures evaluate the services
provided by a collaborative system; they are summarized in Table 12.
Finally, technology level measures aid in evaluating the implementation
of a collaborative system. Example technology level measures can be
found in Table 13.

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 10.       |                          |                               |
| Example   |                          |                               |
| Re        |                          |                               |
| quirement |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+===========+==========================+===============================+
| **Requ    | **Definition**           | **Metric/Measure Components** |
| irement** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Task      | Measure of the state of  | \- Countables (number of      |
| outcome   | a particular task. A set | generated artifacts);         |
|           | of artifacts is produced |                               |
|           | during task execution    | \- task completion (yes/no)   |
|           | (e.g., documents, ideas, |                               |
|           | solutions, defined       | \- expert judgments of        |
|           | processes).              | product quality               |
|           |                          |                               |
|           |                          | \- user ratings of product    |
|           |                          | quality                       |
+-----------+--------------------------+-------------------------------+
| Cost      | Measure of time invested | \- preparation cost measures  |
|           | in the system and the    | (monetary amount and learning |
|           | resources consumed in    | time)                         |
|           | executing an activity    |                               |
|           |                          | \- countables (number of      |
|           |                          | turns)                        |
|           |                          |                               |
|           |                          | \- length of turns            |
|           |                          |                               |
|           |                          | \- execution time             |
+-----------+--------------------------+-------------------------------+
| User      | Subjective measure of    | \- user ratings; e.g.,        |
| sat       | satisfaction with        | satisfaction with: group      |
| isfaction | respect to the four      | process, task outcome,        |
|           | aspects of group work.   | individual's participation,   |
|           |                          | group participation           |
+-----------+--------------------------+-------------------------------+
| Sc        | Measure of a system's    | \- time to complete           |
| alability | accommodation for larger | particular tasks versus       |
|           | or smaller group size.   | number of users               |
|           |                          |                               |
|           |                          | \- resources needed to        |
|           |                          | complete particular task      |
|           |                          | versus number of users        |
|           |                          |                               |
|           |                          | \- expert judgments - yes/no, |
|           |                          | to what degree                |
+-----------+--------------------------+-------------------------------+
| Security  | Measure of the           | \- expert judgments (yes/no   |
|           | protection of            | to a list of features, to     |
|           | information and          | what degree)                  |
|           | resources                |                               |
+-----------+--------------------------+-------------------------------+
| Interop   | Measure of how well      | \- expert judgments           |
| erability | system components work   |                               |
|           | together, sharing        | \- tool usage                 |
|           | functionality and        |                               |
|           | resources                |                               |
+-----------+--------------------------+-------------------------------+
| Part      | Measure of an            | \- countables (e.g., number   |
| icipation | individual's involvement | of sentences, number of floor |
|           | in a group activity      | turns)                        |
|           |                          |                               |
|           |                          | \- user ratings               |
+-----------+--------------------------+-------------------------------+
| E         | Measure of group and     | \- percent efficiency = (task |
| fficiency | system effectiveness and | time - repair activities      |
|           | productivity             | time) / execution time        |
|           |                          |                               |
|           |                          | \- user ratings               |
|           |                          |                               |
|           |                          | \- tool usage                 |
|           |                          |                               |
|           |                          | \- breakdown                  |
+-----------+--------------------------+-------------------------------+
| Consensus | Measure of general       | \- user ratings               |
|           | agreement or group unity |                               |
|           | in outcome               | \- grounding                  |
+-----------+--------------------------+-------------------------------+

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 11.       |                          |                               |
| Example   |                          |                               |
| C         |                          |                               |
| apability |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+===========+==========================+===============================+
| **Cap     | **Definition**           | **Metric/Measure Components** |
| ability** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Awareness | Degree of "having \...   | \- user ratings               |
|           | realization, perception, |                               |
|           | or knowledge" (Webster)  | \- conversational constructs  |
|           | of surroundings and      |                               |
|           | events.                  |                               |
+-----------+--------------------------+-------------------------------+
| Coll      | Measures assess support  | \- expert judgments           |
| aboration | for coordinating         |                               |
| m         | collaboration; e.g.,     |                               |
| anagement | floor control, agenda    |                               |
|           | support, document access |                               |
|           | control, etc.            |                               |
+-----------+--------------------------+-------------------------------+
| Commu     | Measure of the exchange  | \- countables (number of      |
| ni-cation | of information in any of | turns per participant)        |
| (human to | the following forms:     |                               |
| human)    | verbal (spoken or        | \- turn overlap (simple       |
|           | written), visual,        | overlap and interruptions)    |
|           | physical                 |                               |
|           |                          | \- expert judgments           |
|           |                          |                               |
|           |                          | \- user ratings (goodness of  |
|           |                          | communication, getting floor  |
|           |                          | control, getting other        |
|           |                          | participants' attention,      |
|           |                          | ability to interrupt)         |
+-----------+--------------------------+-------------------------------+
| Grounding | Measure of how well      | \- user ratings (e.g.,        |
|           | common understanding is  | reaching common understanding |
|           | established.             | with other participants)      |
|           |                          |                               |
|           |                          | \- countables (number of      |
|           |                          | turns, length of turns)       |
|           |                          |                               |
|           |                          | \- turn overlap               |
|           |                          |                               |
|           |                          | \- conversational constructs  |
+-----------+--------------------------+-------------------------------+
| Coll      | Measure that assesses    | \- expert judgments           |
| aborative | support for              |                               |
| object    | collaborative objects;   | \- tool usage (optimal set of |
| support   | applied to shared        | tools used?)                  |
|           | workspace, object        |                               |
|           | manipulation and         |                               |
|           | management, etc.         |                               |
+-----------+--------------------------+-------------------------------+
| Task      | Measures the ability to  | \- task focus = (overall      |
| focus     | concentrate on the task  | time - transition time -      |
|           | at hand.                 | other time) / overall time    |
+-----------+--------------------------+-------------------------------+
| T         | Assesses support for     | > \- expert judgments         |
| ransition | transitions; used to     | >                             |
| measures  | evaluate collaboration   | > \- user ratings (e.g., flow |
|           | startup, summarization,  | > of transitions between      |
|           | playback, archiving,     | > tasks)                      |
|           | object exporting and     | >                             |
|           | importing, distribution  | > \- conversational           |
|           | of objects, translation  | > constructs                  |
|           | between modalities,      |                               |
|           | meeting notification,    |                               |
|           | etc.                     |                               |
+-----------+--------------------------+-------------------------------+

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 12.       |                          |                               |
| Example   |                          |                               |
| Service   |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+-----------+--------------------------+-------------------------------+
| **        | **Definition**           | **Metric/Measure Components** |
| Service** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Breakdown | Measures how often the   | \- conversational constructs  |
|           | user has to rationalize  |                               |
|           | a problem experienced.   | \- repair activities          |
|           | Breakdowns can occur in  |                               |
|           | communication, in        |                               |
|           | coordination, in the     |                               |
|           | system components, etc.  |                               |
+-----------+--------------------------+-------------------------------+
| Tool      | The degree to which the  | \- tool usage (which tools    |
| usage     | optimal tools are used   | were used and how often?)     |
|           | for a particular task.   |                               |
|           |                          | \- expert judgments (were the |
|           |                          | right tools used?)            |
+-----------+--------------------------+-------------------------------+

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 13.       |                          |                               |
| Example   |                          |                               |
| T         |                          |                               |
| echnology |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+-----------+--------------------------+-------------------------------+
| **Tec     | **Definition**           | **Metric/Measure Components** |
| hnology** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Usability | Evaluates the ease,      | \- expert judgments (standard |
|           | accessibility, and       | user interface evaluation     |
|           | intuitiveness of the     | questions)                    |
|           | specific graphical user  |                               |
|           | interfaces of the system | \- user ratings (standard     |
|           | tools and components.    | user interface evaluation     |
|           | Since usability          | questions)                    |
|           | evaluation is done on    |                               |
|           | specific user            | \- tool usage (which tools    |
|           | interfaces, the          | were used and how often? Was  |
|           | usability measures are   | the set of optimal tools      |
|           | realized at the          | used?)                        |
|           | technology level         |                               |
|           | although the component   | \- repair activities          |
|           | measures are also based, |                               |
|           | in part, on measures     | \- breakdown (defined at the  |
|           | taken at different       | service level)                |
|           | levels of the framework. |                               |
|           |                          | \- awareness (defined at the  |
|           |                          | capability level)             |
+-----------+--------------------------+-------------------------------+
| Specific  |                          | \- tool usage (which tools    |
| t         |                          | were used and how often?)     |
| echnology |                          |                               |
|           |                          | \- expert judgments (were the |
|           |                          | right tools used?)            |
+-----------+--------------------------+-------------------------------+

## 5.4Number range Level2Number range Level3Number range Level4 Data Collection Methods: Logging

We often need to collect data about running systems. Since videotaping
every session can be difficult, automatic logging can be important in
developing easy, repeatable, evaluation scenarios. A multi-modal logging
capability can support these efforts.

The MITRE Multi-Modal Logger (MML) is an example of this class of
logging tool. MITRE's MML can be used to log multi-modal data while
collaborative systems are running. This data can be fine-grained
(individual X or window events, for instance) or coarse-grained (a
record of which windows the user interacts with); automatically gathered
(via instrumentation) or manually created (via post-hoc annotation).
This data can be gathered at the level of the physical or virtual device
(for example, window events or the output of an audio device); at the
level of the interface (for example, a record of menu selections made
and the content of text entry fields); or at the level of the
application (a record of actions taken, such as a retrieval of
information or a command issued). This information can also be in a
variety of modalities, such as text, images, and audio.

Since this information will typically be gathered for multiple users and
multiple interactions with the system in question, the notion of a
"trial" or "session" is supported. In addition, each trial might require
information to be gathered from multiple components simultaneously (for
example, when a speech recognizer is used in conjunction with an
independent multi-modal system). Therefore, the MML also supports
sharing each trial among multiple components, potentially running on
different hosts.

MITRE\'s MML provides an application programming interface (API) for
instrumenting existing applications. It also provides a set of tools for
reviewing and annotating data collected via instrumentation.

MITRE's MML offers a solution to the question of what granularity and
levels of data may be collected. The instrumenter inserts whatever
logger calls are desired into the source code and is thus in complete
control of where, in the code, the log entries are generated, how many
are generated, and what data types are assigned to them. In general, one
can instrument any application for which the source code is available.

MITRE has also developed a log review and annotation tool to distribute
with the logger. It allows users to view the data that has been logged
for a given session. One may view all the data or select a subset by
application, data type and/or timestamp. The data is displayed along a
scrollable timeline, with the data sorted into streams by application
and data type. This tool may also be used to add post-hoc annotations to
the data. There is also a replay facility that allows the reviewer to
replay the logged interaction in approximately real time.

The MITRE MML tool suite and documentation are available for downloading
at http://www.mitre.org/technology/logger.

# Section 6Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-6number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading}

# Using the Methodology to Design an Experiment {#using-the-methodology-to-design-an-experiment .Section-Header}

## 6.1Number range Level2Number range Level3Number range Level4 Introduction

This section concentrates on the way that the methodology described in
this paper influences experimental design. Note that the basics of
experimental design will not be covered, but can be found in such
references as Pfleeger (1994). Also, we will not give complete
descriptions of experiments here because they are not needed to
understand how to apply the methodology. (The interested reader can find
a description of an experiment designed with the help of the methodology
in Damianos et. al. 1999.)

## 6.2Number range Level2Number range Level3Number range Level4 Designing an Experiment

To use the methodology, the experimenters need to decide upon a goal.
For example, if the users have a particular set of requirements and the
experimenters are charged with finding which system best meets those
requirements, the experimenters could follow the steps in Table 14.

+------+-----------------+---------------------------------------------+
| #### |                 |                                             |
| ## T |                 |                                             |
| able |                 |                                             |
|  14. |                 |                                             |
|  Exa |                 |                                             |
| mple |                 |                                             |
|  Ste |                 |                                             |
| ps f |                 |                                             |
| or a |                 |                                             |
|  Top |                 |                                             |
| -Dow |                 |                                             |
| n Ev |                 |                                             |
| alua |                 |                                             |
| tion |                 |                                             |
+======+=================+=============================================+
| **St | **Title**       | ###### Description                          |
| ep** |                 |                                             |
+------+-----------------+---------------------------------------------+
| 1    | Identify        | Given a set of requirements, either look at |
|      | candidate       | a checklist of system capabilities similar  |
|      | systems.        | to what is included in table 5 of this      |
|      |                 | document or, if not available, create a     |
|      |                 | similar table that includes the             |
|      |                 | capabilities of interest. The result should |
|      |                 | be a small set of systems that best meet    |
|      |                 | the basic requirements.                     |
+------+-----------------+---------------------------------------------+
| 2    | Determine       | Determine what tasks will need to be        |
|      | tasks.          | performed with the systems and abstract     |
|      |                 | these tasks into the task types described   |
|      |                 | in section 3.                               |
+------+-----------------+---------------------------------------------+
| 3    | Select (or      | Examine a scenario library to see which     |
|      | create/modify)  | scenarios support evaluation of the desired |
|      | a scenario.     | task types. If an appropriate scenario      |
|      |                 | cannot be found, create a scenario (see     |
|      |                 | section 4). The scenario should be targeted |
|      |                 | at the desired evaluation level (e.g.,      |
|      |                 | capability, service, and/or technology).    |
|      |                 | For example, if users need to check out and |
|      |                 | store documents, experimenters might run    |
|      |                 | the scenario at the service level, meaning  |
|      |                 | that several different services for         |
|      |                 | document handling are exercised. Make sure  |
|      |                 | the scenario includes the salient           |
|      |                 | characteristics of their specific tasks; if |
|      |                 | not, tailor the scenario accordingly.       |
+------+-----------------+---------------------------------------------+
| 4    | Determine       | Choose appropriate measures based on the    |
|      | measures.       | type of tasks used in the evaluation, the   |
|      |                 | desired evaluation levels, the hypotheses   |
|      |                 | being tested, and observation resources     |
|      |                 | available (e.g., automated loggers).        |
+------+-----------------+---------------------------------------------+
| 5    | Pilot the       | Run the experiment on a representative,     |
|      | experiment.     | balanced group of users to make sure the    |
|      |                 | appropriate data are collected.             |
+------+-----------------+---------------------------------------------+
| 6    | Run the         | Repeat the scenario for each system or      |
|      | experiment      | subsystem under consideration.              |
+------+-----------------+---------------------------------------------+

The approach just described corresponds to a top-down evaluation,
beginning with the requirements level and moving to the technology
level. Alternately, the experimenters might be given an alpha version of
a new system and asked to determine the tasks for which it will be
useful. This corresponds to a bottom-up approach, starting with the
specific technology and determining the services, capabilities, and
requirements it may support. The experimenters could follow the steps
outlined below in Table 15.

+------+-----------------+---------------------------------------------+
| #    |                 |                                             |
| #### |                 |                                             |
| # Ta |                 |                                             |
| ble  |                 |                                             |
| 15.  |                 |                                             |
| Exam |                 |                                             |
| ple  |                 |                                             |
| Step |                 |                                             |
| s fo |                 |                                             |
| r a  |                 |                                             |
| Bott |                 |                                             |
| om-U |                 |                                             |
| p Ev |                 |                                             |
| alua |                 |                                             |
| tion |                 |                                             |
+======+=================+=============================================+
| **St | **Title**       | ###### Description                          |
| ep** |                 |                                             |
+------+-----------------+---------------------------------------------+
| 1    | Compile a list  | Create a list of basic services provided by |
|      | of services.    | the system.                                 |
+------+-----------------+---------------------------------------------+
| 2    | Determine       | Based on the services the system supports,  |
|      | tasks.          | determine what capabilities this system     |
|      |                 | will provide. Based on this, devise a list  |
|      |                 | of tasks the intended users might perform   |
|      |                 | which would be supported by these services. |
|      |                 | These tasks should be expressed in terms of |
|      |                 | the work tasks described earlier.           |
+------+-----------------+---------------------------------------------+
| 3    | Select (or      | Examine a scenario library to see which     |
|      | create/tailor)  | scenarios support evaluation of the desired |
|      | a scenario.     | work task types. If an appropriate scenario |
|      |                 | cannot be found, create a scenario. Make    |
|      |                 | sure the scenario includes the salient      |
|      |                 | characteristics of their specific tasks; if |
|      |                 | not, tailor the scenario accordingly.       |
+------+-----------------+---------------------------------------------+
| 4    | Determine       | Using those listed in section 5 as a        |
|      | measures.       | starting point, choose appropriate measures |
|      |                 | based on the type of tasks used in the      |
|      |                 | evaluation, the desired evaluation levels   |
|      |                 | (e.g., service, capabilities, and           |
|      |                 | requirements), the hypotheses being tested, |
|      |                 | and the observation resources available.    |
+------+-----------------+---------------------------------------------+
| 5    | Set criteria.   | Set minimum accepted thresholds for those   |
|      |                 | measures, based on common sense, "typical"  |
|      |                 | standards that are currently generally      |
|      |                 | accepted.                                   |
+------+-----------------+---------------------------------------------+
| 6    | Pilot the       | Run the experiment on a representative,     |
|      | experiment.     | balanced group of users to make sure the    |
|      |                 | appropriate data are collected.             |
+------+-----------------+---------------------------------------------+
| 7    | Run the         | Enact the scenario for the system being     |
|      | scenario.       | evaluated to ensure that the system meets   |
|      |                 | minimum acceptable thresholds for providing |
|      |                 | those services, capabilities, and/or        |
|      |                 | requirements.                               |
+------+-----------------+---------------------------------------------+

## 6.3Number range Level2Number range Level3Number range Level4 An Example: The Map Navigation Experiment

As an example of how an experiment could be designed using the
methodology, consider a task where two people need to collaborate to
share route information. The will need to agree on the best driving
route to use given specific map information, and communicate the
information to another party. This task may be typical of a military
mission, where ground-based units must collaborate to share information
on how to avoid enemy locations and booby-traps when moving troops.

Suppose that the military planners use MITRE's CVW to perform these
types of route planning tasks. Making audio available in CVW under field
conditions is difficult, so military planners might be interested in
determining whether audio significantly enhances task performance. This
research question can be expressed as the following hypotheses:

> 1\. People collaboratively plan a route faster when audio
> communication is available.
>
> 2\. People collaboratively plan a better route when audio
> communication is available.
>
> 3\. Participants will be more satisfied with collaborative route
> planning when audio communication is available.

These hypotheses can be evaluated as part of a top-down evaluation at
the service level. Top-down evaluations begin with requirements, which
in this case are the need to exchange information and make plans (step 1
in table 14). These requirements correspond to the generic task types of
planning and information dissemination (step 2).

The experimenters could search the EWG scenario repository, and if no
appropriate scenarios are available, create a scenario (step 3). The
scenario should spring from a readily accessible problem domain, in this
case, collaborative problem solving. The experimenters could craft the
scenario to be readily familiar to a wide range of potential test
subjects: route finding using a street map. The scenario would require
information sharing and collaborative planning by providing each
participant with private information. The scenario could be expressed
this way:

> A frantic call comes in. Colleague #1 is late for an important meeting
> across town. You know of a few obstacles such as road construction
> blockages between Colleague #1 and her meeting location, but Colleague
> #2 has the latest report from the local news radio. Together, you and
> Colleague #2 must devise the quickest route to the meeting. You must
> agree on the route and inform Colleague #1.

Thus, two people must work together to determine the quickest and best
route between two locations. Each would have a copy of the same map, but
each would also have additional (not shared) information about obstacles
in the route (for example, annotations indicating heavy traffic, one-way
roads, construction sites, and turning restrictions). A time constraint
would be implied; the participants could be told that their colleague
was in a hurry to get to a meeting and was awaiting directions. The work
would be completed once both participants had agreed on a route. The
scenario could be tailored (step 3) to ensure that this exercise
reflects a planning task done under time constraints, along with the
need to pool information and provide the results to non-collocated
colleagues.

Metrics should be based on the hypotheses being tested (step 4).
Material presented in Section 5 of this document provides ideas about
how and what to measure. The metrics presented in Table 16 could be
used.

####### Table 16. Metrics and Corresponding Hypotheses

  ----------------------------------------------- -----------------------
  **Metric**                                      **Hypothesis tested**

  Overall task time                               1

  Expert judgment/quality of route                2

  User questionnaire                              3
  ----------------------------------------------- -----------------------

After performing pilot experiments and refining the experimental design,
the experiment should be run under both audio and non-audio conditions
(step 6).

A similar experiment was performed as an initial validation of this
methodology. It is described on the IC&V web site at
http://www.antd.nist.gov/\~icv-ewg/experiments/mapnav/mapnav.html.

# Bibliography {#bibliography .Back-Matter-Header}

Bass, L. (1996). "Mini-workshop: scenarios for CSCW systems." In
Engineering for

Human-Computer Interaction: proceedings of the IFIP TC2/WG2.7 working
conference on engineering for human-computer interaction, Yellowstone
Park, USA, August 1995, editors Bass, L. J. and Unger, C. Champman &
Hall, 1996, pp. 333-338.

Damianos, L., Hirschman, L., Kozierok, R. and Kurtz, J. (1999). The
MITRE Map Navigation Experiment. ACM Computing Surveys, March 1999
issue.

Grudin, J. (1988). Why CSCW Applications Fail: Problems in the Design
and Evaluation of Organizational Interfaces, in Proceedings of ACM
CSCW\'88 Conference on Computer-Supported Cooperative Work, Perspectives
on Evaluation, pp. 85-93.

McGrath, J. E. (1984), "Groups: Interaction and Performance," Englewood
Cliffs, N. J.,

Prentice-Hall.

Nielsen, J. and Molich, R. (1990). [Heuristic Evaluation of User
Interfaces]{.underline}. CHI 1990 Conference on Human Factors in
Computing, Association of Computing Machinery.

Open Software Foundation (1993). [OSF/Motif Style Guide]{.underline}.
Englewood Cliffs, NJ, PTR Prentice-Hall.

Pfleeger, S. (1994). "Experimental Design and Analysis in Software
Engineering, Part 1." ACM SIFSOFT Engineering Notes, vol 19, no. 4. Also
parts 2 -- 5 in vol. 20, nos. 1, 2, 3, and 5.

Pinsonneault, A. and Kraemer, K. (1993). "The Impact of Technological
Support on Groups: An Assessment of the Empirical Research" in Readings
in Groupware and Computer Supported Cooperative Work, edited by Baecker,
pp. 754-773.

Polson, P. G., C. Lewis, et al. (1992). "Cognitive Walkthroughs: Method
for Theory-Based Evaluation of User Interfaces." [International Journal
of Man-Machine Studies]{.underline} vol. 36, no. 5.

Nardi, B. A. (1995). "Some reflections on scenarios." In Scenario Based
Design: Envisioning Work and Technology in System Development, edited by
Carroll, J. M. pp. 397-399.

Potts, C. (1995). "Using schematic scenarios to understand user needs,"
in DIS95, Ann Arbor, MI, pp. 247-256.

Tsai, Y. (1977). "Hierarchical structure of participation in natural
groups," Behavioral Science, vol. 22, pp. 38-40.

Villegas, H. and Williams, M. (1997). [Benchmarks for Workspace
Awareness in Collaborative Environments]{.underline}. World
Multiconference on Systemics, Cybernetics and Informatics, Caracas,
Venezuela, International Institute of Informatics and Systemics.

Watts, L., Monk, A., and Daly-Jones, O. (1996). "Inter-Personal
Awareness and Synchronization: Assessing the Value of Communication
Technologies." International Journal of Human-Computer Studies, vol. 44,
no. 6, pp. 849-873.

Webster\'s Ninth New Collegiate Dictionary (1985). Merriam-Webster Inc.

[^1]: This original IC&V Evaluation Working Group, had members from CMU,
    focused on the effect of computer tools on group problem solving and
    communication, especially collaborative writing; the MITRE
    Corporation, focused on logging and data collection for evaluation
    and training; NIMA, focused on the testing of methodologies and
    tools for visualization, networking, and distributed systems; NIST,
    focused on evaluation methodology and infrastructure, and Amerind
    (PI Lloyd Brodsky), focused on system dynamics modeling for
    evaluation of collaborative planning systems. Lynette Hirschman,
    MITRE, chaired the EWG. The first draft of this report was written
    by researchers from NIST and MITRE. This updated document Revision
    4.0 was written by a team at MITRE, consisting of Laurie Damianos,
    Jill Drury, Tari Fanderclai, Jeff Kurtz, Lynette Hirschman, and
    Frank Linton.