# Section 1Number range Level1Number range Level2Number range Level3Number range Level4Number range FigureNumber range Table {#section-1number-range-level1number-range-level2number-range-level3number-range-level4number-range-figurenumber-range-table .Section-Heading .unnumbered}

# Metrics and Measures {#metrics-and-measures .Section-Header}

## 1.1Number range Level2Number range Level3Number range Level4 Introduction

In the preceding sections, we presented a framework for classifying
collaborative systems and an approach to evaluation that employs
scenarios. Here we enumerate the metrics and measures introduced in
Section 3 and discuss techniques for gathering the metrics.

Metrics are indicators of system, user, and group performance that can
be observed, singly or collectively, while executing scenarios. Metrics
-- such as time, length of turn, and other countable events -- are
directly measurable and can often be collected automatically.

Measures can be taken at each of the four levels of the collaborative
framework. Measures are derived from interpretations of one or more
metrics. For example, task completion time (a requirement-level measure)
is based on start and end time metrics. For asynchronous tasks, it may
be useful to distinguish time on task from elapsed time. A measure can
also be a combination of interpreted metrics and other measures. A
complicated measure, like efficiency, is partially derived from the
interpretation of metrics such as time, as well as user ratings and tool
usage. In addition, measures of system breakdown (taken at the service
or technology levels) contribute to efficiency.

A simple way to distinguish between metrics and measures is by the
following statement: a metric is an observable value, while a measure
associates meaning to that value by applying human judgment.

As another example of the metric and measure relationship, consider the
turn overlap metric. Overlap of speaker A by speaker B can be counted if
the start time of speaker B is greater than the start time of speaker A
but less than the end time of speaker A. Further interpretation is
required to determine if a particular occurrence of turn overlap is an
attempt to gain the floor (interruption in communication) or a
back-channel response indicating attentiveness (support, grounding).
This might be an interesting measure at the capability level, to
understand the ease and effectiveness of multi-person conversational
interaction.

The metrics and measures described here are relevant to laboratory
experiments. The goal of the Evaluation Working Group is to define
inexpensive evaluation technology that can be reproduced in a
laboratory. Hence, the measures do not address organizational impact and
other measures that require fielding production-quality systems.

Experiments involving human subjects, or sets of subjects, are expensive
and time consuming, both in terms of collection and analysis. In many
cases, the measures must be developed and validated before they can be
applied with confidence. Despite these difficulties, we describe options
for evaluating collaborative systems at all four levels of the framework
discussed in Section 3.

We begin with an overview of the methods, metrics and measures, and then
present the measures found at each level of the framework. Metrics are
introduced first because they are components of measures, including
methods used to collect metrics. Next we discuss measures, before
turning to automated logging techniques that may aid data collection and
analysis.

Figure 2 illustrates the relationship of data collection methods,
metrics, measures, and human judgment. The diagram also shows how the
measures of the four levels of the collaborative framework are nested,
emphasizing the mappings between these levels. As illustrated at the
bottom of the diagram, we use data collection methods, such as a logging
tool or video taping, to gather metrics on a system (the next level up).
Metrics provide the raw input necessary for refining measures for
requirements, capabilities, services, and technologies. Human
judgements, from both experts and users are associated with all four
levels.

![](media/image1.wmf)

###### Figure 1. Overview of the Levels of Measures

Metrics based on human judgment also support measures for making
judgments about the system. Questions to both experts and users of the
system can provide valuable data points in the evaluation of a system.

## 1.2Number range Level2Number range Level3Number range Level4 Metrics

The following table of metrics comprises the observable data elements
that were identified as useful indicators of system and group
performance. A single metric is just a number (e.g., number of turns) or
a set of numbers (e.g., start time and end time). Metrics can also be
used in combinations (e.g., the repair activities metric is partially
built up from the number of undos and the number of cancels and is the
deviation from some established 'right path'). For each metric, we
present in Table 8 definitions and examples of ways to capture the
metric. Where applicable, the metrics are enumerated or are broken down
into finer granularity.

We attempt to provide some guidelines for gathering each metric. It is
important for the evaluator to record how each metric was observed so
that comparisons can be made with data collected from multiple
experiments. Methods used to gather each metric are listed in
Section 5.2.1.

+---------+-------------------+---------------------------------------+
| **T     |                   |                                       |
| able 1. |                   |                                       |
| Me      |                   |                                       |
| trics** |                   |                                       |
+=========+===================+=======================================+
| Metric  | Definition        | Examples                              |
+---------+-------------------+---------------------------------------+
| Cou     | Any items that    | • Turns (spoken utterances, e-mail    |
| ntables | can be counted,   | messages, typed lines, etc.)          |
|         | once a clear      |                                       |
|         | definition has    | • Moves (a subtype of turn, e.g.,     |
|         | been developed.   | turns taken in a game)                |
|         | The difficulty    |                                       |
|         | with countables   | • Steps (number of high-level steps   |
|         | is in ensuring    | to produce result; number of mouse    |
|         | that the          | clicks or button presses towards      |
|         | definitions are   | accomplishing a task)                 |
|         | consistent and    |                                       |
|         | that items are    | • Trials (number of aborted attempts  |
|         | counted in the    | and successful attempts to reach      |
|         | same manner.      | goal)                                 |
|         |                   |                                       |
|         |                   | • Ideas generated                     |
|         |                   |                                       |
|         |                   | • Responses (replies to e-mail        |
|         |                   | message; responses to a question)     |
|         |                   |                                       |
|         |                   | • Cancel button presses (Note that    |
|         |                   | there could be different reasons for  |
|         |                   | this: 1) the user made a mistake, or  |
|         |                   | 2) the user changed his/her mind)     |
+---------+-------------------+---------------------------------------+
| Length  | The length or     | • Length of spoken utterance = end    |
| of turn | size of any type  | time (of utterance) - start time      |
|         | of turn. A turn   |                                       |
|         | is a single unit  | • Length of e-mail message = number   |
|         | of communication  | of lines                              |
|         | or activity.      |                                       |
|         |                   | \- or - = number of sentences         |
|         |                   |                                       |
|         |                   | \- or - = number of words             |
|         |                   |                                       |
|         |                   | \- or - = size in bytes               |
|         |                   |                                       |
|         |                   | • Length of turn in chat session =    |
|         |                   | number of sentences                   |
|         |                   |                                       |
|         |                   | \- or - = number of words             |
|         |                   |                                       |
|         |                   | • Length of move (subtype of turn) =  |
|         |                   | time at end of move --                |
|         |                   |                                       |
|         |                   | time at start of move                 |
+---------+-------------------+---------------------------------------+
| Task    | Whether or not a  | Yes or no (completion or              |
| com     | task is           | non-completion). In some cases, the   |
| pletion | completed.        | degree of completion can be measured. |
+---------+-------------------+---------------------------------------+
| Time    | Time, as a        | • Overall task execution time: the    |
|         | metric, supports  | interval from task beginning to task  |
|         | a number of       | end. This is the total time it takes  |
|         | measures. Base    | the collective group to complete      |
|         | metrics for start | execution of the task.                |
|         | and end time can  |                                       |
|         | be interpreted to | • Task time: the time spent on the    |
|         | support various   | actual task. This does not include    |
|         | measures. Time    | transition time or time spent doing   |
|         | can be measured   | non-task-related activities.          |
|         | with respect to   |                                       |
|         | an individual, as | • Transition time: the amount of time |
|         | a sum of all      | it takes the group to transition from |
|         | individual times, | one task to another. This includes    |
|         | and/or as the     | set up time.                          |
|         | longest           |                                       |
|         | individual time   | • Other time: determined by the       |
|         | in the group.     | following formula: other time =       |
|         |                   | overall time - (task time +           |
|         |                   | transition time)                      |
|         |                   |                                       |
|         |                   | • Repair activities time: the time    |
|         |                   | spent going down a wrong path in      |
|         |                   | addition to the time spent repairing  |
|         |                   | those actions. (See the definition of |
|         |                   | repair activities below. In           |
|         |                   | asynchronous tasks, repair activities |
|         |                   | time = overall time -- task time --   |
|         |                   | transition time -- idle time.         |
+---------+-------------------+---------------------------------------+
| Prep    | These metrics     | • Dollar amount.                      |
| aration | include the       |                                       |
| cost    | monetary amount   | • Learning time.                      |
| metrics | of a system and   |                                       |
|         | the learning time | • Learning cost = labor cost \* time. |
|         | for individual    |                                       |
|         | users.            |                                       |
+---------+-------------------+---------------------------------------+
| Expert  | Questions posed   | Yes/no/to what degree/quality         |
| ju      | to experts        | questions relating to:                |
| dgments |                   |                                       |
|         |                   | • Scalability                         |
|         |                   |                                       |
|         |                   | • Security                            |
|         |                   |                                       |
|         |                   | • Interoperability                    |
|         |                   |                                       |
|         |                   | • Collaboration management measures   |
|         |                   |                                       |
|         |                   | • Communication                       |
|         |                   |                                       |
|         |                   | • Collaborative object support        |
|         |                   | measures                              |
|         |                   |                                       |
|         |                   | • Transition measure                  |
|         |                   |                                       |
|         |                   | usability                             |
|         |                   |                                       |
|         |                   | product quality                       |
|         |                   |                                       |
|         |                   | task outcome                          |
|         |                   |                                       |
|         |                   | the set of tools used to accomplish   |
|         |                   | the task (the efficiency of the tools |
|         |                   | used)                                 |
+---------+-------------------+---------------------------------------+
| User    | Questions posed   | • Task outcome                        |
| ratings | to users          |                                       |
|         |                   | -- Product quality                    |
|         |                   |                                       |
|         |                   | • User satisfaction                   |
|         |                   |                                       |
|         |                   | -- Satisfaction with the group        |
|         |                   | process                               |
|         |                   |                                       |
|         |                   | -- Satisfaction with task outcome or  |
|         |                   | final solution                        |
|         |                   |                                       |
|         |                   | -- Satisfaction with an individual\'s |
|         |                   | participation                         |
|         |                   |                                       |
|         |                   | -- Satisfaction with the group        |
|         |                   | participation                         |
|         |                   |                                       |
|         |                   | • Participation of:                   |
|         |                   |                                       |
|         |                   | -- An individual                      |
|         |                   |                                       |
|         |                   | -- The group                          |
|         |                   |                                       |
|         |                   | • Efficiency                          |
|         |                   |                                       |
|         |                   | -- Efficiency of the system           |
|         |                   |                                       |
|         |                   | • Consensus on:                       |
|         |                   |                                       |
|         |                   | -- The solution                       |
|         |                   |                                       |
|         |                   | -- The task outcome                   |
|         |                   |                                       |
|         |                   | • Awareness of:                       |
|         |                   |                                       |
|         |                   | -- Other participants                 |
|         |                   |                                       |
|         |                   | -- Objects                            |
|         |                   |                                       |
|         |                   | -- Actions                            |
|         |                   |                                       |
|         |                   | • Communication                       |
|         |                   |                                       |
|         |                   | -- Whether communication was possible |
|         |                   |                                       |
|         |                   | -- The goodness of the communication  |
|         |                   |                                       |
|         |                   | -- Ability to get floor control       |
|         |                   |                                       |
|         |                   | -- Ability to ask a question / make a |
|         |                   | response                              |
|         |                   |                                       |
|         |                   | • Grounding                           |
|         |                   |                                       |
|         |                   | -- Establishing common understanding  |
|         |                   | with other participants               |
|         |                   |                                       |
|         |                   | -- Understanding what other           |
|         |                   | participants were talking about       |
|         |                   |                                       |
|         |                   | • Transition                          |
|         |                   |                                       |
|         |                   | -- Smoothness of the transition       |
|         |                   |                                       |
|         |                   | • Usability                           |
|         |                   |                                       |
|         |                   | -- Standard user interface evaluation |
|         |                   | and usability questions               |
+---------+-------------------+---------------------------------------+
| Tool    | The frequency and | • Can be measured as the deviation    |
| usage   | distribution of   | from a pre-determined "correct" tool  |
|         | tools used to     | usage established by experts. Experts |
|         | accomplish a      | can rate the use of sets of tools for |
|         | particular task.  | each subtask, and tool usage can be   |
|         | Also the way in   | measured as the deviation from those  |
|         | which tools are   | ratings.                              |
|         | used to           |                                       |
|         | accomplish a      |                                       |
|         | particular task.  |                                       |
+---------+-------------------+---------------------------------------+
| Turn    | The overlap in    | • Occurs when the start time of a     |
| overlap | communication by  | turn happens before the end time of a |
|         | two or more       | previous turn. Turn overlaps can be   |
|         | people.           | counted and categorized into          |
|         |                   | interruptions, backchannel            |
|         |                   | communications, etc.                  |
+---------+-------------------+---------------------------------------+
| Repair  | Include all       | • The repair activities metric is     |
| act     | errors, following | made up of the number of undos and    |
| ivities | down a wrong      | the number of cancels, as well as     |
|         | path, and all     | repetitive or useless keystrokes.     |
|         | actions performed |                                       |
|         | to correct those  |                                       |
|         | errors. Repair    |                                       |
|         | activities also   |                                       |
|         | include not       |                                       |
|         | knowing what to   |                                       |
|         | do and seeking    |                                       |
|         | help. In order to |                                       |
|         | have this metric, |                                       |
|         | there must be an  |                                       |
|         | established       |                                       |
|         | 'right path', or  |                                       |
|         | a list of         |                                       |
|         | possible 'right   |                                       |
|         | paths'. Experts   |                                       |
|         | determine the     |                                       |
|         | 'right paths'.    |                                       |
+---------+-------------------+---------------------------------------+
| C       | A general         | Given topic segmentation, we can      |
| onversa | category of       | measure:                              |
| -tional | metrics that      |                                       |
| con     | includes semantic | • Topic mention -- the number of      |
| structs | and grammatical   | times a topic is mentioned            |
|         | content of        |                                       |
|         | communication.    | • Distance of topic from a given turn |
|         | Topic             |                                       |
|         | segmentation and  | • Number of supportive responses      |
|         | labeling is the   | (backchannelling, explicit agreement, |
|         | ability to        | etc.)                                 |
|         | segment dialog    |                                       |
|         | into multiple,    | References can be counted by their    |
|         | distinct topics   | occurrence, type, and distribution in |
|         | with each segment | a transcript. The use of pronouns can |
|         | labeled as to its | indicate group inclusion (you, we) or |
|         | topic. A          | exclusion (I).                        |
|         | reference is the  |                                       |
|         | use of a          |                                       |
|         | grammatical       |                                       |
|         | construction      |                                       |
|         | (e.g., a definite |                                       |
|         | noun phrase or a  |                                       |
|         | pronoun) to refer |                                       |
|         | to a preceding    |                                       |
|         | word or group of  |                                       |
|         | words.            |                                       |
+---------+-------------------+---------------------------------------+

### 1.2.1Number range Level3Number range Level4 Data Collection Methods

The following is a list of methods, or data-collection instruments, used
for gathering the metrics described above:

Logs

Direct observation

Questionnaires/interviews/rating scales (open-ended or closed/fixed
alternatives)

Tape recorder/transcription

Video recorder/annotation

Many of the metrics can be gathered by a variety of tools or methods.
For example, countables can be obtained by studying logs, observations,
audio transcripts, and video recordings. (Automated methods of
collecting data are discussed in Section 5.4.) Each method offers
potential opportunities not available by other means but also has
inherent limitations. For this reason, we suggest using multiple methods
to obtain each metric. The possibilities are enumerated in Table 9
below.

  ----------------------- ---------- -------------- ----------------- ----------- -----------
  **Table 2. Table of                                                             
  Metrics vs. Data                                                                
  Collection Methods**                                                            

  **Metrics**             **Logs**   Observations   Question-naires   **Audio**   **Video**

  Countables              x          x                                x           x

  Expert judgments                   x              x                             x

  Length of turns         x          x                                x           x

  Turn overlap            x                                           x           

  Resource costs          x          x              x                 x           x

  Task completion         x          x                                            

  Time                    x          x                                x           x

  Tool usage              x          x              x                             x

  User ratings                                      x                             

  Conversational          x          x                                x           x
  constructs                                                                      

  Repair activities       x          x              x                 x           x
  ----------------------- ---------- -------------- ----------------- ----------- -----------

## 1.3Number range Level2Number range Level3Number range Level4 Measures

The framework described in Section 3 of this document divides the
collaborative system design space into four levels: requirement,
capability, service, and technology. Measures are associated with each
level. For example, participation is a requirement level measure
correlated with various metrics: the total number of turns per
participant, the total number of turns per group, and user ratings. By
contrast, usability is a technology level measure.

For each measure, we present:

A definition

The "building blocks": metrics and other measures

Associated task types

The "building blocks" of a measure are metrics and other measures. A
building block can be a combination of metrics or arithmetic formulas
involving metrics. Note that many of the components of each measure are
examples; exact components may vary depending upon the specific
situation. Where available, we have included references to supporting
research. For some measures, we have also included metrics and measures
that might show some indirect relevance.

In general, all measures can be applied to all tasks. However, some
measures may have little or no relevance for a particular instantiation
of a task type. For example, measuring participation is usually not
important to a dissemination of information task (Type 9), but is often
very important in a brainstorming and creativity task (Type 2). With
each measure, we include a list of what we believe to be the applicable
task types that the measure helps to evaluate. The task types are
enumerated and explained in Section 3 of this document.

The requirement level measures are summarized in Table 10 and include
definitions and components. Capability level measures evaluate general
capabilities of collaborative systems (see Section 3), and are
summarized in Table 11. Service level measures evaluate the services
provided by a collaborative system; they are summarized in Table 12.
Finally, technology level measures aid in evaluating the implementation
of a collaborative system. Example technology level measures can be
found in Table 13.

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 3.        |                          |                               |
| Example   |                          |                               |
| Re        |                          |                               |
| quirement |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+===========+==========================+===============================+
| **Requ    | **Definition**           | **Metric/Measure Components** |
| irement** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Task      | Measure of the state of  | \- Countables (number of      |
| outcome   | a particular task. A set | generated artifacts);         |
|           | of artifacts is produced |                               |
|           | during task execution    | \- task completion (yes/no)   |
|           | (e.g., documents, ideas, |                               |
|           | solutions, defined       | \- expert judgments of        |
|           | processes).              | product quality               |
|           |                          |                               |
|           |                          | \- user ratings of product    |
|           |                          | quality                       |
+-----------+--------------------------+-------------------------------+
| Cost      | Measure of time invested | \- preparation cost measures  |
|           | in the system and the    | (monetary amount and learning |
|           | resources consumed in    | time)                         |
|           | executing an activity    |                               |
|           |                          | \- countables (number of      |
|           |                          | turns)                        |
|           |                          |                               |
|           |                          | \- length of turns            |
|           |                          |                               |
|           |                          | \- execution time             |
+-----------+--------------------------+-------------------------------+
| User      | Subjective measure of    | \- user ratings; e.g.,        |
| sat       | satisfaction with        | satisfaction with: group      |
| isfaction | respect to the four      | process, task outcome,        |
|           | aspects of group work.   | individual's participation,   |
|           |                          | group participation           |
+-----------+--------------------------+-------------------------------+
| Sc        | Measure of a system's    | \- time to complete           |
| alability | accommodation for larger | particular tasks versus       |
|           | or smaller group size.   | number of users               |
|           |                          |                               |
|           |                          | \- resources needed to        |
|           |                          | complete particular task      |
|           |                          | versus number of users        |
|           |                          |                               |
|           |                          | \- expert judgments - yes/no, |
|           |                          | to what degree                |
+-----------+--------------------------+-------------------------------+
| Security  | Measure of the           | \- expert judgments (yes/no   |
|           | protection of            | to a list of features, to     |
|           | information and          | what degree)                  |
|           | resources                |                               |
+-----------+--------------------------+-------------------------------+
| Interop   | Measure of how well      | \- expert judgments           |
| erability | system components work   |                               |
|           | together, sharing        | \- tool usage                 |
|           | functionality and        |                               |
|           | resources                |                               |
+-----------+--------------------------+-------------------------------+
| Part      | Measure of an            | \- countables (e.g., number   |
| icipation | individual's involvement | of sentences, number of floor |
|           | in a group activity      | turns)                        |
|           |                          |                               |
|           |                          | \- user ratings               |
+-----------+--------------------------+-------------------------------+
| E         | Measure of group and     | \- percent efficiency = (task |
| fficiency | system effectiveness and | time - repair activities      |
|           | productivity             | time) / execution time        |
|           |                          |                               |
|           |                          | \- user ratings               |
|           |                          |                               |
|           |                          | \- tool usage                 |
|           |                          |                               |
|           |                          | \- breakdown                  |
+-----------+--------------------------+-------------------------------+
| Consensus | Measure of general       | \- user ratings               |
|           | agreement or group unity |                               |
|           | in outcome               | \- grounding                  |
+-----------+--------------------------+-------------------------------+

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 4.        |                          |                               |
| Example   |                          |                               |
| C         |                          |                               |
| apability |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+===========+==========================+===============================+
| **Cap     | **Definition**           | **Metric/Measure Components** |
| ability** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Awareness | Degree of "having \...   | \- user ratings               |
|           | realization, perception, |                               |
|           | or knowledge" (Webster)  | \- conversational constructs  |
|           | of surroundings and      |                               |
|           | events.                  |                               |
+-----------+--------------------------+-------------------------------+
| Coll      | Measures assess support  | \- expert judgments           |
| aboration | for coordinating         |                               |
| m         | collaboration; e.g.,     |                               |
| anagement | floor control, agenda    |                               |
|           | support, document access |                               |
|           | control, etc.            |                               |
+-----------+--------------------------+-------------------------------+
| Commu     | Measure of the exchange  | \- countables (number of      |
| ni-cation | of information in any of | turns per participant)        |
| (human to | the following forms:     |                               |
| human)    | verbal (spoken or        | \- turn overlap (simple       |
|           | written), visual,        | overlap and interruptions)    |
|           | physical                 |                               |
|           |                          | \- expert judgments           |
|           |                          |                               |
|           |                          | \- user ratings (goodness of  |
|           |                          | communication, getting floor  |
|           |                          | control, getting other        |
|           |                          | participants' attention,      |
|           |                          | ability to interrupt)         |
+-----------+--------------------------+-------------------------------+
| Grounding | Measure of how well      | \- user ratings (e.g.,        |
|           | common understanding is  | reaching common understanding |
|           | established.             | with other participants)      |
|           |                          |                               |
|           |                          | \- countables (number of      |
|           |                          | turns, length of turns)       |
|           |                          |                               |
|           |                          | \- turn overlap               |
|           |                          |                               |
|           |                          | \- conversational constructs  |
+-----------+--------------------------+-------------------------------+
| Coll      | Measure that assesses    | \- expert judgments           |
| aborative | support for              |                               |
| object    | collaborative objects;   | \- tool usage (optimal set of |
| support   | applied to shared        | tools used?)                  |
|           | workspace, object        |                               |
|           | manipulation and         |                               |
|           | management, etc.         |                               |
+-----------+--------------------------+-------------------------------+
| Task      | Measures the ability to  | \- task focus = (overall      |
| focus     | concentrate on the task  | time - transition time -      |
|           | at hand.                 | other time) / overall time    |
+-----------+--------------------------+-------------------------------+
| T         | Assesses support for     | > \- expert judgments         |
| ransition | transitions; used to     | >                             |
| measures  | evaluate collaboration   | > \- user ratings (e.g., flow |
|           | startup, summarization,  | > of transitions between      |
|           | playback, archiving,     | > tasks)                      |
|           | object exporting and     | >                             |
|           | importing, distribution  | > \- conversational           |
|           | of objects, translation  | > constructs                  |
|           | between modalities,      |                               |
|           | meeting notification,    |                               |
|           | etc.                     |                               |
+-----------+--------------------------+-------------------------------+

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 5.        |                          |                               |
| Example   |                          |                               |
| Service   |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+-----------+--------------------------+-------------------------------+
| **        | **Definition**           | **Metric/Measure Components** |
| Service** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Breakdown | Measures how often the   | \- conversational constructs  |
|           | user has to rationalize  |                               |
|           | a problem experienced.   | \- repair activities          |
|           | Breakdowns can occur in  |                               |
|           | communication, in        |                               |
|           | coordination, in the     |                               |
|           | system components, etc.  |                               |
+-----------+--------------------------+-------------------------------+
| Tool      | The degree to which the  | \- tool usage (which tools    |
| usage     | optimal tools are used   | were used and how often?)     |
|           | for a particular task.   |                               |
|           |                          | \- expert judgments (were the |
|           |                          | right tools used?)            |
+-----------+--------------------------+-------------------------------+

+-----------+--------------------------+-------------------------------+
| **Table   |                          |                               |
| 6.        |                          |                               |
| Example   |                          |                               |
| T         |                          |                               |
| echnology |                          |                               |
| Level     |                          |                               |
| M         |                          |                               |
| easures** |                          |                               |
+-----------+--------------------------+-------------------------------+
| **Tec     | **Definition**           | **Metric/Measure Components** |
| hnology** |                          |                               |
+-----------+--------------------------+-------------------------------+
| Usability | Evaluates the ease,      | \- expert judgments (standard |
|           | accessibility, and       | user interface evaluation     |
|           | intuitiveness of the     | questions)                    |
|           | specific graphical user  |                               |
|           | interfaces of the system | \- user ratings (standard     |
|           | tools and components.    | user interface evaluation     |
|           | Since usability          | questions)                    |
|           | evaluation is done on    |                               |
|           | specific user            | \- tool usage (which tools    |
|           | interfaces, the          | were used and how often? Was  |
|           | usability measures are   | the set of optimal tools      |
|           | realized at the          | used?)                        |
|           | technology level         |                               |
|           | although the component   | \- repair activities          |
|           | measures are also based, |                               |
|           | in part, on measures     | \- breakdown (defined at the  |
|           | taken at different       | service level)                |
|           | levels of the framework. |                               |
|           |                          | \- awareness (defined at the  |
|           |                          | capability level)             |
+-----------+--------------------------+-------------------------------+
| Specific  |                          | \- tool usage (which tools    |
| t         |                          | were used and how often?)     |
| echnology |                          |                               |
|           |                          | \- expert judgments (were the |
|           |                          | right tools used?)            |
+-----------+--------------------------+-------------------------------+

## 1.4Number range Level2Number range Level3Number range Level4 Data Collection Methods: Logging

We often need to collect data about running systems. Since videotaping
every session can be difficult, automatic logging can be important in
developing easy, repeatable, evaluation scenarios. A multi-modal logging
capability can support these efforts.

The MITRE Multi-Modal Logger (MML) is an example of this class of
logging tool. MITRE's MML can be used to log multi-modal data while
collaborative systems are running. This data can be fine-grained
(individual X or window events, for instance) or coarse-grained (a
record of which windows the user interacts with); automatically gathered
(via instrumentation) or manually created (via post-hoc annotation).
This data can be gathered at the level of the physical or virtual device
(for example, window events or the output of an audio device); at the
level of the interface (for example, a record of menu selections made
and the content of text entry fields); or at the level of the
application (a record of actions taken, such as a retrieval of
information or a command issued). This information can also be in a
variety of modalities, such as text, images, and audio.

Since this information will typically be gathered for multiple users and
multiple interactions with the system in question, the notion of a
"trial" or "session" is supported. In addition, each trial might require
information to be gathered from multiple components simultaneously (for
example, when a speech recognizer is used in conjunction with an
independent multi-modal system). Therefore, the MML also supports
sharing each trial among multiple components, potentially running on
different hosts.

MITRE\'s MML provides an application programming interface (API) for
instrumenting existing applications. It also provides a set of tools for
reviewing and annotating data collected via instrumentation.

MITRE's MML offers a solution to the question of what granularity and
levels of data may be collected. The instrumenter inserts whatever
logger calls are desired into the source code and is thus in complete
control of where, in the code, the log entries are generated, how many
are generated, and what data types are assigned to them. In general, one
can instrument any application for which the source code is available.

MITRE has also developed a log review and annotation tool to distribute
with the logger. It allows users to view the data that has been logged
for a given session. One may view all the data or select a subset by
application, data type and/or timestamp. The data is displayed along a
scrollable timeline, with the data sorted into streams by application
and data type. This tool may also be used to add post-hoc annotations to
the data. There is also a replay facility that allows the reviewer to
replay the logged interaction in approximately real time.

The MITRE MML tool suite and documentation are available for downloading
at http://www.mitre.org/technology/logger.