Step 1 Proposal

**Intelligent Systems: NRA2-38169**

***Title:* Development and Application of a Compositional Parametric
Approach to Human Performance Prediction**

*Principal Investigator Name:* Alonso Vera

*Department:* Human Factors Division

*Institution:* NASA Ames Research Center

*Address:* Mail Stop 262-4

*City:* Moffett Field *State:* CA *Zip:* 94024

*Country:* USA *E-mail:* Alonso.Vera@nasa.gov

*Telephone:* 650-604-6294 *Fax:* 650-604-0801

*Co-Investigators:*

*Name Institution Telephone Electronic Mail*

Richard Lewis University of Michigan 734-763-1466 rickl@umich.edu

Andrew Howes Cardiff University, UK 4429-2087-6707 ahowes@cardiff.ac.uk

*Budget (in \$K):*

*NASA: 1st Yr.:* 130K *2nd Yr.:* 120K *3rd Yr.:* 110K

UMich: *1st Yr.:* 130K *2nd Yr.:* 120K *3rd Yr.:* 110K

Cardiff: no cost

***Total:*** 720K

*Requested Start Date:* 10/01/03 *Requested Duration:* 3 years

*Technical Area Proposing Under:* TA-3 Human-Centered Computing

Select only one

(TA choices: 1) TA-1 Automated Reasoning, 2) TA-2 Intelligent Data
Understanding, 3) TA-3 Human-Centered Computing, 4) TA-4 Revolutionary
Computing, or 5) TA-5 Evolvable Systems.

Authorizing Official:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

(Name) (Institution)

# Abstract

The aim of the proposed research is to further develop a modeling tool
that supports the representation of human capabilities and limitations
in skilled computer-supported task performance, and to understand how
that tool can be used in the process of redesigning interfaces. The
tool, called CPM-X, builds on previous methods for predicting skilled
performance and embodies a novel technique for reasoning about
constraints on cognition. The specific objective is to develop CPM-X
through its application to the design of efficient and cognitively
compatible interfaces for NASA space systems technologies.

# Background and primary objectives of proposed effort

As NASA moves toward increasing automation across missions, it must
develop appropriate interfaces for the engineers and scientists that
interact with the affected systems. Much of this interaction is highly
skilled, with pressure to complete tasks rapidly and without error.
Significant components of ground control of remote vehicles, launch and
flight control, shuttle, and space-station operations fit this
description. The time pressures in flight control speak for themselves,
but they are ubiquitous---for example, they even arise in the off-line
task of specifying the next 24-hour mission for the Mars Exploration
Rover.

The nature of the interfaces for these complex systems affects both task
efficiency and error rates, but the consequences of interface changes or
redesigns are at best hard to predict and at worst counter-intuitive.
For example, there have been cases where the introduction of automation,
a putatively straightforward way to decrease task completion time, has
instead had the opposite effect of increasing task time (Ballas,
Heitmeyer & Perez, 1992). Similarly, increasing the efficiency of
individual interface operations does not necessarily increase overall
task efficiency, if those operations are not on the critical path that
determines overall time. The a priori prediction of performance time and
accuracy is difficult because human performance depends on many complex,
interacting constraints that derive both from the task environment and
from the nature of the human cognitive, perceptual, and motor processing
architecture. In fact, skilled performance of a routine task usually
involves the execution of a number of parallel but interdependent
streams of activity. For example, one hand may move to a mouse, while
the other finishes typing a word; the eyes track a target while a
command label is retrieved from long-term memory. The details of how
these cognitive and perceptual/motor processes are scheduled and the
nature of their dependencies have significant consequences for the
overall time requirement and probable error rate.

The rich set of constraints on skilled human behavior means that a
formal cognitive model of some form must be constructed if accurate and
useful performance predictions are to be made. There have been major
advances in cognitive modeling over the last ten years but current tools
are still difficult to use and force a set of unfortunate tradeoffs on
the modeler. Presently one must choose between modeling approaches that
are intended for interface evaluation but which cover a limited range of
psychological phenomena and methods that are psychologically rich but
require significant detail in the specification.

We contend that, apart from making modeling expensive, the expertise and
time required to model in existing platforms has had another unfortunate
side-effect: there are remarkably few examples of the application of
formal modeling methods to real interface design problems. One
consequence is that little is known about exactly how such modeling
tools might be integrated into the overall interface design process, and
exactly how much the design process and resulting interfaces can benefit
from such integration.

There are two principle objectives of the current proposal effort: (1)
Develop a cognitive modeling tool that supports parametric prediction of
human behavior in tasks that are a mix of routine and novel
interactions, and (2) Explore the role such a tool can have in the
process of redesigning complex computer system interfaces. We are able
to do this combined effort because members of our research team are
currently involved in both the development of new methods of cognitive
modeling and in the design of fielded space system interfaces.

Our specific technical approach draws on important recent developments
in cognitive modeling but departs from them in significant ways. In what
follows, we briefly describe (a) these current methods and their
limitations; (b) a prototype modeling tool called CPM-X we have
developed that points the way to a new approach to parametric modeling
for design; and (c) the specific research objectives for our proposed
project, including the application of our new tools to interface
redesign problems in three specific NASA space systems.

# Current practice in cognitive modeling

The state-of-the-art in cognitive modeling has advanced considerably
over the past ten years. There are two streams of work making contact
with the cognitive-perceptual-motor level: (a) computational cognitive
architectures, and (b) CPM-GOMS. Both approaches have their advantages.
With CPM-GOMS pert-charts it is possible to make detailed predictions of
time-course, that straightforwardly yield to critical path analysis (to
expose aspects of the interface that make a difference to performance),
without recourse to detailed specification of the operator
state-changes. In contrast, using a cognitive architecture, such as
ACT-R (Anderson & Lebiere, 1998) or EPIC (Kieras & Meyer, 1997), it is
possible to make detailed time course predictions for a range of
different interfaces without changing the underlying cognitive model. It
is also possible to test the consequences of different strategies, and
to reuse explicitly embodied architectural theory. To some extent the
consequences of learning can be modeled, and in addition, parametric
variation can be used to expose the implications of, for example,
different working memory capacities.

However, both architectural and CPM-GOMS approaches also have
disadvantages. CPM-GOMS modeling suffers from the fact that composition
of a model is an error-prone and labor intensive process. It is not
possible to "execute" a model on different interfaces. Recalculation of
the entire model may be required if one small change is made. In
contrast, cognitive architectures suffer from the fact that so much
detail is required; the architecture is hard to modify (e.g., to explore
the implications of different theories); it is difficult to
automatically extract behaviors over a space of parametric variation;
there is no explicit support for hierarchies as specification; and, it
is hard to visualize constraints on behavior (e.g., constraints on
critical path).

# The emergence of a formal psychological theory of behavior composition

We have recently implemented CPM-GOMS in a multi-resource
scheduling/simulation platform (Apex; Freed, 1998). This implementation
demonstrates that reusable "templates" of human behavior, a simple
procedure description language capable of representing hierarchical goal
decomposition, and automatic expansion and scheduling of cognitive,
perceptual and motor operations, can model highly skilled performance on
routine tasks with much less effort than previous modeling techniques
(John, Vera, Matessa, Freed, & Remington, 2002).

An important insight to be gleaned from this work is that it is possible
to formalize the *psychological principles of behavior composition* with
sufficient detail to automate the composition of new streams of behavior
from existing behavioral building blocks. Such a formal theory specifies
exactly how the subprocesses from two separate behaviors overlap or
"interleave" when they are combined. The composition processes and the
details of the resulting behavior are a function of *architectural
constraints*, *task/interface constraints*, and *strategies* that the
human chooses to employ in the task.

A general and formal theory of composition is important for three
critical reasons. First, it means that we can begin to apply the
fundamental software engineering principles of *abstraction,
composition,* and *reuse* to the problem of rapidly building cognitive
models. Second, it means that we can begin to capitalize on a large body
of empirical and theoretical work in cognitive psychology on task
switching and dual-tasking. Modeling tools for automated composition can
provide a vehicle for bringing the results of simple and abstract
laboratory tasks into contact with the complexities of real-world
interface design. Third, the modeling it enables provides quantitative
support for usability claims in the context of the development of
software interfaces.

One of the central purposes of the proposed research is to build on
these recent advances and create an approach to modeling, optimized for
interface design, that exhibits some of the strengths of architectures
with the strengths of CPM-GOMS pert chart modeling. Although the current
Apex-based automation tool represents a significant step forward,
neither it nor existing cognitive architectures such as ACT-R or EPIC
make it easy to independently manipulate the architecture,
task/interface, and strategic constraints that give rise to specific
behaviors. This is largely because the three kinds of constraints are
not explicitly represented in a consistent or accessible way (e.g., in
ACT-R or EPIC, the basic architectural assumptions are encoded in the
underlying Lisp code and are not typically modified at all). The
approach we sketch below is based in part on a principle of explicitly
and consistently representing all three levels of constraints as a set
of declarative assertions.

# Toward fast parametric compositional modeling: The CPM-X prototype tool

We have constructed a prototype software tool, called CPM-X, for
predicting the time course of interactive cognitive skill. CPM-X is a
formal analysis method that supports the representation of human roles,
capabilities, and limitations in skilled computer-supported task
performance taking into account basic human perceptual, motor, and
cognitive abilities. The tool calculates a prediction of an operator
schedule, given a declarative description of the task and psychological
constraints. In this respect, the tool can be contrasted to
simulation-based approaches to modeling cognition such as ACT-R and
EPIC, as well as to real-time schedulers such as Apex. Predictions are
derived by reasoning about constraints on cognitive processing rather
than by simulating transitions through a state-space.

The input to CPM-X consists of a declarative description of the task and
psychological constraints on task performance. The task constraints are
described in terms of a hierarchical analysis of the interaction between
a person and the task environment. The psychological constraints are
described in terms of a distributed set of processors each with its own
processing capabilities. In this respect, CPM-X is influenced by the
Model Human Processor (Card, Moran, Newell, 1983). Each processor is
defined in terms of a set of properties. For example, a processor might
typically execute an operator with a cycle of 50ms.

Following John and Gray (1995) operators are defined in terms of their
processor requirements and dependencies. The state changes caused by
operators are not represented (in contrast to simulation architectures).
Dependencies specify the preconditions for operator scheduling. For
example, a dependency may specify that a vision operator for perceiving
words must be preceded by a cognitive operator that attends to the
appropriate modality. Similarly, given the task to type 'ab', the
operator that initiates 'type b' is dependent on the completion of the
operator that initiates 'type a'.

CPM-X provides a means of expressing a theory of cognition as an
interacting set of simple information processors. In the models that we
have built we have used a set of processors that were derived from
CPM-GOMS. These include a cognitive processor, one or more motor
processors, and one or more perceptual processors. In addition to a
description of the psychological processors and their operators, CPM-X
also requires the description of strategies by which the processors work
together to produce desired behaviors. For a particular description of
the task and psychological constraints there are usually many possible
ways in which the cognitive, perceptual, and motor operators can be
scheduled. CPM-X uses an algorithmic method to calculate an operator
schedule that is consistent with the specified constraints and which,
given the rules, takes as little overall time as possible. The CPM-X
prototype is implemented in Prolog and has been applied to a handful of
Human-Computer Interaction problems. Initial findings indicate that the
technique makes millisecond accurate predictions of interactive skilled
behavior.

# Specific project objectives

The aim of our research is to build a tool that supports quantitative
prediction of human-computer interaction by: (1) enabling formal
reasoning about interaction without trying to simulate cognitive
processes; (2) providing a consistent and accessible declarative
language for expressing constraints on task, strategy, and architecture;
and (3) requiring the specification of only that information required to
evaluate an interface design.

The specific objectives of the proposed project are:

> A. To extend the formal theory of composition articulated in our
> earlier work.
>
> B. To extended the range of psychological constraints to which CPM-X
> is sensitive (including constraints on memory and constraints on the
> outcomes of learning).
>
> C. To create a library of NASA-relevant behavioral building blocks.
>
> D. To extend the interface to CPM-X by improving the usability of the
> constraint description language and of the summative and formative
> (Pert-like) output.
>
> E. To provide support for parametric modeling for populations of users
> over the lifecycle of the system. Parameters include quantitative
> variables such as degree of practice with subcomponents of the task;
> cognitive, perceptual or motor cycle times; memory decay parameters;
> as well as discrete parameters such as strategy choice.
>
> F. To test the theory and tool in three design contexts relevant to
> NASA. As part of the software design cycle for each of these three
> domain applications, CPM-X models will be used to determine what
> interface designs best support efficient, accurate task execution. The
> three design contexts are:

*Mars Exploration Rover (MER)/Mars Science Laboratory (MSL) Uplink
process*. The MER '03 mission uplink process begins with the setting of
data collection goals for the Rover on a given Martian day (sol) and
ends with the radiation of an uplink sequence from earth to the Rover.
Three components of this uplink process are performed every sol in a
fixed (and relatively short) period of time by skilled staff (CE,
MAPGEN, & RSVP). MSL '09 will share these activities with MER but many
parts of the process are intended to be automated. Nevertheless,
significant components will remain for input by human operators,
particularly the translation of scientific intent into the Goal Nets
that will represent them. We are already working with the CE/MAPGEN Team
(K. Rajan) and will be in position during surface operations on Mars in
early '04 to collect data on the use of these two systems as well as
RSVP. A. Vera led the redesign of CE based on an HCI analysis of task
and context demands. J. Trimble and A. Vera, through the Applied
Human-Computer Interaction Team they co-lead, will be funded by JPL (A.
Sax & M. Sander) in collaboration with T. Starbird at JPL to *develop*
the interfaces for the Goal Net system. CPM-X modeling will provide the
*research* foundation for this work (not funded by JPL).

*Shuttle Cockpit Upgrade.* In collaboration with R. McCann in the Human
Factors Division at Ames, we will use CPM-X to model and evaluate
designs for the Shuttle cockpit upgrade. A Shuttle cockpit simulator
(Isis) is now complete and data will be collected during summer '03
using experienced pilots. This data will be modeled and used to generate
feedback for the redesign.

*Deep Space Network (DSN).* In collaboration with D. Atkinson at JPL we
will contribute to the redesign of Goldstone Control Station's systems
and procedures based on CPM-X models of current use.

# References

Anderson, J. R. & Lebiere, C. (1998). *The Atomic Components of
Thought.* Erlbaum: Mahwah, NJ.

Ballas, J. A., Heitmeyer, C. L., & Perez, M. A. (1992). *Direct
manipulation and intermittent automation in advanced cockpits*.
Technical Report NRL/FR/5534-929375. Naval Research Laboratory,
Washington, D. C.

Card, S.K., Moran, T. P., & Newell, A. (1983). *The psychology of human
computer interaction.* Erlbaum: Hillsdale, NJ.

Freed, M. (1998). *Simulating Human Performance in Complex, Dynamic
Environments*. Doctoral Dissertation, Northwestern University.

John, B. E. & Gray, W. D. (1995). *GOMS Analyses for Parallel
Activities.* Tutorial materials, CHI'95 (Denver CO, May 7-11, 1995) ACM,
New York.

John, B. E., Vera, A., Matessa, M., Freed, M., & Remington, R. (2002).
Automating CPM-GOMS. *Proceedings of CHI*'*02* (Minneapolis, April
20-25, 2002) ACM, NY.

Kieras, D. & Meyer, D. E. (1997). An overview of the EPIC architecture
for cognition and performance with application to human-computer
interaction*. Human-Computer Interaction,* 12, 391-438.