USING PSYCHOLOGICALLY PLAUSIBLE OPERATOR COGNITIVE MODELS TO ENHANCE
OPERATOR PERFORMANCE

Chris Forsythe, Michael Bernard, Patrick Xavier,

Robert Abbott, Ann Speed, & Nathan Brannon

Sandia National Laboratories

Albuquerque, New Mexico

> Research by Sandia National Laboratories (SNL) is currently being
> conducted that seeks to embody human-like cognitive capacities in
> machines by transforming the human-machine interaction so that it more
> closely resembles a human-to-human interaction. This document reports
> on the initial phase of research and development by SNL in creating a
> capability whereby a machine-based cognitive model provides a
> real-time awareness of the cognitive state of an operator. In the
> capability referred to as "Discrepancy Detection," the machine uses an
> operator's cognitive model to monitor its own state and when there is
> evidence of a discrepancy between the actual state of the machine and
> the operator's perceptions concerning the state of the machine, a
> discrepancy may be signaled. The current project offers successful
> evidence that a machine may accurately infer an operator's
> interpretation of situations based on an individualized cognitive
> model of the operator.

**SCOPE OF RESARCH**

Our research is based on an individualized cognitive model of an
operator's knowledge for a specific task domain (Forsythe & Xavier,
2002). We utilized the cognitive framework previously developed by SNL
to computationally model an operator's cognitive processes. This
framework, shown in Figure 1, was partially inspired by naturalistic
decision making (Klein, 1997; Klein, Calderwood, & MacGregor, 1989) and
oscillating systems theory (Klimesch, 1996). The model emphasizes the
processes whereby an individual recognizes situations on the basis of
patterns of cues within the environment. In its simplest form, the model
consists of (1) a collection of situations, (2) cues associated with
those situations, and (3) mathematical expressions for the recognition
of situations based on patterns of cues.

In a functional cognitive model, streaming input from the task
environment feeds perceptual algorithms. These algorithms operate in
parallel and, depending on the corresponding cue in the associative
network, provide either a continuous or discrete level of activation.
For a given perceptual algorithm, if the level of activation is
sufficient, a response will be elicited from the corresponding node in
an associative network.

The associative network is modeled as an oscillating system wherein each
node is a separate oscillator (cf. Klimesch, 1996). Perceptual
activation causes corresponding nodes to begin oscillating with
activation spreading to other nodes for which there exists an
associative relationship. Spreading activation between nodes in the
associative network may be sufficient to cause a node, receiving
activation, to begin oscillating or may prime the node such that the
node exhibits a lowered threshold for activation by perceptual
processes.

Situation recognition occurs when there is activation of a pattern of
nodes in the associative network that corresponds to a given situation.
Computationally, recognition is based on an evidence accumulation[^1]
process in which cues differentially contribute to or subtract from the
evidence for different situations, and the evidence for each situation
is periodically (i.e., every 250 msec) updated providing an ongoing
level of evidence for each situation. At any given point in time,
multiple situations may have sufficient evidence for their recognition
(i.e., multi-tasking).

Implemented in systems for augmented cognition, a customized model is
developed to reflect the knowledge of a specific operator as the person
interacts with that system. As shown in Figure 2, operating in
real-time, this model is used to interpret the ongoing state of the
system. To the extent that the system has accurate and timely data
concerning its own state, and the operator on whom the model was based
has a fundamental understanding of the system and associated tasks, this
serves as an idealized cognitive processor in that it has an essentially
unlimited capacity to process multiple cues in parallel.

Discrepancy detection is accomplished using real-time data from an
operator. The system monitors the operator's actions, or lack of action.
For each situation, typical actions have been identified. Discrepancy
detection is triggered when an operator either commits an action that is
inappropriate given relevant situations or an excessive period of time
transpires in which an operator does not execute an action that is
expected given relevant situations. Further development of the
capability for discrepancy detection will utilize additional sources of
input such as eye-tracking and physiological indicators associated with
situation recognition processes.

# METHOD

*Task Materials.* The Distributed Dynamic Decision-making (DDD) Airborne
Warning and Control System (AWACS) simulation trainer was chosen as the
task environment for prototype development. This environment offered a
reasonably high-level of complexity in which it was necessary for
operators to multi-task as they simultaneously controlled multiple
assets and responded to multiple threats. Participants practiced until
they achieved a high level of expertise in playing the game.

The criterion for expert performance consisted of having the knowledge
and skills to consistently play the game without the loss of friendly
aircraft due to lack of fuel or enemy missiles, without enemy incursion
within the friendly zone, as well as the destruction of all enemy jets
(see Figure 3).

*The Cognitive Model.* Research has shown that with dynamic environments
such as the DDD-AWACS task, individuals tend to have distinct cognitive
models that are tied to their skill level, as well as their past
experiences with the task environment (e.g., Goodrich & Boer, 1999).
Therefore, a key question concerned whether the differences between
individual cognitive models are sufficient to necessitate development of
individualized models. A comparison of cognitive models of two operators
with equivalent expertise demonstrated that individualized cognitive
models are required for systems to perform reliably in that only 26% of
the total cues and situations associated with the cues identified by the
two participants were used by both participants. Consequently,
individualized cognitive models were used in our design.

In order to populate each cognitive model with expert knowledge, we
utilized an individualized knowledge elicitation method developed at SNL
that was designed specifically for this cognitive framework. This method
was explicitly designed to acquire both explicit and implicit knowledge.

*Discrepancy Detection.* A method of Discrepancy Detection that is based
on the actions expected under various conditions (i.e., situations) was
implemented. The general idea is that in the context of a particular
situation instance, it is expected that an operator will perform certain
actions--- "what needs to be done"---with respect to the entities in the
situation instance. The pattern of action expectations is derived from
the knowledge elicitation process. The Discrepancy Detector module takes
as real-time input, the current actions and situation instances inferred
by the cognitive model.

## RESULTS

*Assessing Expert Performance Accuracy.* Formal testing evaluated the
accuracy of the cognitive model for interpreting situations within the
task environment, relative to the interpretations of the actual
participant on whom the cognitive model was based. For the evaluation, a
session was used in which one of the participants obtained a perfect
performance score. The session was recorded so it could be played back
readily. To create a reference against which the situation
interpretation of the model could be assessed, the participant
repeatedly reviewed the recording indicating points in time when
situations became active and subsequently, when situations were
completed.

The results revealed a system capability to infer an expert operator's
ongoing cognitive interpretation of the state of the system with an
accuracy of 87%. Moreover, only considering the occurrence of
situations, as opposed to recognizing when situations were not preset,
the cognitive model was 91% accurate. There was a 10% incidence of false
positive and 3% incidence of false negatives.

*Assessing Novice Performance Accuracy.* As a further evaluation, a
second reference was created in which the participant deliberately
performed poorly. Enemy jets were allowed to destroy two friendly jets,
and two jets were purposely aloud to run out of fuel, and thus were
destroyed. In addition, one enemy jet was allowed to pass through the
friendly zone, thereby lowering the defense score. The cognitive model
performed equally well in this condition, suggesting that the prototype
system is relatively robust.

**DISCUSSION**

This project demonstrated that a machine can accurately infer in
real-time the cognitive processes of an operator using an individualized
cognitive model of the operator. This provides the foundation for a
variety of concepts for augmentation that rely upon a capability for a
machine to accurately model the cognitive processes of an operator. The
current project illustrates one manifestation of this capability.
Through discrepancy detection, the machine not only detects that the
operator has committed an error, but the machine recognizes the nature
of the error within the context of the operator's overall knowledge and
understanding of the system and associated tasks. Consequently, the
machine may interact with the operator in a manner that is substantially
more meaningful than the familiar error messages presented by current
software applications and operating systems. Furthermore, to the extent
that there exists deficiencies in the operator's knowledge of the system
and associated tasks, there is a meaningful basis for training
intervention to correct deficiencies in the operator's knowledge.

The results obtained through the current project have broad application.
An immediate next step concerns the integration of the prototype
demonstrated here with other sources of input to provide the machine
with a more comprehensive ongoing representation of an operator.
Initially, plans are to explore eye-tracking in addition to operator
actions as one source of input. Whereas actions provide explicit data
concerning an operator's interpretation of situations, eye-tracking
offers a more subtle indicator. To the extent that recognition of a
situation requires that an operator process visual cues within the
environment that are associated with the situation, data indicating
visual cues that most likely have or have not been processed should
enable somewhat better predictions.

Another comparable approach to improving machine awareness of an
operator is to provide input from electrophysiological and brain imaging
sources. Efforts have begun to identify measurable physiological
correlates to the cognitive processes modeled in the Sandia cognitive
framework. Again, these data offer potential for systems to more
accurately model the ongoing cognitive processes of an operator.
However, the benefit should go two ways. Where physiological measures
are employed to derive a real-time indication of workload, these
indications may be enhanced by input supplied by a cognitive model. For
example, the cognitive model may indicate that there are six different
situations that characterize events at a given point in time. Such data
could then be used as a modifier for physiological measures (i.e., if
the cognitive model indicates that workload should be high, this serves
to confirm physiological measures suggesting that workload is high).

**Acknowledgements**

This work was performed at Sandia National Laboratories. Sandia is a
multi-program laboratory operated by Sandia Corporation, a
Lockheed-Martin Company, for the United States Department of Energy
under Contract DE-AC04-94AL85000.

**REFERENCES**

Coles, M. G. H., Snid, H. G. O. M, Scheffers, M. K. & Otten, L. J.
(1995). Mental chronometry and the study of human information
processing. *Electrophysiology of mind: Event-related brain potentials
and cognition*, Oxford: Oxford Science Publications, 86-131.

Goodrich, M. A., & Boer, E. R. (1999). Multiple mental models,
automation strategies, and intelligent vehicle systems. *International
Conference on Intelligent Transportation* Systems Cat*. No. 99^th^
8383*, 859-864.

Forsythe, C., & Xavier, P. G. (2002). Human emulation: progress towards
realistic synthetic human agents. *Proceedings of the 11^th^ Conference
on Computer Generated Forces and Behavioral Representation*, Orlando,
Florida.

Klein, G. (1997). An overview of naturalistic decision making
applications. In C. E. Zsambok & G. Klein *Naturalistic Decision
Making*, Mowah, NJ: Lawrence Earlbaum, 49-59.

Klein, G., Calderwood, R., & MacGregor, D.

> (1989). Critical decision method for eliciting knowledge. *IEEE
> Transactions on Systems, Man, and Cybernetics, 19*, 462-472.

Klimesch, W. (1996). Memory processes, brain oscillations and EEG
synchronization. *International Journal of Psychophysiology*, *24*,
61-100.

Kok, A. (1990). Internal and external control: A two-factor model of
amplitude change of event-related potentials. *Acta Psychologica*, *74*,
203-236.

Kounios, J., Montgomery, E. C., & Smith R.W. (1994). Semantic memory and
the granularity of semantic relations: Evidence from speed-accuracy
decomposition. *Memory and Cognition*, *22*(6), 729-741.

Wilding, E. L. (2000). In what way does the parietal ERP

> old/new effect index recollection. *International Journal of
> Psychophysiology*, *35*, 81-87.

![](media/image2.wmf)

*Figure 1.* Framework for Computationally Modeling Individualized
Knowledge and the Cognitive Processes Enabling Situation Recognition

![](media/image4.png){width="1.3104166666666666in"
height="1.1215277777777777in"}

*Figure 2.* Example of a System a Using Machine-Based Cognitive Model of
an Operator for Augmented Cognition. In this Case, the Objective is for
the System to Detect Discrepancies between the Actual State of the
System and the Operator's Perceived State of the System (i.e.,
Discrepancy Detection)

![](media/image5.png){width="6.680555555555555in"
height="5.153472222222222in"}

*Figure 3.* The Screen Layout of the DDD-AWACS Task Environment.

[^1]: The evidence accumulation approach to situation recognition is
    based on a synthesis of findings from the EEG literature that
    included Coles, et.al. (1995); Kok (1990); Kounios, et.al. (1994); &
    Wilding (2000).