SAfety VEhicles using adaptive

Interface Technology

(Task 5)

Final Report: Phase 2

Algorithms to Assess Cognitive Distraction

Prepared by

John Lee

Michelle Reyes

Yulan Liang

Yi-Ching Lee

The University of Iowa

Phone: (319) 384-0810

Email: <john-d-lee@uiowa.edu>

July 2007

Table of Contents

[5.1 Executive summary 6](#executive-summary)

[5.2 Program Overview 10](#program-overview)

[5.3 Introduction and objectives 10](#introduction-and-objectives)

[5.4 Interaction of Cognitive and Visual Distraction
12](#interaction-of-cognitive-and-visual-distraction)

[5.4.1. Experiment 1: Change detection and safety-relevance
15](#experiment-1-change-detection-and-safety-relevance)

[5.4.1.1 Method 15](#method)

[5.4.1.2 Results 18](#results)

[5.4.1.3 Discussion 20](#discussion)

[5.4.2. Experiment 2: Image size and safety relevance
21](#experiment-2-image-size-and-safety-relevance)

[5.4.2.1 Method 21](#method-1)

[5.4.2.2 Results 22](#results-1)

[5.4.2.3 Discussion 24](#discussion-1)

[5.4.3. General discussion 25](#general-discussion)

[5.5 Support Vector Machines to Detect Cognitive Distraction
27](#support-vector-machines-to-detect-cognitive-distraction)

[5.5.1. Model Construction 30](#model-construction)

[5.5.1.1 Data source 30](#data-source)

[5.5.1.2 Model characteristics and training
34](#model-characteristics-and-training)

[5.5.1.3 Model performance measures 37](#model-performance-measures)

[5.5.2. Results 38](#results-2)

[5.5.2.1 Performance of the SVM models
38](#performance-of-the-svm-models)

[5.5.2.2 Comparison with the logistic method
38](#comparison-with-the-logistic-method)

[5.5.2.3 Effect of model characteristics
40](#effect-of-model-characteristics)

[5.5.3. Discussion 42](#discussion-2)

[5.5.4. Conclusion 45](#conclusion)

[5.6 Bayesian Networks to Detect Cognitive Distraction
45](#bayesian-networks-to-detect-cognitive-distraction)

[5.6.1. Model Construction 47](#model-construction-1)

[5.6.1.1 Training of BN models 47](#training-of-bn-models)

[5.6.1.2 Model performance measures 50](#model-performance-measures-1)

[5.6.1.3 Mutual information 50](#mutual-information)

[5.6.2. Results 51](#results-3)

[5.6.2.1 Comparison of model performance
51](#comparison-of-model-performance)

[5.6.2.2 Analysis of mutual information
52](#analysis-of-mutual-information)

[5.6.3. Discussion 54](#discussion-3)

[5.6.4. Conclusion 56](#conclusion-1)

[5.7 Combining Algorithms to Detect Distraction
56](#combining-algorithms-to-detect-distraction)

[5.7.1. Data mining techniques to assess cognitive state
59](#data-mining-techniques-to-assess-cognitive-state)

[5.7.2. Development of SVMs and BNs 60](#development-of-svms-and-bns)

[5.7.2.1 Model training 60](#model-training)

[5.7.2.2 Model evaluation 61](#model-evaluation)

[5.7.3. Model comparison 61](#model-comparison)

[5.7.4. Discussion 64](#discussion-4)

[5.8 References 65](#references)

Table of Figures

[Figure 5.1. The mean d' (± SE) as a function of blanking and auditory
task 7](#__RefHeading___Toc173654911)

[Figure 5.2. Testing accuracy and sensitivity for different parameters
of input data. 8](#__RefHeading___Toc173654912)

[Figure 5.3. Comparisons of model type and number of hidden nodes.
9](#__RefHeading___Toc173654913)

[Figure 5.4. The mean d' (± SE) as a function of blanking and auditory
task in Experiment 1. 18](#__RefHeading___Toc173654914)

[Figure 5.5. The mean d' (± SE) as a function of different types of
changes and blanking and auditory task in Experiment 1.
19](#__RefHeading___Toc173654915)

[Figure 5.6. The mean d' (± SE) as a function of blanking and auditory
task in Experiment 2. 22](#__RefHeading___Toc173654916)

[Figure 5.7. The mean d' (± SE) as a function of different types of
changes and blanking and auditory task in Experiment 2.
23](#__RefHeading___Toc173654917)

[Figure 5.8. A graphical representation of the support vector machine
algorithm. 28](#__RefHeading___Toc173654918)

[Figure 5.9. Illustration of the algorithm used to identify eye
movements. 32](#__RefHeading___Toc173654919)

[Figure 5.10. Results of the SVM and logistic models. The dashed,
dash-dot, and dash-double dot lines indicate chance performance, SVM
average, and logistic average, respectively.
38](#__RefHeading___Toc173654920)

[Figure 5.11. ROC curves for the SVM and logistic models for DRIVE,
STAGE, and STEER. 39](#__RefHeading___Toc173654921)

[Figure 5.12. SVM testing accuracy and sensitivity for the feature
combinations. The braces represent the post hoc comparisons between the
successive combinations using Tukey-Kramer method. \*\* indicates
p\<0.05. 41](#__RefHeading___Toc173654922)

[Figure 5.13. Testing accuracy and sensitivity for different summarizing
parameters of input data. 42](#__RefHeading___Toc173654923)

[Figure 5.14. The plots of fixation distribution over the background of
the driving scenario for IVIS (left) and baseline (right) conditions for
participant SF7. The size of each dot represents fixation duration.
43](#__RefHeading___Toc173654924)

[Figure 5.15. SVM decision variable along the timeline of an IVIS drive
and a baseline drive for participant SF7 (for the same data shown in
Figure 5.14). 44](#__RefHeading___Toc173654925)

[Figure 5.16. Examples of a SBN and a DBN, where H is a hypothesis node,
S is a hidden node, Es are observation nodes, and t represents time.
46](#__RefHeading___Toc173654926)

[Figure 5.17. Constrained DBN Structure, where solid arrows represent
intra-links and dotted arrows represent inter-links.
50](#__RefHeading___Toc173654927)

[Figure 5.18. The Comparisons of Model Type, Number of Hidden Nodes and
the Interaction. 52](#__RefHeading___Toc173654928)

[Figure 5.19. The Mutual Information of Seven Categories of Performance
Measures. 53](#__RefHeading___Toc173654929)

[Figure 5.20. Data fusion that transforms raw driving and eye movement
data into estimates of cognitive distraction.
59](#__RefHeading___Toc173654930)

[Figure 5.21. Constrained DBN Structure, where solid arrows represent
relationships within the first time step, dotted arrows represent
relationships across time steps, and H and E presents the predictive
target and performance measures, respectively.
61](#__RefHeading___Toc173654931)

[Figure 5.22. Comparisons of testing accuracy and sensitivity.
62](#__RefHeading___Toc173654932)

[Figure 5.23. Comparisons of hit and false alarm rates.
63](#__RefHeading___Toc173654933)

[Figure 5.24. Comparison of false alarm rate for DBNs, SBNs, and SVMs
for window sizes of 5 to 30 seconds. 63](#__RefHeading___Toc173654934)

Table of Tables

[Table 5.1. The characteristics of fixations, saccades, and smooth
pursuits. 33](#__RefHeading___Toc173654935)

[Table 5.2. The feature combinations used as model input.
34](#__RefHeading___Toc173654936)

[Table 5.3. Model characteristics and their values.
35](#__RefHeading___Toc173654937)

[Table 5.4. The Comparisons with Different Characteristics.
49](#__RefHeading___Toc173654938)

[Table 5.5. The Mutual-Information Distribution of Nine Participants
across the Most Predictive Variables (dark shading indicates average
normalized mutual information greater than 30%; dark grey indicates
between 30% and 10%; grey indicates between10% and 5%; white indicates
less than 5%). 54](#__RefHeading___Toc173654939)

[Table 5.6. Matrix of data fusion strategies and the availability of
domain knowledge. 57](#__RefHeading___Toc173654940)

# Executive summary

The objective of Task 5 (Cognitive Distraction) is to develop an
algorithm that uses driver state information to predict decrements in
driving performance due to cognitive distraction. Driving performance is
operationalized as the reaction time to driving events that require a
response by the driver. Specifically, our objectives were to develop an
experiment that created a measurable degree of distraction, to evaluate
dependent measures associated with this distraction, and to develop an
algorithm to predict distraction based on those measures.

In Phase 1, a review of literature related to workload estimation
suggested that eye movements and heart rate data might be useful
measures of cognitive distraction in that such data could be collected
unobtrusively, was technically feasible to include in a production
vehicle within the next 10 years, and was reasonably diagnostic. An
experiment was conducted to evaluate how IVIS-task (In-Vehicle
Information Systems) demands influence driving performance at the
control and tactical levels and how this influence correlates with
changes in eye gaze patterns. The control level refers to the
moment-to-moment operation of vehicles, and the tactical level refers to
the choice of maneuvers and immediate goals in getting to a destination.
The study also assessed how well these patterns predicted
distraction-related decrements in driver performance. The experimental
design consisted of twelve combinations of three within-subject
independent variables: 1) lead vehicle braking task (control or
tactical), 2) multiple resource theory (MRT) dimensions of the IVIS task
(verbal or spatial, perceptual or response selection), and 3) response
selection complexity (simple or complex, nested within the Response
condition). The main dependent measures include driving, eye movement,
and electrocardiogram data.

As expected, the tactical braking task had much shorter accelerator
release and brake reaction times than the control task. Both accelerator
release and brake reaction times degraded during IVIS interactions, but
only for tactical braking events. IVIS interactions significantly
degraded speed and steering maintenance, reflected by the effect on
measures associated with vehicle speed, accelerator position, and
steering error and entropy. Three eye movement measures reflected
changes in cognitive load: fixation duration decreased, the distance
between successive fixations or saccade distance increased, and the
proportion of short fixations increased as cognitive load increased.
These results indicated that listening to IVIS information was less
demanding than responding to questions about it. IVIS interactions
degrade a driver's ability to anticipate emerging conflict situations
more than they degrade driver response to a conflict situation (Reyes &
Lee, 2004). Hidden Markov Models were used to predict driver distraction
from eye data with limited success.

In Phase 2 the underlying mechanisms associated with cognitive
distraction were assessed to determine how cognitive distraction might
interact with visual distraction to undermine driving performance. Two
experiments were conducted using a change blindness paradigm in which
the screen of the driving simulator was periodically blanked for one
second to simulate a glance away from the roadway. Drivers also
performed a complex auditory/vocal task representative of an IVIS
destination selection task. Twelve people participated in each of the
experiments. Dependent measures included participants' sensitivity to
vehicle changes and confidence in detecting them.

In the first experiment, cognitive load uniformly diminished
participants' sensitivity and confidence, independent of
safety-relevance or lack of exogenous cues. Periodic blanking, which
simulated glances away from the roadway, undermined change detection to
a greater degree than cognitive load; however, cognitive load diminished
drivers' confidence in their ability to detect changes more. Figure 5.1
shows that cognitive load and short glances away from the road are
additive in their tendency to increase the likelihood of drivers missing
or not recognizing safety-critical events, measured by d' (the number of
standard deviations between the density functions for hits and false
alarms). This study demonstrates the need to consider the combined
consequences of cognitive load and brief glances away from the road in
assessing distraction.

![](media/image1.wmf){width="3.292361111111111in"
height="2.8986111111111112in"}

[]{#__RefHeading___Toc173654911 .anchor}Figure 5.1. The mean d' (± SE)
as a function of blanking and auditory task

To extend the algorithm development of Phase 1, the second experiment
examined the effect of cognitive load on driving performance for
interactions that varied from one to four minutes. Participants
completed four 15-minute drives while performing the IVIS task. There
were three IVIS conditions: interacting with the IVIS system, the
non-IVIS periods during drives where the IVIS task was present, and a
baseline drive with no IVIS interactions. Contrary to our hypothesis,
driver response to the lead vehicle braking events was surprisingly
uniform across IVIS conditions. IVIS interaction did undermine bicycle
detection, and this effect increased with the duration of the task. The
detrimental effect of IVIS interactions persisted even after the
interaction was completed. Eye movements were systematically influenced
by IVIS conditions, although gaze concentration, as measured by the
product of the standard deviation of vertical and horizontal fixation
locations, responded to IVIS conditions in a manner counter to previous
research, with gaze concentration diminishing with cognitive load. The
eye movement analysis suggests that two mechanisms might account for the
distraction-related performance decrements in this study: a competition
for processing resources and an interference between competing goals.

Based on the data from this experiment, both Support Vector Machines
(SVMs) and Bayesian Networks (BNs), two data mining techniques, were
selected to assess driver cognitive distraction using eye movement and
driving performance measures. In the SVM analysis, each subject's data
were used to train and test both SVM and logistic regression models for
that subject. Three different model characteristics were investigated:
how distraction was defined, which data were input to the model, and how
the input data were summarized. SVM models were able to detect driver
distraction with an average accuracy of 81.1% and outperformed more
traditional logistic regression models. The best-performing SVM model
(96.1% accuracy) resulted when distraction was defined using
experimental conditions (i.e., IVIS drive or baseline drive), the input
data were comprised of eye movement and driving measures, and these data
were summarized over a 40-second window with 95% overlap of windows.
Figure 5.2 shows the influence of window size and overlay on
predictions.

![](media/image2.wmf){width="2.9180555555555556in"
height="2.3569444444444443in"}![](media/image3.wmf){width="2.9097222222222223in"
height="2.3534722222222224in"}\
Testing accuracy (percent hits) Sensitivity (d')

[]{#__RefHeading___Toc173654912 .anchor}Figure 5.2. Testing accuracy and
sensitivity for different parameters of input data.

In the BN analysis, models were trained and tested to investigate the
influence of three model characteristics on distraction detection:
time-relationship of driver behavior, the inclusion of an intermediate
variable (hidden node) that groups model inputs, and summarizing data
with different time windows and length of training sequences. Figure 5.3
shows the performance of BNs and the relative benefit of Dynamic BNs
(DBN). The results demonstrated that BNs could identify driver
distraction for any given driver reliably with an average accuracy of
80.1%. DBNs that considered time-dependencies of driver behavior
produced more sensitive models than SBNs. Longer training sequences
improved DBN model performance. Blink frequency and fixation measures
were particularly indicative of distraction. These results indicate that
BNs, especially DBNs, are able to detect driver cognitive distraction by
extracting information from complex behavioral data.

![](media/image4.jpeg){width="5.488194444444445in"
height="3.027083333333333in"}

[]{#__RefHeading___Toc173654913 .anchor}Figure 5.3. Comparisons of model
type and number of hidden nodes.

In combination, these experiments indicate that eye movements represent
a promising approach to assessing cognitive distraction in real time.
Although not perfect, the SVM and BN models provide substantial
precision in detecting instances of cognitive distraction, with
accuracies of 75-95% depending on the algorithm and input data. The
experiments also suggest that cognitive and visual demands are additive.
This finding suggests that estimates for degraded driving performance
associated with cognitive demand can be added to those associated with
visual demand to estimate the combined total. At the same time, these
experiments show that cognitive distraction is not a unitary construct
and can influence driving tasks differently, as, for example, in the
differential effect of cognitive distraction on pedestrian detection and
vehicle control seen here. Another important finding is that cognitive
distraction can display inertia, affecting driver performance after the
task has been completed. This finding supports the need to model task
interruptability in predicting IVIS demand (Task 6).

# Program Overview

Driver distraction is a major contributing factor to automobile crashes.
The National Highway Traffic Safety Administration (NHTSA) has estimated
that approximately 25% of crashes are attributed to driver distraction
and inattention (Wang, Knipling, & Goodman, 1996). Recent estimates from
the 100-Car study suggest that distraction may contribute to more than
three quarters of all crashes (Dingus, Klauer, Neale, Petersen, Lee,
Sudweeks, Perez, Hankey, Ramsey, Gupta, Bucjer, Doersaph, Jermeland, &
Knipling, 2006). The issue of driver distraction may become more
critical in the coming years because increasingly elaborate electronic
devices (e.g., cell phones, navigation systems, wireless Internet and
email devices) are being brought into vehicles that may further
compromise safety. In response to this situation, the John A. Volpe
National Transportation Systems Center (VNTSC), in support of NHTSA\'s
Office of Vehicle Safety Research, awarded a contract to a diverse team
led by Delphi Electronics & Safety including Ford, the University of
Michigan Transportation Research Institute (UMTRI) and the University of
Iowa. The goal of this program was to develop, demonstrate, and evaluate
the potential safety benefits of adaptive interface technologies that
manage the information from in-vehicle systems based on real-time
monitoring of the roadway and the state of the driver. The contract,
known as SAfety VEhicle(s) using adaptive Interface Technology
(SAVE-IT), is designed to mitigate distraction with effective
countermeasures and enhance the effectiveness of safety warning systems.

The SAVE-IT program serves several important objectives. Perhaps the
most important objective is that of demonstrating a viable proof of
concept that is capable of reducing distraction-related crashes and
enhancing the effectiveness of safety warning systems. Program success
is dependent on integrated closed-loop principles that incorporate the
state of the driver. This closed-loop vehicle system is achieved by
measuring the driver's state, assessing the situational threat,
prioritizing information presentation, providing adaptive
countermeasures to minimize distraction, and optimizing collision
warning systems.

# Introduction and objectives

The objective of Task 5 (Cognitive Distraction) is to develop an
algorithm that uses driver state information to predict decrements in
driving performance due to cognitive distraction. Driving performance is
operationalized as the reaction time to driving events that require a
response by the driver. Our specific objectives were to develop an
experiment that created a measurable degree of distraction, to evaluate
dependent measures associated with this distraction, and to develop an
algorithm able to predict distraction based on those measures.

The need to predict cognitive distraction is being driven by the
migration of complex technology into cars and trucks. Many drivers are
transforming their vehicles into mobile offices, with devices that allow
them to use the Internet, send and receive faxes, receive news, and
converse on cell phones (Dewar & Olson, 2002). These systems promise
benefits of increased comfort, productivity, and mobility. However, they
may also distract drivers and undermine driving safety (Goodman,
Tijerina, Bents, & Wierwille, 1999; J. D. Lee & Strayer, 2004).

Driving makes intense demands on visual perception (Dewar & Olson,
2002). As a result, operating devices that require glances away from the
road result in structural interference, which can have obvious negative
effects on driving performance. Increasing the duration of glances away
from the road increases the probability of lane departure, such that
glances of two seconds lead to 3.6 times more lane departures than
glances of one second (Green, 1999). Cognitive interference has less
obvious consequences. Operating devices that do not require glances away
from the road, such as speech recognition systems, can nevertheless
impose a cognitive load that may interfere with driving performance.
This cognitive load has the potential to impair drivers' ability to
maintain vehicle control (Rakauskas, Gugerty, & Ward, 2004). Cognitive
load can also delay or interrupt cognitive processing of roadway-related
information, resulting in longer reaction times (Alm & Nilsson, 1994;
1995; J. D. Lee, Caven, Haake, & Brown, 2001), degraded speed and
headway control (Strayer & Drews, 2004), and less effective use of
environmental cues to anticipate when to brake (Jamson, Westerman,
Hockey, & Carsten, 2004).

The effect of cognitive and structural interference depends on the type
of task. Multiple resource theory suggests that two tasks that draw upon
the same mode (e.g., information received through the eye only, or
through the eye and the ear), code (i.e., analogue/spatial vs.
categorical/verbal processes) or stage of processing (e.g., perceptual,
cognitive, the selection and execution of response) will interfere with
each other more than two tasks that draw upon different resources
(Wickens, 1984; 2002). In driving, a concurrent spatial task interferes
with drivers' eye movements to a larger degree than a concurrent verbal
task (Recarte & Nunes, 2000). Cognitive interference is greatest for
tasks that demand the same resources.

Further, recent extensions of multiple resource theory identified
separate visual processing resources: ambient and focal. In driving,
ambient vision supports lane keeping and focal vision is critical for
event detection (Wickens, 1984; 2002). A meta-analysis of the effect of
cell phone use on driving performance showed that hand-held phones that
demand focal processing had a relatively small effect on lane keeping,
but that hands-free cell phones had a substantial effect on event
detection and response (Horrey & Wickens, 2006). However, even tasks
that draw upon different resources, such as cell phone conversations
(auditory verbal) and driving (visual motor spatial), can compete for
central processing resources (Pashler, 1998). Such competition can
undermine drivers' ability to respond to the roadway environment. This
issue is addressed with an experiment that examines the interaction of
visual and cognitive distraction.

Many studies have investigated cognitive distraction and how it affects
eye movement patterns and driving behavior. Recarte and Nunes (Recarte &
Nunes) found that increased cognitive load was associated with longer
fixations, smaller visual functional-field, and less frequent glances at
mirrors and the speedometer. Cognitive distraction undermines driving
performance by disrupting the allocation of visual attention to the
driving scene and the processing of attended information. For example,
cognitive workload impaired the ability of drivers to detect targets
across the entire visual scene and caused gaze to be concentrated in the
center of the driving scene (Recarte & Nunes, 2003a; Victor, 2005). In
addition, cognitive distraction associated with cell phone conversations
negatively affected both implicit perceptual memory and explicit
recognition memory for items that drivers fixated while driving
(Strayer, Drews, & Johnston, 2003b). A meta-analysis of twenty-three
studies investigating the effects of cell phone conversation found that
cognitive distraction delays driver response to hazards (Horrey &
Wickens, 2006). For example, drivers reacted more slowly to brake events
(Lamble, Kauranen, Laakso, & Summala, 1999; J. D. Lee, Caven, Haake, &
Brown, 2001) and missed more traffic signals (Strayer & Johnston, 2001)
when they were performing email, math, or cell phone conversation tasks
while driving. Although the negative effects of cognitive distraction on
driving have been demonstrated, little research has considered how such
effects could be used to detect cognitive distraction in real time.

A promising strategy to address this challenge is to classify driver
state in real time and use this classification to adapt in-vehicle
technologies to mitigate the effects of distraction (Donmez, Boyle, &
Lee, 2003a, 2003b). This strategy is not new. For example, "attentive
autos," which monitor driver attention and emit an alert when the driver
looks away from road or when driving demands require a high level of
attention, have been studied (Gibbs, 2005). Smith and his colleagues
developed a robust system using head rotation and eye blinking to
monitor the lack of visual attention due to fatigue while driving
(Smith, Shah, & Lobo, 2003).The degree of driver stress (Healey &
Picard, 2005) and vigilance (Bergasa, Nuevo, Sotelo, Barea, & Lopez,
2006) was predicted from physiological measures and used to help manage
IVIS functions. Also, some studies have used data mining techniques to
predict drivers' intent to change lanes to enhance driver-assistance
systems (Mandalia & Salvucci, 2005; McCall, Mipf, Trivedi, & Rao, in
press; McCall & Trivedi, 2006).

Obviously, measuring driver state is a core function in such systems. To
fulfill this function and avoid intrusive measurement (e.g., measuring
galvanic skin response using electrodes), this paper presents an
unobtrusive approach that uses eye movements and driving behavior to
detect driver cognitive distraction.

The following sections first present two experiments aimed at
understanding the mechanisms underlying cognitive distraction and its
interaction with visual distraction. Following the experiments, several
different algorithms are developed to assess cognitive distraction in
real time. The first of these examines the potential of support vector
machines, and the second uses Bayesian networks. Finally a comparison
between support vector machines and Bayesian networks is presented.

# Interaction of Cognitive and Visual Distraction 

The behavioral manifestation of cognitive distraction is often the
failure to detect events and respond in a timely manner. One possible
cause of failure to respond to the environment is that performing a
secondary task degrades the encoding and transferring of foveated visual
information into short-term memory. Studies have shown that drivers
detected (McCarley et al., 2004) and recognized (Strayer, Drews, &
Johnston, 2003a) fewer objects when performing a secondary task while
driving compared to driving only; however, the number of fixations on
the target region were not different for the two conditions. The
difficulty in responding to and recognizing previously fixated stimuli
in a dual-task condition may relate to the tendency for one stimulus to
interfere with the processing of a subsequent stimulus (Shapiro & Luck,
1999), such that drivers' attention to a non-driving task interferes
with the consolidation of information into short-term memory during
fixations.

Failure to respond to the environment may also be caused by disruptions
in the distribution of visual attention while performing a secondary
task. Several researchers have evaluated eye movement patterns to assess
how drivers' inspection behavior changes as a function of cognitive
load. Drivers glanced at the mirror and the speedometer less frequently,
and their distribution of glances to the road became more concentrated,
when performing cognitively demanding tasks while driving (Recarte &
Nunes, 2000, 2003b). This reduction of the area scanned by the driver
could decrease the probability of detecting traffic events in that
attention is not directed toward those events. Non-driving secondary
tasks may disrupt drivers' attention to the roadway, resulting in fewer
objects and changes being fixated and attended to.

Visual attention can be guided to objects in the visual field by
endogenous control (also called goal-driven, conceptually-driven, or
top-down control) and by exogenous control (also called data-driven,
stimulus-driven, or bottom-up control). Endogenous control refers to
strategic information processing because an observer intentionally
directs attention towards relevant stimuli. Exogenous control refers to
the direction of attention elicited by characteristics of the visual
field and implies automatic or mandatory information processing (Jonides
& Irwin, 1981; Posner, 1980; Theeuwes, 1991). Previous studies using a
cue-target paradigm have manipulated the predictive validity of a
centrally located symbolic cue that pointed to one of several stimulus
positions to assess the role of endogenous control. These studies have
also assessed the role of exogenous control through a non-predictive
abrupt onset (Jonides & Irwin, 1981; Posner, 1980). Results have
generally shown that reaction times are shorter when a target appears in
a cued, rather than an uncued, location. Jonides (1981) found an
interactive effect such that endogenous control in response to a central
predictive cue was affected by concurrent memory-load, whereas exogenous
control in response to a non-predictive cue was not.

Generalizing to driving, when particular information is relevant to the
driver, endogenous control purposely directs attention to particular
features in the driving environment. On the other hand, exogenous cues,
such as abrupt movements, draw attention to a particular object or
location without drivers' intention. Based on Jonides' findings,
cognitive load would be expected to interfere more with drivers'
attention to safety-relevant objects, which is governed by endogenous
control, than with their attention to salient objects, which depends on
exogenous control.

The change blindness paradigm offers a promising way to assess the
effect of cognitive load on visual attention. When changes occur during
a brief occlusion of the scene, as in the flicker paradigm, observers
have trouble detecting them even when the changes are large, are
presented repeatedly, and are expected to occur (Rensink, O\'Regan, &
Clark, 1997). Observers do not have trouble detecting changes without
the brief occlusion. A common explanation for these findings is that the
brief occlusion of the scene disrupts and masks the exogenous cues
associated with the abrupt onsets that would normally guide attention to
the change (Rensink et al., 1997). Several variations of the change
blindness paradigm support this explanation, although visual working
memory limits may also contribute to these effects (Luck & Vecera,
2002). When an abrupt onset was added to the pre-change image prior to
the disruption of the scene, detection was easier if the changed item
was the abrupt onset (Scholl, 2000). Likewise, when high-contrast
patterns and changes were both presented in a scene, as in the mudsplash
paradigm, observers struggled to detect changes because the
high-contrast patterns served as exogenous cues that drew attention away
from the location of the change (Rensink et al., 1997). Such results
suggest that the brief disruption in the change blindness paradigm
interferes with change detection by masking abrupt onsets that normally
support exogenous control of visual attention (Jonides & Irwin, 1981;
Simons & Rensink, 2005).

Consistent with the characteristics of endogenous control, in the
presence of a brief disruption, objects that are more meaningful
(Pringle, Irwin, Kramer, & Atchley, 2001), more relevant to traffic
safety (Dornhoefer, Unema, & Velichkovsky, 2002), or are of central
interest (e.g., objects considered to be important in the scene)
(Rensink, 1997) are better detected. Others have observed that change
detection was impaired when the advantage for changes of central
interest was eliminated by inverting the scenes (Kelley, Chun, & Chua,
2003; Shore & Klein, 2000). These results suggest that the change
blindness paradigm undermines exogenous control of attention, but leaves
endogenous control relatively unaffected. However, it is also possible
that, in the presence of a brief disruption, observers must rely on
visual short term memory to determine if there is a change (Hollingworth
& Henderson, 2002; Hollingworth, Williams, & Henderson, 2001). Without
the brief disruption, memory is less critical in detecting changes
because all the information is available to the observer.

In the driving domain, several researchers have used the change
blindness flicker paradigm (Rensink et al., 1997) to study how drivers
detect roadway events. According to this paradigm, participants view a
sequence of unaltered and altered images of a traffic scene from the
driver's perspective, with a brief gray screen between the images
(McCarley et al., 2004; Richard et al., 2002). Cognitive load undermined
detection of driving-relevant (objects that contained important driving
information) and driving-irrelevant (details that were not associated
with driving) changes to a similar degree (Richard et al., 2002). In
another study, there was a tendency for cognitive load to impair
knowledge-driven orienting of attention in older adults, but not in
younger drivers (McCarley et al., 2004). In related work, Zheng (2004)
developed a dynamic change blindness paradigm in which he asked drivers
to detect changes that occurred during brief disruptions in a simulator
drive. The results indicated that detection of safety-relevant changes
(vehicles that changed location in traffic lanes) was more affected by
cognitive load compared to safety-irrelevant ones (changes in vehicle
features in traffic lanes). However, safety relevance was confounded
with vehicle location and vehicle features, making a definitive
interpretation of these data difficult. Generally speaking, cognitive
load undermines detection of changes that are relevant to the driving
task more than detection of irrelevant changes. These results have
partially confirmed Jonides' (1981) finding that cognitive load was
particularly detrimental to the endogenous control of attention.

Previous studies have not, however, addressed the combination of
cognitive load with and without visual disruption in a dynamic driving
environment. Whether short glances away from the driving scene and
cognitive load have an additive or interactive effect on drivers'
ability to detect changes has important practical and theoretical
implications.

The objective of the current study is to compare the effects of
cognitive load on the endogenous and exogenous guidance of visual
attention using a dynamic change blindness paradigm. Changes that
occurred during driving scenarios were masked by a one-second gray
screen so that the effects of cognitive load on endogenous and exogenous
control of attention could be compared. The duration of the visual
disruption simulated the time drivers might glance away from the forward
view to either check the rearview mirror or interact with an in-vehicle
information system (Sodhi, Reimer, & Llamazares, 2002). It was
anticipated that cognitive load would diminish endogenous, rather than
exogenous, control. Specifically, cognitive load was expected to
undermine change detection to a greater degree in the presence of visual
disruptions compared to detection performance in the absence of visual
disruptions.

## Experiment 1: Change detection and safety-relevance

A dynamic change blindness paradigm (Zheng, 2004) was implemented in a
driving simulator. A brief visual disruption was designed to remove the
transients that normally accompany changes in the visual field, leaving
visual attention to be guided by endogenous control. An in-vehicle
information system imposed a cognitive loading task that required
drivers to listen to auditory messages and respond to questions.

### Method 

*Participants*. Twelve native English speakers (5 men and 7 women)
participated in the experiment. Participants ranged in age from 22 to 28
years, with an average age of 25 (standard deviation (sd) = 2.2). All
drivers were screened for visual acuity, color perception, and depth
perception using an Optec Vision Tester. The drivers had at least five
years of driving experience, drove at least three times per week, and
possessed a valid driver's license. Participants were paid \$15 an hour,
with additional compensation (up to \$10) based on auditory task
performance. The purpose of providing a bonus was to encourage
participants to engage in the secondary task.

*Apparatus and tasks*. A fixed-based, medium-fidelity driving simulator
was used to conduct the experiment. The simulator uses a 1992 Mercury
Sable vehicle cab that has been modified to include a 50-degree visual
field of view, force feedback steering wheel, and a rich audio
environment. The fully textured graphics are generated by DriveSafety's
Vection^TM^ software that delivers a 60-Hz frame rate at 1024 x 768
resolution. Data were collected at a rate of 60 Hz.

Each of the four driving scenarios included a straight, four-lane
suburban road with a parking lane on each side. Each drive was
approximately 6.5 miles long, and participants were asked to maintain a
speed of 30 mph. The drive took approximately 13 minutes to complete.
Participants were instructed to drive normally, as they would in a real
driving environment.

During two of the four drives, the change detection task was
administered using a dynamic change blindness paradigm. The projection
screen was blanked for one second and replaced with a homogeneous gray
screen. Participants were told that the projection screen might blank,
and that a change to one of the surrounding vehicles could occur during
the blank. In the other two drives, participants were told that the
screen would not blank, but that changes would occur during the drive.

Changes occurred when participants reached pre-designated locations.
These locations were situated approximately every 200 meters, or every
15 seconds if the driver maintained the recommended speed. Participants
were asked to identify the type of change by pressing buttons on the
steering wheel. Two response buttons on the left of the steering wheel
were used to identify forward and backward vehicle changes in the
traffic lane. Two response buttons on the right were used to identify
movement changes (forward and backward) and property changes (color and
identity) in the parking lane. Buttons were labeled so participants
could easily identify which to use.

While driving, participants were also asked to listen and respond to an
auditory task (Reyes & Lee, 2004), which presented information about
cost (one or two dollar signs), quality (one or two stars), and wait
time (short or long) at three different restaurants. The following is an
example of an auditory message:

> "There are three restaurants located in the area. One restaurant is
> Louee's Diner, which has an average entrée price of one dollar sign
> and a quality rating of one star. There is currently a long wait time
> at Louee's Diner. Another restaurant is Pat's Place, which has an
> average entrée price of one dollar sign and a quality rating of two
> stars. There is currently a long wait time at Pat's Place. The last
> restaurant is Tee Jay's Pizza, which has an average entree price of
> two dollar signs and a quality rating of two stars. There is currently
> a short wait time at Tee Jay's Pizza."

Questions posed at the end of each message required participants to
transform the presented information and relate it to categories of
restaurants. For example, a question, delivered in an auditory format,
was: "Which restaurant could have an average entrée price of \$5 and has
a quality rating of more than 10 positive recommendations?" Participants
learned the definitions of the restaurant categories and were given two
sample messages during the practice session. They were required to
answer each question verbally with the appropriate restaurant name, and
were encouraged to provide their best answer if they were unsure. The
voice of the auditory stimuli was a synthetic English-speaking male
adult.

*Experimental design and independent variables.* The experiment was a 2
(blanking: blank, no-blank) x 3 (change: forward, backward,
parked-vehicle) x 2 (auditory task: task, no-task) within-subjects
design. Each participant drove four experimental drives, two with
blanking of the screen (blank) and the other two without (no-blank). The
order of the drives was counterbalanced according to a Latin square
design. There were three possible changes to the vehicles in front of
the participant vehicle, which had different degrees of
safety-relevance. The backward changes in the traffic lane were
considered to be more safety-relevant and the forward changes in the
traffic lane less safety-relevant. The changes to parked-vehicles were
considered to be safety-irrelevant. Both lead and parked vehicles were
initially located 60 meters ahead of the participant vehicle. The
forward change moved the lead vehicle in the right lane (directly ahead
of the participant) forward 18 meters. The backward change moved the
lead vehicle in the right lane 18 meters closer to the participant
vehicle. A parked-vehicle change consisted of either changing the
vehicle's location along the parking lane (backward or forward) by 18
meters, or changing its color or identity. Each type of change was
encountered twelve times during a drive, and each change was accompanied
by a screen blanking in the blank condition. Twelve no-change catch
trials were included to prevent participants from associating changes
with the blanking. The same 36 changes occurred at different locations
in the no-blank condition.

One drive from each blanking condition contained an auditory task with
four unique message sets. Each message was played twice for a total of
150 seconds. Immediately after the repetition of the message, drivers
were asked six questions about the restaurants.

*Procedure*. After participants signed the necessary IRB consent forms,
they were introduced to the driving task, the change detection task, and
the auditory task. They then drove a ten-minute practice drive to become
familiar with the dynamics of the simulator and experience the
change-detection task and the message system. For each drive,
participants were instructed to always maintain their position in the
center of the right lane. Drivers were also instructed to press one of
the response buttons when they detected a change.

During each auditory task condition, four sets of pre-recorded auditory
messages were played. Participants were asked to answer the questions as
quickly as possible while driving and performing the change-detection
task. Upon completion of each drive, participants were asked to rate on
a 1 to 10 scale (1 = least confident; 10 = most confident) their
subjective confidence that they had detected the changes and answered
the auditory task questions correctly. The experiment took approximately
two hours to complete.

*Dependent variables and scoring*. The dependent variables included
drivers' sensitivity to changes (*d'*), subjective confidence ratings,
and performance on the auditory task. The confidence ratings were
collected using a single item rating in which drivers rated their
subjective confidence in their detection performance. A signal-detection
approach was used to analyze change-detection performance. A hit was
counted if participants detected a change and correctly pressed the
corresponding button within 2.5 sec after the onset of the change event.
A miss was counted if, within 2.5 seconds, participants either failed to
press a button or pressed the incorrect button. A false alarm was
defined as pressing a button when there was no change. A correct
rejection was defined as not pressing any button when there was no
change in the blank conditions. In order to count the number of false
alarms and correct rejections in the no-blank conditions, twelve
pre-designated locations were time-stamped to correspond to the twelve
no-change catch trials in the blank conditions. d' values were
calculated based on the difference between the likelihood of pressing a
button correctly when there was a change and the likelihood of pressing
a button in the no-change conditions (Macmillan & Creelman, 2005).

### Results 

The effects of the independent variables on d' and confidence were
analyzed with a repeated measures ANOVA. The statistical model was
designed to compare the effects of the auditory task and blanking on
change detection. Changes were distinguished according to their safety
relevance to drivers, with changes that moved toward the drivers being
more safety-relevant, changes that moved away from drivers being less
safety-relevant, and changes in the parking lane being
safety-irrelevant. Results for the color/identity changes in the parking
lane were excluded from the analysis because these changes were not
comparable to the forward and backward changes in the traffic and
parking lanes. The data were checked to ensure compliance with the
normality assumptions (Kolmogorov-Smirnov test for normality, p = .058)
and homogeneity of variance (Levene's test, p value ranged from .052 to
.898, except for auditory task on confidence, F(1,142) = 5.48, p =
.021). Cohen's d was also calculated to show the magnitude of the effect
of the auditory task and blanking on d' and confidence. Post-hoc tests
were conducted using pair-wise comparisons with Bonferroni adjustments.

*Sensitivity to changes.* Participants were less sensitive to vehicle
changes during the blank condition (F(1,121) = 34.73, p \< .0001, d =
.88); the auditory task also diminished sensitivity to changes (F(1,121)
= 4.23, p = .042, d = .28). The magnitude of the effect of blanking was
greater than the effect of the auditory task. The significance of the
main effects and non-significance of the interaction effect (F(1,121) =
.50, p = .481) suggest that blanking and the auditory task had an
additive effect on sensitivity (Figure 5.4).

![](media/image5.wmf)

[]{#__RefHeading___Toc173654914 .anchor}Figure 5.4. The mean d' (± SE)
as a function of blanking and auditory task in Experiment 1.

Participants were most sensitive to changes when the lead vehicle moved
backward (d' = 1.95) toward the participant and least sensitive to
parked-vehicle changes (d' = 1.11) (F(2,121) = 8.98, p = .0002). The
mean sensitivity of forward vehicle changes was 1.61. The backward
movement increased the visual angle of the lead vehicle from 0.86 to
1.15 degrees, an increase of 33.7%. In contrast, the forward movement
decreased the visual angle to 0.67 degrees, a decrease of 22.1%. To
determine whether the superior change detection was influenced by size
or safety, a subsequent experiment was conducted (Experiment 2).

The interaction between type of change and blanking failed to reach
significance (F(2,121) = 2.60, p = .078), though the means were in the
expected direction (Figure 5.5). Parked-vehicle changes were often
unnoticed (d' = .38) when they occurred during blanking. The effect of
the auditory task on d' was similar for different types of changes.

*Confidence in detecting changes*. Participants were less confident in
detecting changes during the blank condition (F(1,121) = 9.10, p = .003,
d = .38) and when they were cognitively loaded with an auditory task
(F(1,121) = 19.92, p \< .0001, d = .58). The magnitude of the effect of
the auditory task was greater than that of blanking, which is contrary
to the effect sizes for d' (the black condition: 0.88; auditory
condition: 0.28). The interaction between auditory task and blanking was
not significant (F(1,121) = 1.46, p = .230).

Confidence was highest with the backward changes (mean = 7.48), followed
by the forward changes (mean = 6.51), and finally, the parked-vehicle
changes (mean = 5.39) (F(2,121) = 27.61, p \< .0001). There were no
significant interactions between the type of change and either auditory
task or blanking.

The relationship between d' and confidence was positive in all the
experimental conditions. The correlation between d' and confidence was
significant for the task (r(72) = .29, p = .014) and no-task (r(72) =
.55, p \< .0001) conditions and for the blank (r(72) = .41, p = .0003)
and no-blank conditions (r(72) = .35, p = .002).

![](media/image6.wmf)

[]{#__RefHeading___Toc173654915 .anchor}Figure 5.5. The mean d' (± SE)
as a function of different types of changes and blanking and auditory
task in Experiment 1. []{.mark}

*Secondary task performance*. Performance on the auditory task was not
strongly related to participants' ability to detect changes (r(72) =
.05, p = .666). Participants did not systematically neglect the auditory
task to improve their detection performance, nor did they neglect the
detection task to focus only on the auditory task. However, participants
answered slightly fewer questions correctly during the blank condition
(mean = 79%) (F(1,59) = 6.42, p = .013) than the no-blank condition
(mean = 83%). This finding suggests that drivers considered the auditory
task secondary to driving and that there was a slight tendency to
neglect it when the change detection demands increased.

### Discussion 

The introduction of auditory tasks and brief blanking of the driving
scene diminished participants' sensitivity to changes, as well as their
confidence in detecting them. The diminished sensitivity to changes is
consistent with Zheng's (2004) findings. Even though the safety-relevant
changes were detected more reliably compared to the safety-irrelevant
changes, cognitive load uniformly diminished the detection of both types
of changes. This finding concurs with that of Richard et al. (2002), who
observed that performing a non-driving secondary task impaired drivers'
ability to detect driving relevant and irrelevant changes to a similar
degree. The decreased confidence in detecting changes suggests that
participants were aware that the cognitive load of the auditory task and
the blanking both diminished their performance.

Blanking and the auditory task affected d' and confidence to different
degrees. Blanking had a much stronger effect on drivers' sensitivity to
detecting changes compared to the auditory task; however, the auditory
task had a stronger effect on confidence in detecting changes. The
stronger effect of blanking on d' than on confidence suggests that
drivers may not be aware of the influence that brief glances can have on
performance. They may think that they detected changes efficiently when
in fact they did not. The correlations show a positive relationship
between d' and confidence, suggesting that participants were aware of
the effect of the experimental conditions on their change-detection
performance.

We hypothesized that cognitive load would be particularly detrimental to
detecting changes during the blanking condition, when endogenous control
guides attention. The non-significant interactions between auditory task
and blanking suggest that cognitive load diminishes detection
performance to a similar degree whether exogenous cues are available to
guide attention or not. This finding indicates that cognitive load and
short visual disruptions are additive in their tendency to undermine
detection of roadway events. The lack of an interaction may be the
result of drivers compensating with methods such as attending less to
the auditory task. In fact, participants did answer fewer questions
correctly during the blanking condition, in which change detection
depended on endogenous control.

Participants were most sensitive to changes when the lead vehicle moved
backward. One explanation is that backward movements were more
safety-relevant and might have required driver intervention. The
safety-relevant movement may have influenced the endogenous control of
attention, thereby drawing drivers' attention toward it. Another
explanation is that this change also caused the image size of the lead
vehicle to increase, and the retinal expansion may have contributed to a
looming cue, making the backward change a salient exogenous cue (D. N.
Lee, 1998; Regan & Vincent, 1998). Experiment 2 was designed to further
investigate whether the relatively higher d' for backward changes was
due to the endogenous influence of safety relevance or the exogenous cue
associated with the increased visual angle.

## Experiment 2: Image size and safety relevance

In Experiment 1, participants were most sensitive and confident in
detecting backward movements of the lead-vehicle. This backward movement
made for a larger, more salient exogenous cue. It also imposed a
safety-relevant situation, and so constituted a stronger endogenous cue
compared to a forward change in the lead-vehicle position. Experiment 2
was designed to identify the cause of higher d' for detecting vehicles
that moved closer to drivers.

### Method 

The protocol for Experiment 2 is discussed only to the extent that it
differs from the protocol used in Experiment 1.

*Participants*. Twelve native English speakers (3 men and 9 women)
participated in the experiment. Participants ranged in age from 20 to 26
years, with an average age of 22 (sd = 1.7). No participants took part
in both experiments.

*Apparatus and tasks*. Arrangement of the response buttons on the
steering wheel was slightly different in Experiment 2: the upper left
button corresponded to change-to-left-lane changes, the upper right
button corresponded to change-to-right-lane changes, the lower left
button corresponded to color/identity changes in the parking lane, and
the lower right button corresponded to location changes in the parking
lane.

*Experimental design and independent variables*. The experiment used a 2
(blanking: blank vs. no-blank) x 3 (change: left vs. right vs. parked
vehicle) x 2 (auditory task: task vs. no-task) within-subjects design. A
left change moved a vehicle that drove ahead of the participant vehicle
to the left lane, out of the participants' lane. A right change moved a
vehicle from the left lane to the right lane, directly ahead of the
participant vehicle. The right changes are of immediate safety-relevance
to drivers since they place the vehicle directly into the lane in which
the participant is driving. In contrast, the left changes are less
safety-relevant. The left and right changes were further broken into two
location categories: near and far. For both left and right changes, six
occurred at the near location and six at the far location. The vehicle
arrangements and changes were purposely configured to be comparable to
those in Experiment 1. The near location corresponded to the end
position of a backward change and the far location corresponded to the
initial position of a backward change. Parked-vehicle changes were the
same as those in Experiment 1. A pace car was placed seven meters ahead
of the participant vehicle in the left lane and drove at 30 mph. The
participants were asked to maintain their speed relative to the pace car
and to keep it in sight throughout the drives.

### Results 

Results for the color/identity changes in the parking lane were excluded
in the analysis. As with the first experiment, the assumptions for
normality (Kolmogorov-Smirnov test for normality, p = .061) and
homogeneity of variance (Levene's test, p values ranged from .36 to
1.00, except for the effect of change type on confidence, F(2,141) =
3.67, p = .028) were verified before the analysis of variances were
conducted.

*Sensitivity to changes*. Participants were less sensitive to changes
during the blank condition (F(1,121) = 44.25, p \< .0001, d = .60) and
while performing the auditory task (F(1,121) = 16.05, p = .0001, d =
.35). As in Experiment 1, the magnitude of the effect of blanking was
greater than that of the auditory task. Similar to Experiment 1, the
non-significant auditory x blanking interaction (F(1,121) = 0.18, p =
.674) suggests that the effects of blanking and cognitive load are
additive (Figure 5.6).

Participants were similarly sensitive to vehicles moving to the left (d'
= 2.79) or the right (d' = 2.91), but were less sensitive to the changes
to parked vehicles (d' = 1.15) (F(2,121) = 133.32, p \< .0001). To
identify the cause of higher d' for detecting vehicles that moved closer
to drivers, we performed a separate analysis comparing the main effect
of change location on d'. Change location of the moving vehicles
affected participants' sensitivity (F(3,165) = 3.90, p = .010), with
greater sensitivity for the close location vehicles (d'=2.46) compared
to the far location vehicles (d'=2.25). Post-hoc comparisons showed that
vehicle changes to the right were detected no better than vehicle
changes to the left at close (t(165) = 1.19, p = 1.000) and far (t(165)
= 0.64, p = 1.000) locations. This finding suggests that perhaps image
size and location, rather than safety relevance alone, affects
sensitivity in detecting changes.

![](media/image7.wmf){width="3.5208333333333335in"
height="3.1034722222222224in"}

[]{#__RefHeading___Toc173654916 .anchor}Figure 5.6. The mean d' (± SE)
as a function of blanking and auditory task in Experiment 2.

![](media/image8.wmf){width="3.15625in"
height="2.7680555555555557in"}![](media/image9.wmf){width="3.1034722222222224in"
height="2.722916666666667in"}

[]{#__RefHeading___Toc173654917 .anchor}Figure 5.7. The mean d' (± SE)
as a function of different types of changes and blanking and auditory
task in Experiment 2.

The blanking x change type interaction for d' was significant, F(2,121)
= 21.39, p \< .0001 (Figure 5.7). Blanking diminished d' for
parked-vehicle changes (t(121) = 9.17, p \< .0001), but not for left and
right changes. Parked-vehicle changes were often unnoticed (d' = .36)
when they occurred during blanking. The effect of the auditory task on
d' was uniform for different types of changes (Figure 5.7). Neither
safety-relevance, changes into or out of the drivers' lane, nor
centrality of the change affected the degree to which cognitive load
impaired detection. Given that blanking had a significant effect on
detecting parked-vehicle changes, while the auditory task did not, we
would expect to have a significant three-way (auditory task x blanking x
type of change) interaction. However, our results did not reveal this
effect (F(2,121) = 0.38, p = .684).

*Confidence in detecting changes*. Consistent with change-detection
performance, participants were less confident during the blank condition
(F(1,121) = 31.53, p \< .0001, d = .50) and when there was an auditory
task (F(1,121) = 29.95, p \< .0001, d = .48). Unlike Experiment 1, where
cognitive load had a greater effect on confidence than on change
detection performance, here the effect on confidence was similar, even
though blanking had a larger effect on change-detection performance.

Participants were similarly confident in detecting the left (mean =
7.68) and right (mean = 7.77) changes and were less confident in
detecting the parked-vehicle (mean = 4.64) changes (F(2,121) = 110.71, p
\< .0001). There were no significant interactions between the type of
change and the auditory task or blanking for confidence.

The correlation between d' and confidence was significant for the task
(r(72) = .42, p = .0002) and no-task conditions (r(72) = .36, p = .001)
and for the blank (r(72) = .47, p \< .0001 and no-blank conditions
(r(72) = .33, p = .004). The magnitude of the correlations was
comparable between Experiment 2 and Experiment 1.

*Secondary task performance*. There was little evidence of auditory
task-detection trade-off (r(72) = .03, p = .772), suggesting that
participants did not neglect the auditory task to improve their
detection performance. Contrary to Experiment 1, no differences were
observed in secondary task performance between the blank and no-blank
conditions (F(1,59) = 1.85, p = .179). It is possible that because the
detection task was less demanding with lateral movements, participants
could devote more attention to the auditory task. The magnitude of the
effect of the auditory task for the two experiments was similar for d'
(.28 vs. .35) and confidence (.60 vs. .50). In contrast, sensitivity in
detecting a change in the lead vehicle was substantially lower in
Experiment 1 (d'= 1.78) compared to Experiment 2 (d'=2.85). It is most
likely that the lack of difference in secondary task performance in
Experiment 2 was due to the more demanding detection task in Experiment
1.

### Discussion 

Consistent with the findings in Experiment 1, the presence of an
auditory task diminished participants' sensitivity to changes and their
confidence in detecting them. Cognitive load uniformly decreased the
detection of all types of changes. The decreased confidence in detecting
changes suggests that participants were aware of their performance
degradation when they were cognitively loaded with an auditory task. The
tendency for cognitive load and short glances to be additive in
affecting drivers' sensitivity to changes and confidence in detecting
them suggests that drivers will be least sensitive to roadway events
when structural and cognitive interference occur simultaneously.

Participants were similar in their sensitivity and confidence in
detecting right and left changes, even though the vehicle moving from
the left to right lane was assumed to be more safety-relevant. Similar
to Zheng's (2004) results, drivers were slightly more sensitive to
changes at near locations when compared to far, but much less sensitive
to changes in the parking lane. In combination, these results suggest
that drivers are sensitive to safety-relevant locations, such as the
traffic ahead of them, rather than to safety-relevant events. More
thorough manipulations of location and safety relevance are needed to
confirm these results.

The significant interaction between blanking and change type for d'
suggests that when searching is guided by endogenous control, changes
that are safety-irrelevant are less likely to be noticed. However, the
concept of safety-relevance coincides with spatial location in the
current study. We did not have safety-relevant events in the parking
lane. Therefore our results could also be explained in terms of the
location in the visual field such that drivers pay more attention to
objects in the traffic lanes and neglect objects on the side of the
road. Our results suggest that exogenous cues may be particularly
important in guiding drivers' attention to events that occur away from
the center of the road. More research is needed to provide further
understanding of whether drivers attend to objects according to their
safety-relevance or spatial location.

As in Experiment 1, d' was positively related to confidence, suggesting
that drivers were aware of how the experimental conditions affected
their detection performance. The strength of this relationship was
similar in the two experiments, even though the lateral movements of the
lead vehicle were more easily detected in the second experiment. Drivers
seem to be able to adjust their assessment of their performance on the
detection task according to its difficulty.

## General discussion

Using a dynamic change blindness paradigm, two experiments were designed
to investigate the effect of cognitive load on drivers' ability to
detect changes in the driving environment. The dynamic change blindness
paradigm creates a condition in which exogenous cues are masked by
visual disruptions, resulting in a situation in which visual attention
is primarily guided by endogenous control. We hypothesized that
cognitive load would diminish drivers' sensitivity and confidence in
detecting changes under these circumstances. The results indicate that
cognitive load uniformly diminishes participants' sensitivity to changes
and their confidence in detecting them, independent of safety-relevance
or lack of exogenous cues.

Jonides (1981) found that endogenous control was affected by concurrent
memory-load, whereas exogenous control was not. In his experiment, the
demands of a memory task interfered with endogenous control associated
with the central cue, but left the exogenous control associated with the
peripheral cue relatively unaffected. Instead of confirming this
interactive effect, we found that cognitive load undermined change
detection to a similar degree when exogenous cues were masked and when
they were not. In addition, cognitive load undermined detection of
safety-relevant and irrelevant events similarly. Therefore, our results
suggest that cognitive load undermines both endogenous and exogenous
control of attention---the safety-relevance or saliency of an object
does not guarantee detection if drivers are cognitively loaded.

Both experiments also showed that masking exogenous cues greatly
diminishes drivers' detection of events that occur away from the center
of the roadway. Driver training and experience may lead people to
monitor the center of the road and to depend on exogenous cues for
safety-relevant events that occur on the side of the road. Such
expectations enabled drivers to accommodate the lack of exogenous cues
in detecting changes in the center of the road, but left them vulnerable
to those occurring on the side. Such a process may be an effective
adaptation to routine driving situations in which drivers and
pedestrians obey the rules of the road, but may fail when the unexpected
occurs. Overall, drivers' ability to detect roadway events is affected
by a combination of structural and cognitive interferences, with
structural interference being particularly detrimental to events away
from the center of the road.

An alternate explanation of these results is that cognitive load
diminishes event detection primarily because it degrades information
consolidation. Drivers miss detecting objects because these objects
(even though previously fixated) are not properly consolidated and
transferred into short-term visual memory. This is similar to the
attentional blink phenomenon (Shapiro & Luck, 1999), which states that
drivers may fail to respond appropriately even if they have looked at
objects in the scene because they do not form a durable short-term
memory of them. Additional research is needed to understand how drivers
scan the environment due to cognitive load. Eye-movement analyses can
provide more insight on how cognitive load influences the way in which
drivers detect objects in the roadway. One possibility is that the
probability of fixating certain objects declines when drivers are
cognitively loaded. Another possibility is that the probability of
detecting a change given a fixation declines. The first alternative
would suggest a failing of visual attention and the second would support
a failing of consolidation.

Further, the visual disruptions may not have neatly separated the two
mechanisms that guide attention. Unlike many change blindness
experiments that present people with unique changes, this experiment
included a limited number of changes and locations. In the absence of
blanking, the repetition of changes likely led participants to monitor
changes according to an attentional set, which suggests that
exogenous-guided attention may be influenced by endogenous factors
(Folk, Remington, & Johnston, 1992). Furthermore, in situations where
exogenous cues were supposedly eliminated by the visual disruptions, the
post-blank vehicle that had undergone a backward movement was
substantially larger, and the retinal expansion or looming effect (D. N.
Lee, 1998; Regan & Vincent, 1995) may have made the vehicle more salient
(Franconeri & Simons, 2003). Thus, the looming vehicle may provide an
additional exogenous cue that is not eliminated by the visual
disruptions. The dynamic change blindness paradigm offers a promising,
but imperfect, method for assessing the role of endogenous and exogenous
control of attention in driving.

The dynamic change blindness paradigm is a more ecologically valid
approach to studying how drivers attend to events in the environment
when compared to the static change blindness paradigm. However, its
validity is challenged by the artificial technique used to simulate
glances away from the road and eliminate exogenous cues associated with
roadway changes. In contrast to a natural driving situation, drivers did
not choose when they would "glance away" from the road. In reality,
drivers might carefully time glances and be particularly attentive to
the situation before and after such a glance. The decrement in change
detection observed in this study may overestimate the consequence of
short glances away from the road. In addition, this study required
drivers to engage in a cognitively demanding task, one that many drivers
might not attempt. However, the number of drivers who use cell phones
and even read newspapers while driving suggests that such tasks are not
beyond what many drivers might attempt in the coming years
(Glassbrenner, 2005).

Although the artificial nature of some aspects of this study limit its
generalization to actual driving situations, the results show that both
cognitive and structural distractions can have profound consequences for
detecting changes in the driving environment and that drivers may not
always be aware of these consequences. Even brief glances away from the
road may make drivers vulnerable to neglecting changes, particularly
those occurring in the periphery. This could exacerbate drivers'
tendency to neglect safety-critical events that occur to the side of the
roadway (Fisher et al., 2002). Drivers' appreciation for these
consequences is imperfect; they may underestimate the consequence of
seemingly inconsequential distractions---a brief glance---compared to
more obvious distractions. These results suggest that drivers may
benefit from feedback regarding how in-vehicle information systems
undermine visual attention (Donmez et al., 2003b).

# Support Vector Machines to Detect Cognitive Distraction 

Providing drivers with feedback regarding their distraction requires
that that distraction first be estimated. One promising way to estimate
distraction is by monitoring drivers' eye movements. Three fundamental
types of eye movements---fixations, saccades, and smooth pursuit---could
reflect allocation of visual attention and consolidation of fixated
information. Fixations occur when an observer's eyes are nearly
stationary. The position and duration relate to attention orientation
and the amount of information perceived from the fixated location,
respectively. Saccades are very fast and straight eye movements that
occur when visual attention shifts from one location to another. Smooth
pursuits occur when the observer tracks a moving object, such as a
passing vehicle; these eye movements serve to stabilize an object on the
retina so that visual information can be perceived even when the object
is moving relative to the observer. In the context of driving, smooth
pursuits function similarly to fixations, since most observed objects in
the scene are moving. Nonetheless, pursuits depict a dynamic eye
movement while fixations depict a static movement. To reflect this
difference, we use two sets of measures to describe these movements in
this study.

Some studies have shown links between eye movements, cognitive workload,
and distraction (Hayhoe, 2004). The range of saccade distances decreases
as tasks become complex, which indicates that saccades may be a valuable
index of mental workload (May, Kenned, Williams, Dunlap, & Brannan,
1990). Rantanen and Goldberg (1999) found that visual field shrunk 7.8%
during a moderate-workload counting task, and 13.6% during a
heavy-workload counting task. Similarly, increased cognitive workload
during driving decreased spatial gaze variability, defined by the area
covered by one standard deviation of gaze location in both the
horizontal and vertical directions (Recarte & Nunes, 2000, 2003b). These
links between eye movements and cognitive workload show that eye
movement measures are good candidates for predicting cognitive
distraction.

Although some studies have related cognitive distraction to eye
movements and disruptions in visual attention, little research has
considered how eye movement data may be used to detect distraction in
real time. Furthermore, most studies (May, Kennedy, Williams, Dunlap, &
Brannan, 1990; Recarte & Nunes, 2000, 2003b; Strayer et al., 2003a)
consider the relationship between cognitive distraction and eye movement
using linear, univariate approaches (e.g., ANOVA). Here, we develop a
method of real-time detection of cognitive distraction and degraded
driving performance using Support Vector Machines (SVMs), which can
capture non-linear relationships and the interaction of multiple
measures that other approaches cannot.

Proposed by Vapnik (1995), SVMs are based on a statistical learning
technique and can be used for pattern classification and inference of
non-linear relationships between variables (Cristianini & Taylor., 2000;
Vapnik, 1999). This method has been successfully applied to the
detection, verification, and recognition of faces, objects, handwritten
characters, speech, speakers, and retrieval of information (Byun & Lee,
2002).

Figure 5.8 shows a simple representation of SVM classification of two
classes; the filled and open circles represent instances from each
class. These classes could represent, for example, distracted and
attentive driver states. The boundary between these classes is shown by
the line that encircles the filled circles in the graph on the left and
the line that divides the circles on the right. Each of the circles
represents an instance of one of the two classes and is defined by a
vector of numbers. In the case of driver distraction, these vectors
might include numbers describing the driver's behavior over time, such
as the average fixation duration over the previous 20 seconds and the
standard deviation of fixation location. Formally, this
distraction-related driver state data can be considered as labeled
binary-class data
($D = \left\{ \left( x_{i},y_{i} \right) \right\}_{i = 1}^{l}$,
$x_{i} \in X \subset R^{d}$, $y_{i} \in Y = \left\{ - 1, + 1 \right\}$,
where *x~i~* is a *d*-dimension real vector and *y~i~* is the class
label indicating which class the point *x~i~* belongs). The number of
dimensions, *d*, corresponds to the number of elements of the vector
used to describe the driver's state (e.g., average fixation duration,
standard deviation of fixation location). Using SVMs to identify when a
driver is distracted relies on the principle that a vector of numbers
can describe the state of the driver and that this vector can be
classified as either distracted or attentive.

SVMs identify the state vectors as belonging to one of the two classes
by dividing them with a hyperplane, which is a linear boundary in
*d*-dimensional space. The hyperplane is represented
by$w \cdot x - b = 0$, where w is a *d*-dimension vector that indicates
the boundary and *b* specifies the intercept. For SVM classification, an
optimal hyperplane is the one that provides the greatest separation from
the closest points from both classes, shown as the line in the graph on
the right of Figure 5.8. The greatest separation, also called maximum
margin, is the length of the orthogonal line between two hyperplanes,
which parallel the optimal hyperplane and touch the closest training
data points from each class. The data points that touch these
hyperplanes are the support vectors.

![](media/image10.wmf)

[]{#__RefHeading___Toc173654918 .anchor}Figure 5.8. A graphical
representation of the support vector machine algorithm.

The hyperplanes are defined as
$w \cdot x + b = 1$and$w \cdot x + b = - 1$, and maximum margin is
calculated by 2/$\left\| w \right\|$, where $\left\| \right\|$ is the
Euclidean norm. In this way, the training problem can be formulated as\
$\begin{array}{r}
\underset{w,b}{\text{min}}\ \frac{1}{2}\left\| w \right\|^{2} \\
s\text{.}t\text{.}\ y_{i}(w \cdot x_{i} + b) \geq 1,\ \forall i
\end{array}$ (1)\
where the constraint means that all the training data are located above
or below the hyperplanes and not between the optimal hyperplane and the
hyperplanes that touch the data points. The maximum margin presents the
minimized upper bound of generalization error.

When the training data can be separated by a linear boundary they are
linearly separable. Frequently, data are not linearly separable and
several techniques have been developed to address this situation. One
technique relies on the concept of a soft margin, which allows for
mislabeled data and makes it possible to choose the hyperplane boundary
that can split training data as cleanly as possible, recognizing that it
is not possible to draw a line that will place all members of a given
class on one side of the hyperplane. Such soft margins are implemented
by adjusting the formulation in (1)\
$\begin{array}{r}
\underset{\omega,b,\varepsilon}{\text{min}}\ \frac{1}{2}\left\| w \right\|^{2} + C\sum_{i = 1}^{l}\varepsilon_{i} \\
s\text{.}t\text{.}\ y_{i}(w \cdot x_{i} + b) \geq 1 - \varepsilon_{i}, \\
\ \varepsilon_{i} \geq 0
\end{array}$ (2)\
where $\varepsilon_{i}$presents a non-zero penalty of mislabeling data
point xi, and C is a predefined parameter used to balance a large margin
and large number of mislabeled points.

A second technique to accommodate data that are not linearly separable
is to create a non-linear hyperplane using a kernel that transforms the
original data to a high-dimensional form using a
function$\Phi( \cdot )$, and then identifies the linear hyperplane in
the high-dimensional space. The two graphs in Figure 5.8 show this
transformation. The left graph indicates the original data. These data
are transformed to another space, the right graph, by$\Phi( \cdot )$, in
which the transformed data can be separated by a linear hyperplane. The
linear hyperplane for the transformed data yields a nonlinear boundary
(the circle on the left graph) for the original data. The formulized
training problem with the kernel trick $x_{i}$ with $\Phi(x_{i})$, shown
in (3). The dot product of$\Phi( \cdot )$,
$K(x_{i},x_{j}) = \Phi(x_{i})^{T}\Phi(x_{j})$, is called a kernel
function and is more convenient to use in solving the training problem
than$\Phi( \cdot )$.\
$\begin{array}{r}
\underset{\omega,b,\varepsilon}{\text{min}}\ \frac{1}{2}\left\| w \right\|^{2} + C\sum_{i = 1}^{l}\varepsilon_{i} \\
s\text{.}t\text{.}\ y_{i}(w \cdot \Phi\left( x_{i} \right) + b) \geq 1 - \varepsilon_{i}, \\
\ \varepsilon_{i} \geq 0
\end{array}$ (3)

The SVM method is well-suited for measuring the cognitive state of
humans for several reasons. First, SVMs can accommodate the
relationships between human states and the measurable indicators of the
state that are often non-linear. SVMs can generate both linear and
nonlinear models and compute the nonlinear models as efficiently as the
linear ones. Second, SVMs can extract information from noisy data (Byun
& Lee, 2002) and do not require prior knowledge before training.
Although the relationship between cognitive distraction and eye movement
patterns has no well-defined theoretical basis, we believe that complex
relationships do exist between the two and that the SVM method may be
able to extract these relationships. Third, the SVM method avoids
overfitting by minimizing the upper bound of the generalization error
(Amari & Wu, 1999) to produce more robust models than traditional
learning methods (e.g., logistic regression) that only minimize training
error. Thus, the SVM method is a promising technique for detecting
cognitive distraction using eye movement patterns even though the
relationship between eye movements and cognitive state does not have a
clear theoretical relationship.

To evaluate the proposed method of using SVMs to detect driver
distraction and degraded driving performance in real time and to work
towards the implementation of such a system, this paper evaluates the
detection performance of SVMs, compares SVM performance with that of
logistic regression, and discusses the effects of three SVM model
characteristics on performance. Testing accuracy and signal detection
theory measures of sensitivity and response bias are used to assess the
models. Based on the characteristics of SVMs and eye movements, we
expect that the SVM method will be able to detect cognitive distraction
and degraded driving performance from eye movement and driving measures,
and that this method will outperform logistic regression.

## Model Construction

To build models, eye and driving data were collected in a simulator
experiment, during which ten participants performed three driving tasks
and interacted with an in-vehicle information system (IVIS). Raw eye
data were transformed into fixation, saccade, and smooth pursuit eye
movements, and steering wheel position was transformed into steering
error (Nakayama, Futami, Nakamura, & Boer, 1999b). In constructing the
models, SVM models were trained for each participant with different
values for three characteristics, including different definitions of the
model outputs (distraction definition), different combinations of input
variables (feature combination), and different summarizing parameters
for inputs (window size with overlap between windows). The resulting
models generate binary predications of the state of distraction (i.e.,
distracted or not distracted). Testing accuracy, model sensitivity, and
response bias were used to measure and compare model performance.

### Data source

Another experiment beyond that described earlier in this report was used
as a source of data for the algorithm development. This experiment is
described in more detail in the Phase 2 report for Task 6 IVIS Demand.

Participants: Ten drivers (6 male and 4 female), having normal or
corrected to normal vision and not wearing glasses, participated in this
experiment. They were between the ages of 35 and 55 (mean = 45, s.d. =
6.6), possessed a valid U.S. driver's license, had at least 19 years of
driving experience (mean = 30), and drove at least 5 times a week. They
were compensated \$15 per hour and could earn a bonus (up to \$10) based
on their performance of the secondary task.

Driving Task: The experiment took place in a fixed-based,
medium-fidelity driving simulator. The driving scenes were displayed on
a rear-projection screen with 768 x 1024 resolution; the screen was
placed 1.95 meters in front of drivers, producing approximately 50
degrees of visual field. The simulator collected data at 60 Hz.
Participants drove along a straight suburban street with two lanes in
each direction. The subject vehicle (SV; vehicle driven by the
participants) was equipped with a simulated cruise control system that
engaged automatically at 45 mi/h and disengaged when drivers pressed the
brake pedal. The participants were instructed to follow the vehicle in
front of them (the lead vehicle; LV) and to use the cruise control as
much as possible. The LV was coupled to the SV by a 1.8-second tailway.

The participants performed three driving tasks during each of six
15-minute drives. The first task was to follow the LV and respond to six
LV braking events during each drive. The timing of each braking event
was determined by the status of the IVIS task and was initiated by the
experimenter. During the events, the LV braked at a rate of 0.2 g until
it reached a minimum speed of no more than 20 mi/h and the participant
had braked at least once. Following a brief, random delay (0 to 5
seconds), the LV accelerated at a rate of 0.25 g until it reached a
speed of 25 mi/h. The LV was then coupled again to the SV with the
1.8-second tailway. The second task was to keep the vehicle from
drifting toward the lane boundaries and to drive in the center of the
lane as much as possible. The final task was to detect the appearance of
a bicyclist on the right side of the road in the driving scene by
pressing a button on the steering wheel. The bicyclist appeared about
three times per minute and was visible, on average, for about 2.8
seconds.

In-Vehicle Information System (IVIS) Task: During four of the six
drives, participants interacted with the IVIS, an auditory stock ticker.
The stock ticker was composed of 3-letter stock symbols (e.g., BYS, NUD,
VBZ, etc.) each followed by its value (a whole number from 1 to 25).
Participants were instructed to keep track of the values of the first
two stocks (i.e., the target stocks) presented during each interaction.
Each time the drivers heard one of the target stock symbols, they
determined whether the value of that stock had increased or decreased
since the last time they heard it mentioned, and pressed the
corresponding button on the steering wheel. At the end of the
interaction with the IVIS, the driver identified the overall trend
followed by each of the target stocks from four choices: hill, valley,
upward, and downward. Each IVIS drive included four interactions with
the stock ticker, whose lengths were 1, 2, 3, and 4 minutes. The order
of interactions was counterbalanced for the four IVIS drives using a
Latin square, and a one-minute interval separated consecutive
interactions.

Eye Movement Measures: Eye movement data were collected at 60 Hz using a
Seeing Machines' FaceLab^TM^ eye tracking system (version 4.1). The eye
tracking system uses two small video cameras, posited on the right and
left sides of the dashboard, to track head and eye movements. The system
calculates, among other measures, horizontal and vertical coordinates
for a gaze vector that intersects the simulator screen. The eye tracker
was calibrated at the beginning of the experiment and the calibration
was checked after each experimental drive. No participant wore glasses
or eye makeup during the experiment because the eye tracker has
difficulty tracking eyes accurately under these conditions. The tracking
error is within 5 degrees of visual angle. The system does not require
any head-mounted or chin-rest hardware, and is quite unobtrusive.

The gaze vector-screen intersection coordinates were transformed into a
sequence of fixations, saccades, and smooth pursuits. To identify these
three eye movements, two characteristics were used: dispersion and
velocity (see Table 5.1). Dispersion describes the span (in radians of
visual angle) that the gaze vector covers during a movement, and
velocity describes the speed (in radians of visual angle per second) and
the direction of the gaze vector during a movement.

The continuous data describing eye movement was transformed to identify
which of three eye states: fixations, saccades, or smooth pursuit
movements. This transformation was achieved by first calculating the
posterior probabilities of the three states by multiplying the
likelihoods of the values of states for various periods (see Figure
5.9). That is, for one period of data, the likelihood that the
particular state occurred could be calculated based on the typical value
for the three movements (see Table 5.1). With equal initial prior
probabilities, the posterior probabilities for the movements were
calculated by multiplying the prior probabilities by the likelihoods.
Figure 5.9 shows that the identification procedure calculated the
posterior probabilities based on the value of dispersion first, then
used these posterior probabilities as the prior probabilities, and
calculated final posterior probability based on the value of velocity.
The movement with the highest posterior probability that was also
greater than 0.3 was identified as the eye movement occurring in that
period. The range of characteristic values for each eye movement and the
cutoff probability, 0.3, were chosen according to previous research
(Jacob, 1995), and adjusted according to the particular characteristics
of our data (see the distributions in Figure 5.9). The final result is a
series of fixations, pursuit eye movements, and saccade eye movements.

![](media/image11.wmf)

[]{#__RefHeading___Toc173654919 .anchor}Figure 5.9. Illustration of the
algorithm used to identify eye movements.

[]{#__RefHeading___Toc173654935 .anchor}Table 5.1. The characteristics
of fixations, saccades, and smooth pursuits.

  ---------------- ------------------------ ------------------------------
  **Types**        **Dispersion**           **Velocity**

  Fixation         Small (≤ 1°)             Low, random direction

  Saccade          Large (\> 1°)            400-600 °/sec, straight

  Smooth pursuit   Target decided (\> 1°)   1-30 °/sec, target trajectory
  ---------------- ------------------------ ------------------------------

The identification process began with a segment of six frames. Based on
the characteristics in Table 5.1, the posterior probabilities of the eye
movements were calculated for the segment (see Figure 5.9). If the
highest probability was greater than a threshold, the segment was
identified as that eye movement. The segment was then increased by one
frame, and the process was repeated to check if the eye movement
continued in the new frame. If no movement could be identified, the
segment was decreased by one frame, and the posterior probabilities were
calculated again. If this process resulted in less than three frames in
a segment, the eye movement was identified using only the speed
characteristic. When speed was high, the movement was labeled as a
saccade, when low, it was labeled as a smooth pursuit. After each eye
movement was identified, the identification process began again with a
new six-frame segment.

Error: Reference source not found shows the ten categories of eye
movement measures included in the model development. The fixation and
pursuit duration represented the temporal characteristics of eye
movements, the horizontal and vertical position of fixations described
the spatial distribution of gaze, and their standard deviations (s.d.)
represented the variability of gaze. The pursuit distance, direction,
and speed captured the characteristics of smooth pursuit, and the
percentage of time spent in pursuit movements described the degree to
which drivers followed objects continuously rather than discretely
sampling the roadway. Saccade measures were excluded because saccade
distance was capture by fixation position or by the starting position of
pursuit movements. After identifying eye movements, the resulting data
were summarized across a window (specifics of this process will be
discussed in a later section).

[]{#__RefHeading___Toc173654936 .anchor}Table 5.2. The feature
combinations used as model input.

  ---------------------------------- ---------------- ----------- ------------
                                     **Feature                    
                                     combinations**               

  **Eye-movement and driving         **eye minus      **eye       **eye plus
  measures**                         spatial data**   data**      driving**

  fixation duration                  √                √           √

  mean of horizontal position of                      √           √
  fixation                                                        

  mean of vertical position of                        √           √
  fixation                                                        

  s.d. of fixation position          √                √           √

  pursuit duration                   √                √           √

  pursuit distance                   √                √           √

  pursuit direction                  √                √           √

  pursuit speed                      √                √           √

  percentage of pursuit in time      √                √           √

  mean of blinking frequency         √                √           √

  s.d. of steering wheel position                                 √

  mean of steering error                                          √

  s.d. of lane position                                           √
  ---------------------------------- ---------------- ----------- ------------

Driving performance measures: The driving measures consist of s.d. of
steering wheel position, mean of steering error, and s.d. of lane
position. The driving simulation directly outputted steering wheel
position and lane position at 60 Hz. Steering error was calculated at 5
Hz, and is the difference between the actual steering wheel position and
predicted steering wheel position. A second-order Taylor expansion is
used to calculated the predicted steering wheel position using the
earlier steering wheel position and velocity (Nakayama et al., 1999b).
The statistical measures were summarized across the same window as the
eye movement measures.

### Model characteristics and training

*Distraction Definitions*: Distraction definitions reflect the criteria
used to classify the driver as distracted or not distracted. Four
different definitions were tested (see Table 5.3). The first two were
based on experimental conditions, which described the assumed cognitive
states of drivers based on the tasks they were asked to perform. In the
experiment, participants drove 4 drives with the IVIS task that included
both IVIS and non-IVIS stages and 2 baseline drives without the IVIS
task. The DRIVE definition classified the IVIS drives as "distracted"
and the baseline drives as "not distracted." The STAGE definition
classified the stages with IVIS interaction as "distracted" and the
non-IVIS stages and baseline drives as "not distracted." Thus, the
difference between DRIVE and STAGE lies in the one-minute non-IVIS
intervals between IVIS interactions: DRIVE defined these as
"distracted," while STAGE defined these as "not distracted." Since some
cognitive states change gradually and continuously, participants may
have remained distracted during the non-IVIS intervals even though the
secondary task was not present. Thus, we speculated that DRIVE and STAGE
would capture distraction differently.

The other two definitions, STEER (steering error) and RT (acceleration
release reaction time), were based on driving performance. RT was
defined by the amount of time it took for the subject to release the gas
pedal after the lead vehicle started braking. This would be used to
attempt a prediction of driver response to breaking events using eye
movements and driving performance. STEER was used to explore the
association between eye movements and driving steering control. For each
participant, the data for STEER and RT were categorized. The upper
quartile of each data set was considered "distracted" and the remainder
"not-distracted."

The other two definitions, STEER and RT, were based on driving
performance---steering error and acceleration release reaction time.
Acceleration release reaction time was defined by the interval from lead
vehicle brake to subject releasing the gas pedal. For each participant,
we categorized the data that had steering error values or accelerator
release reaction times greater than their 75% upper quartile as
"distracted" and the remainder as "not-distracted." The purpose of using
STEER as a distraction definition was to explore the association between
eye movements and driving steering control. The purpose of RT, on the
other hand, was to assess whether eye movements and driving performance
could predict driver response to the braking events.

[]{#__RefHeading___Toc173654937 .anchor}Table 5.3. Model characteristics
and their values.

  ------------------- --------------- ---------------------------------------------
  **Model             **Values        
  characteristics**   ("distracted"   
                      vs. "not        
                      distracted" for 
                      distraction     
                      definition)**   

  **Distraction       DRIVE           IVIS drives vs. baseline drives
  definition**                        

                      STAGE           IVIS stage vs. non-IVIS stage and baseline

                      STEER           steering error \>75% upper quartile vs. ≤75%

                      RT              acceleration release reaction time \>75%
                                      upper quartile vs. ≤75%

  **Feature           Eye - spatial   
  combination**       info Eye Eye +  
                      driving         

  **Window size**     5, 10, 20, 40   
                      seconds         

  **Overlap**         1% 25% 50% 75%  
                      95%             
  ------------------- --------------- ---------------------------------------------

*Feature Combinations:* The three different combinations of input
variables consisting of eye movement and driving measures were
investigated (see Table 5.2.)The inputs were the ten eye movement
measures and three driving measures. feature combinations were designed
to assess the importance of specific variables for distraction
detection. First, we compared \"eye minus spatial data\" and \"eye
data.\" \"Eye minus spatial data\" excluded horizontal and vertical
fixation position from \"eye data\" because the spatial distribution of
fixations may be an important indicator of an eye-scanning pattern and a
particularly helpful means of detecting driver distraction. Second, the
comparison of \"eye data\" with \"eye plus driving\" evaluated how
valuable driving measures were in detecting distraction. Since the
driving measures included mean steering error, the \"eye plus driving\"
combination was not used to identify distraction defined by steering
error.

*Summarizing Parameters of Inputs:* Two parameters summarized the
inputs: window size and overlap between windows. These parameters were
used to summarize eye movement and driving performance measures to form
"instances," which were used as the inputs to the SVM models. Window
size denotes the period over which eye movement and driving data were
summarized. The comparisons of window size serve to identify the
appropriate length of data that can be summarized to reduce the noise of
the input data. We chose four window sizes: 5, 10, 20, and 40 seconds.

Overlap is the percentage of data in the current window also contained
in the previous window, and reflects the redundancy between one instance
of input data and another. For the DRIVE, STAGE, and STEER definitions,
five overlaps (1%, 25%, 50%, 75%, and 95%) were used for window sizes
5-20 seconds. Only three overlaps (50%, 75%, and 95%) were used for the
40-second window because there were not enough instances to train and
test the models when 1% and 5% overlaps were used. For the RT
definition, overlaps were not applied because the brake events were
discrete. In this case, the models used only the instance occurring
immediately before the brake events to predict the RT performance. In a
real-time application, window size and overlap interact to affect the
computational load on the detection system.

*Model Training*: First, the data were summarized across the windows to
form instances. For each participant, instances with the same window
size and overlap were merged, IVIS drives first followed by the baseline
drives, and normalized using a z-score. Then, the instance was labeled
as either "distracted" or "not distracted" according to the distraction
definitions. For each participant, different window-sizes and overlaps
produced a different number of instances that were used for training and
testing the corresponding models. For the distraction definitions that
considered continuous data (i.e., DRIVE, STAGE, and STEER), we randomly
picked 200 training instances (100 for each class, taking 1-30 percent
of total instances depending on the window-size and overlap) and used
the remaining instances for testing. That is, these models were all
trained with 200 instances and tested with various numbers of instances.
For the RT definition, which considered response to discrete braking
events, about 25% of the total instances, evenly divided between the two
classes, were used for training. We also used the same training and
testing datasets to build logistic regression models. Logistic
regression is a binary classifier that uses a generalized linear model
based on a logit function ($\text{log}\left( \frac{p}{1 - p} \right)$),
where *p* is the probability of a datum belonging to one class. The
logit function serves as the dependent variable for the linear
regression model. It is used extensively in the medical and social
sciences.

We chose the Radial Basis Function (RBF) as the kernel function for the
SVM,
$K\left( x_{i},x_{j} \right) = e^{- \gamma\left| x_{i} - x_{j} \right|^{2}}$,
where $x_{i}$ and $x_{j}$ represent two instances and$\gamma$is defined
before training. The radial basis function is a function whose value
depends only on the distance from a central point. The RBF is a commonly
used kernel function in SVM applications due to its robustness. It
allows for both non-linear and linear mapping by changing the values of
$\gamma$ and C (Hsu, Chang, & Lin, 2006). C governs the soft margin and
accommodates imperfect classification of all instances. Using the RBF
for SVM models requires only two parameters (C and $\gamma$) to be
defined before training, avoiding numerical difficulties and producing
more robust results than other kernels, such as polynomial (Hsu et al.,
2006). To obtain the appropriate parameter values, we searched for C and
$\gamma$ in a large range (2^-5^ to 2^5^) to find the best model
performance. We used the "LIBSVM" Matlab toolbox (Chang & Lin, 2006) to
train and test the SVM models. For the logistic regression models, we
used the Matlab Statistics Toolbox (i.e., the glmfit and glmval
m-functions).

Data from ten participants were used for the DRIVE, STAGE, and STEER
definitions, and data from nine participants were used for the RT
definition. Some data were excluded because the participants had failed
to follow instructions or the eye tracker had malfunctioned. For
example, one participant drove too slowly be coupled to the lead
vehicle, which produced too few brake events to train models for RT. For
most participants, we built 54 models (3 feature combinations x 18
window size-overlap combinations) for each of the DRIVE and STAGE
definitions, 36 models (2 feature combinations x 18 window size-overlap
combinations) for the STEER definition, and 12 models (3 feature
combinations x 4 window sizes) for the RT definition. In all, 1548 SVM
models and 1548 logistic models were constructed. These models were
tested using instances from the same participants because a preliminary
analysis had shown that models fit between participants resulted in
highly variable and generally poor prediction accuracy compared to those
within participants.

### Model performance measures 

Model performance was evaluated using three different measures. The
first was testing accuracy, which is the ratio of the number of
instances correctly identified by the model to the total number of
instances in the testing set. The other two measures are associated with
signal detection theory: sensitivity (d'), and response bias (β), which
were calculated according to (4).\
$\begin{array}{r}
d' = Ζ^{- 1}(\text{HIT}) - Ζ^{- 1}(\text{FA}) \\
\beta = e^{\left\{ \frac{\left| Ζ^{- 1}(\text{FA}) \right|^{2} - \left| Ζ^{- 1}(\text{HIT}) \right|^{2}}{2} \right\}}
\end{array}$ (4)\
where HIT is the hit rate and equal to (true positive /( true positive +
false negative)) and also called sensitivity by some researchers. FA is
false alarm rate defined as (false positive /( false positive + true
negative)) and is equivalent to 1-specificity. $Ζ^{- 1}$represents a
z-score transformation of the probabilities. d' represents the ability
of the model to detect driver distraction. The larger the value of d',
the more sensitive the model. β signifies the strategy used by the
model; models can be either liberal (β\<1) or conservative (β\>1). The
two measures separate the sensitivity and the bias of the model in
identifying distraction (Stanislaw & N. Todorov, 1999). These signal
detection measures provide a more precise indicator of model performance
than simply testing accuracy.

## Results

### Performance of the SVM models

Averaging across all four distraction definitions, the SVM models
detected driver distraction with an 81.1% accuracy (s.d. = 9.05%). The
mean sensitivity for all SVM models was 1.54 (s.d. = 1.04), and the mean
response bias was 3.91 (s.d. = 35.22). Both accuracy and sensitivity
significantly exceeded the chance performance described by 50% accuracy
and zero sensitivity (accuracy: t~9~=21.66, p\<0.0001; sensitivity:
t~9~=11.71, p\<0.0001). Accuracy and sensitivity had a moderately
positive relationship with a correlation coefficient of 0.68
(p\<0.0001). The dotted lines and bars for the SVM models in Figure 5.10
illustrate these average results.

![](media/image12.png){width="2.98125in"
height="1.7805555555555554in"}![](media/image13.png){width="3.00625in"
height="1.7763888888888888in"}

Testing accuracy Sensitivity

[]{#__RefHeading___Toc173654920 .anchor}Figure 5.10. Results of the SVM
and logistic models. The dashed, dash-dot, and dash-double dot lines
indicate chance performance, SVM average, and logistic average,
respectively.

### Comparison with the logistic method

After comparing the SVM method with chance performance, we compared it
with the logistic regression method. The paired t-tests showed that the
SVM models were more accurate (SVM: 81.1%, logistic: 72.7%, t(9)=11.74,
p\<0.0001) and more sensitive (SVM: 1.54, logistic: 1.37, t(9)=5.59,
p=0.0003) than the logistic models (see Figure 5.10). When we compared
response bias, we found that the SVM models have a generally
conservative strategy (β=3.91), whereas the logistic models were neutral
(β=1.05). However, no significant difference in response bias was found
(t(9)=1.63, p=0.137), reflecting a large variance in the bias of the
models.

Comparing the performance of the SVM and logistic models for each
distraction definition shows that the SVM models outperformed the
logistic linear models in testing accuracy for all definitions (DRIVE:
t(9)=4.27, p=0.0021; STAGE: t(9)=5.71, p=0.0003; STEER: t(9)=11.47,
p\<0.0001; RT: t(8)=2.43, p=0.041). The SVM models also had greater
sensitivity than the logistic models for DRIVE and STAGE (DRIVE:
t(9)=3.02, p=0.0144; STAGE: t(9)=8.84, p\<0.0001), and marginally better
sensitivity for STEER (t(9)=2.22, p=0.054), but not for RT (t(8)=-1.30,
p=0.23). The response bias was similar for both types of models for both
DRIVE and STAGE (DRIVE: t(9)=0.81, p=0.440; STAGE: t(9)=1.00, p=0.345;
RT: t(8)=1.71, p=0.12). The SVM models were marginally more conservative
than the logistic models (t(9)=2.24, p=0.052) for STEER.

None of the distraction definitions showed a significant difference for
bias between model types, which was consistent with the overall findings
in that the conservative SVMs were not significantly different from the
neutral logistic models. Variance of response bias was then examined.
Bias was significantly greater for the SVM models than for the logistic
models for the first three definitions (DRIVE: F(1,9)=102.20, p\<0.0001;
STAGE: F(1,9)=2862.28, p\<0.0001; STEER: F(1,9)=93201.8, p\<0.0001), but
not for RT (F(1,8)=0.34, p=0.92). The Receiver Operating Characteristic
(ROC) plots in Figure 5.11 clearly show the difference between the two
kinds of models for DRIVE, STAGE, and STEER. In both graphs in Figure
5.11, the dash-dot diagonal represents a neutral strategy (*β*=1).
Models using liberal strategies (*β*\<1) are located on the right side
of the neutral line, and models using conservative strategies (*β*\>1)
are located on the left side.

![](media/image14.jpeg){width="6.495833333333334in"
height="3.2895833333333333in"}

[]{#__RefHeading___Toc173654921 .anchor}Figure 5.11. ROC curves for the
SVM and logistic models for DRIVE, STAGE, and STEER.

The logistic models (the right graph) are located along the diagonal
while the SVM models (the left graph) are located across a much wider
area and show different characteristics for different definitions. The
logistic regression models use a consistent *β* , while the SVM use a
very inconsistent *β*. That is, misses and false alarms equally
contribute to the detection error of logistic models, while SVM models
vary the proportion of misses and false alarms according to the
proportion of "distracted" and "non-distracted" situations. The SVM
models using DRIVE and STAGE tended to use neutral or slightly
conservative strategies (almost equal miss and false alarm rates), and
STEER used substantially conservative strategies (higher miss rate than
false alarm rate). These comparisons show that the SVM models take a
more flexible approach to classification than the logistic regress
models.

### Effect of model characteristics

The results shown above are evidence that the SVM method is a powerful
technique for identifying driver distraction. Next, we studied the
effects of the distraction definitions, feature combinations, window
sizes, and overlaps on model performance in order to offer realistic
suggestions as to how to implement real-time distraction detection
systems using SVMs. We used the mixed linear model with subject as a
repeated measure, and performed post hoc comparisons using the
Tukey-Kramer method with SAS 9.0.

Distraction Definition: Accuracy and sensitivity (d') were significantly
affected by the different distraction definitions (accuracy:
F(3,26)=254.32, p\< 0.0001; sensitivity: F(3,26)=570.17, p\<0.0001). The
models for DRIVE had the highest accuracy and sensitivity, and the
models for STEER and RT had the lowest accuracy or sensitivity (see the
left graphs in Figure 5.12). The differences in model sensitivity (d')
are also captured in Figure 5.12. Most points for STAGE and STEER are
below the reference curve for sensitivity equal to 1.80, whereas most
points for DRIVE are above the reference curve. Despite rather large
differences in mean bias (DRIVE: 1.9; STAGE: 3.6; STEER: 8.33), the
definitions did not differ significantly due to the large variance in
bias discussed earlier.

Comparing the two definitions based on the experimental conditions, the
models for DRIVE were more accurate and more sensitive than the models
for STAGE (accuracy: t(26)=21.84, p\<0.0001; sensitivity: t(26)=17.05,
p\<0.0001). This suggests that distraction defined by IVIS and baseline
drives is more discernible than distraction defined by drivers'
interaction with IVIS system. One interpretation of this result is that
drivers' eye movements are affected by cognitive distraction even after
the task has ended, so that their eye movement patterns during the
one-minute non-IVIS intervals between IVIS interactions maintained the
same or similar patterns as during the interactions.

Comparing the driving-performance-based definitions, distraction defined
by steering error (STEER) was predicted more accurately than that
defined by RT (t(26)=5.99, p=0.0002). The results show eye movements
could reflect the change of steering performance caused by a secondary
cognitive task and predict the response time to the braking of the LV,
but not with a high degree of sensitivity.

Feature Combinations: Feature combinations had a significant effect on
testing accuracy (F(2,18)=25.84, p\<0.0001) and sensitivity
(F(2,18)=44.68, p\< 0.0001) but not on response bias (F(2,18)=2.48,
p=0.1117). Testing accuracy and sensitivity increased with the number of
input variables (see Figure 5.12). Specifically, the spatial information
of eye movements and driving measures both increase model sensitivity
(t(18)=2.97, p=0.0212, and t(18)=6.67, p\<0.0001, respectively). Adding
the driving measures to the eye data increased sensitivity more (0.41)
than did adding the spatial information to the other eye movements
(0.17).

![](media/image15.png){width="5.773611111111111in"
height="3.185416666666667in"}

[]{#__RefHeading___Toc173654922 .anchor}Figure 5.12. SVM testing
accuracy and sensitivity for the feature combinations. The braces
represent the post hoc comparisons between the successive combinations
using Tukey-Kramer method. \*\* indicates p\<0.05.

Because the driving measures improved model performance so dramatically,
we built additional models using only driving measures as inputs.
"Driving alone" achieved accuracy of only 54.4%, sensitivity of 0.89,
and bias of 1.87 (see Figure 5.12). The large differences in accuracy
and sensitivity from "eye plus driving" and "eye data" to "driving
alone" suggest that eye movement features played a more important role
in detection than the driving measures.

Summarizing Parameters of the Input Data: Window size affected testing
accuracy (F(3,27)=33.35, p\< 0.0001) and sensitivity (F(3,27)=44.68, p\<
0.0001) but not response bias (F(3,27)=2.48, p=0.1117). The models'
accuracy and sensitivity increased with window size. Similarly, overlap
increased testing accuracy (F(4,36)=19.01, p\< 0.0001) and sensitivity
(F(4,36)=72.47, p\< 0.0001) but not on response bias (F(4,36)=0.79,
p=0.5421). Increasing the redundancy of input data between adjacent
windows improves model performance. More importantly, window size and
overlap interacted to affect testing accuracy (F(17,153)=35.36, p\<
0.0002) and sensitivity (F(17,153)=51.01, p\< 0.0001) as Figure 5.12
clearly shows, but not response bias (F(17,153)=1.30, p= 0.1966).

![](media/image2.wmf){width="2.9180555555555556in"
height="2.3569444444444443in"}![](media/image3.wmf){width="2.9097222222222223in"
height="2.3534722222222224in"}\
Testing accuracy Sensitivity

[]{#__RefHeading___Toc173654923 .anchor}Figure 5.13. Testing accuracy
and sensitivity for different summarizing parameters of input data.

We then studied the effects of summarizing parameters for the
distraction definition with the best performance, DRIVE. Window size and
overlap show the same trend as in Figure 5.13. The best model, using
40-second window size with 95% overlap, produced 96.08% accuracy,
sensitivity of 3.84, and response bias of 4.25.

## Discussion

Support Vector Machines, a data mining method, offers a promising
approach to detecting driver distraction from eye movements and driving
performance in real time. The results show that the SVM models clearly
outperformed the logistic regression models. The comparisons for the
model characteristics show that, on average, the DRIVE definition, "eye
plus driving" feature combination, and 40-second window size with 95%
overlap, led to the best model performance.

The results indicate that the cognitive task affected the pattern of eye
movements. Figure 5.14 shows these differences for an IVIS and a
baseline drive. When drivers interacted with the IVIS, they had fewer
and shorter fixations on the right side of the road where the bicyclists
appeared. These data suggest that when drivers engaged in the secondary
task, they paid less attention to the bicyclist detection task.
Diminished performance in detecting objects and a decrease in gaze
variability has also been reported in two previous studies (Recarte &
Nunes, 2000, 2003b). This consistency shows that the differences found
in Figure 5.14 are part of what the SVM models captured in
distinguishing distracted from non-distracted drivers.

![](media/image16.png){width="2.94375in"
height="2.2125in"}![](media/image17.png){width="2.9479166666666665in"
height="2.2083333333333335in"}

IVIS drive baseline drive

[]{#__RefHeading___Toc173654924 .anchor}Figure 5.14. The plots of
fixation distribution over the background of the driving scenario for
IVIS (left) and baseline (right) conditions for participant SF7. The
size of each dot represents fixation duration.

One limitation of this experiment is that the bicyclist detection task
likely caused drivers to scan the driving environment in a manner
differently than they would in normal driving. The discrepancy between
the experimental data and reality suggests a need for caution in
generalizing the results. If drivers considered the pedestrian-detection
task something they could shed as the IVIS demands increased, the
corresponding shift in eye movements might make distraction easy to
detect, but such a shift might not generalize to actual driving
situations. Because this experiment used a pedestrian-detection task, it
may have overestimated algorithm performance in that the relatively
explicit need to scan the right side of the road does not exist in many
driving situations. However, monitoring for pedestrians and bicyclists
is a realistic driving task and one that might be neglected in actual
driving situations. As a consequence, the driving scenario in this
experiment reasonably simulated normal driving. Future research needs to
assess the degree to which the driving situation influences algorithm
performance.

In addition to the binary states of drivers, the SVM models also
generated a decision variable for each testing instance. When the
decision variable was positive, SVM models outputted a binary state 1;
when negative, the models outputted 0. This decision variable can
indicate the distance from the instance to the classification boundary
and can be interpreted as the model's confidence in the binary output.
Figure 5.15 shows how the decision variable changed for the "eye plus
driving" feature combination, and the 40-second window with 75% overlap.
The decision variable remains positive in the IVIS drive and drops below
zero in the baseline drive. This figure shows that the decision
variables of the SVM models follow the expected trend corresponding to
the IVIS task. Also evident in Figure 5.15 is a delay that may be
attributed to either the inertia of driver attention, the aggregation of
the data over a large window, or some combination of the two. The degree
to which this delay reflects characteristics of the algorithm or
reflects the driver's state merits further investigation. Clearly, delay
is an issue that must be addressed before SVMs are implemented in a
real-time system, which will be discussed later in this section.

![](media/image18.png){width="6.489583333333333in"
height="2.2069444444444444in"}

[]{#__RefHeading___Toc173654925 .anchor}Figure 5.15. SVM decision
variable along the timeline of an IVIS drive and a baseline drive for
participant SF7 (for the same data shown in Figure 5.14).

Several factors might explain the superior performance of the SVM
algorithm relative to that of the logistic regression. First, the SVM
models used the RBF kernel function that can fit both linear and
nonlinear relationships, whereas the logistic models can only fit linear
models. Second, the training of SVMs minimizes the upper bound of the
generalization error (Amari & Wu, 1999), whereas the logistic method
only minimizes training error. This makes the SVM more robust by
rendering overfitting less likely than with logistic regression. Third,
the SVM method can adjust response bias to increase performance by
changing parameter values (i.e., C and γ). The logistic method looks
only to minimize errors (i.e., false alarms and misses), a strategy that
largely depends on the ratio of instances from the two classes in the
training dataset. In this study, because the training sets were randomly
selected and consisted of equal numbers of instances for the two classes
("distracted" and "not distracted"), minimizing the total number of
errors led to approximately equal occurrences of false alarms and
misses, which in turn resulted in neutral strategies (see right graph in
Figure 5.11). Because the proportion of time a driver is "distracted" or
"not-distracted" on the road is unknown and will vary, it is impossible
to select training data that are representatively proportioned between
the two classes and between false alarms and misses. With the same
training sets, the SVM models produced different strategies when
changing the values of C and γ (see left graph in Figure 5.11).

The spatial and temporal patterns of eye movements have complex
connections with cognition, and only indirectly reflect driver
distraction. These connections have already been demonstrated with
measures aggregated over experimental condition (May, Kennedy et al.,
1990; Rantanen & Goldberg, 1999; Recarte & Nunes, 2000, 2003b). Our
study further supports eye movements as a real-time indicator of driver
distraction. Cognitive distraction can degrade driving performance, and
including measures of driving performance boosted SVM accuracy and
sensitivity compared to using eye data alone. However, using only driver
performance resulted in insensitive models with low accuracy. Similar to
our results, others have found that gaze-related features led to much
better prediction accuracy compared to driving performance measures
alone (Hayhoe, 2004).

There are some additional, practical limitations to implementing
distraction detection systems. The first is how to obtain consistent and
reliable sensor data. Eye trackers may lose tracking accuracy when
vehicles are traveling on rough roads or when lighting conditions are
variable. More robust eye tracking techniques are needed to make these
detection systems a reality. Steering data can be obtained directly from
the angle of the steering wheel, however, and some have developed robust
measures of lane position in real driving environments \[14\]. Second,
delay of detection needs to be assessed to evaluate whether it is
appropriate for the application. The delay in real-world systems can
come from three sources. One is sensor delay. For example, the eye
tracker used in this research took approximately 2.6 s to translate the
camera image to numerical data. The second source is the data-reduction
and computational time of SVM models. It took about one second to reduce
data and compute 15-s-long data in this study. These two kinds of delays
can be reduced with the advance of sensor and computer technology and
the improvement of the data reduction algorithms. The third source
derives from summarizing data across windows. Larger windows cause a
longer delay. The consequence of these lags will depend on the
particular distraction mitigation strategy they support. Developing a
systematic approach to balance the cost of time lags with the precision
of distraction estimates for particular mitigation strategies represents
an important research issue.

## Conclusion

SVMs provide a viable means of detecting cognitive distraction in
real-time and outperformed the more traditional approach of logistic
regression. Comparing the performance of SVM using eye movements to a
case without eye movements clearly demonstrates the importance of eye
movements as an input to a distraction detection algorithm. Including
both eye and driving measures as inputs to a distraction detection
algorithm is recommended. These data were collected in a driving
simulator with relatively homogenous traffic and roadside objects.
On-road data in a more diverse set of conditions is needed to assess the
generality of the results.

# Bayesian Networks to Detect Cognitive Distraction

Although Support Vector Machines are a promising approach to detecting
driver distraction, other data mining approaches may provide a
complementary---or even potentially more effective---means of detecting
distraction. One such approach is a Bayesian Network. A Bayesian Network
represents conditional relationships between a network of random
variables. In Figure 5.16, the nodes present random variables which are
identified by concepts or measures in the area of interest. The nodes
can present hypotheses, observation evidence, or hidden states.
Hypothesis nodes, like H, H^t-1^, and H^t^ in Figure 5.16 represent
decision variables and the beliefs (expressed as probabilities) of each
possible value. For the models used in this study, the binary hypothesis
node was driver distraction (distracted or not distracted). Evidence
nodes, like E1, E2, E1^t-1^, and E1^t^, represent observed measures used
to infer the state of decision variables. Summarized eye movement and
driving measures, such as mean fixation duration or mean lane position,
are the evidence nodes for this application. Hidden nodes, like S,
S^t-1^, and S^t^, are abstract concepts that cannot be directly
measured, but may be important for belief inference, because they can
group related evidence. Therefore, hidden nodes can help make models
more meaningful and form a hierarchical network structure. Possible
intermediate abstract concepts represented by hidden nodes could be
overall eye scanning pattern or an overall level of driving performance.
These classes of nodes are exclusive to each other. For example, Hidden
Markov Models (HMMs), a simple BN, have hidden hypothesis nodes.

The arrows represent conditional relationships between the nodes in a
BN. For example, in Figure 5.16, the arrow from S to E2 on the left
graph means that by giving the value of S, we can know the probability
of E2 being a certain value. By knowing S, the value of E2 is
independent on any other node.

  ----------------------------------- -----------------------------------
  ![](media/image19.wmf)              ![](media/image20.wmf)

  **Static Bayesian Network**         **Dynamic Bayesian Network**
  ----------------------------------- -----------------------------------

[]{#__RefHeading___Toc173654926 .anchor}Figure 5.16. Examples of a SBN
and a DBN, where H is a hypothesis node, S is a hidden node, Es are
observation nodes, and t represents time.

BNs are either Static (SBNs) or Dynamic (DBNs). SBNs, shown on the left
in Figure 5.16, consider evidence and beliefs at a single point in time
and cannot describe probabilistic relationships across time. DBNs, shown
on the right in Figure 5.16, can model time-series events as a Markov
process. DBNs connect two SBNs that are identical in structure from two
successive time steps. That is, the state of a node at time t is
conditionally dependent on all nodes in both the previous (t-1) and
current (t) time steps.

Many studies have used BNs to describe human behavior (Guhe et al.,
2005b; Ji, Zhu, & Lan, 2004a; Kumagai & Akamatsu, 2006a; Li & Ji,
2005a). BNs have several advantages that make them well suited for this
application. First, human behavior is usually complicated, dynamic, and
stochastic. BNs are capable of representing these complex relationships.
The hierarchical structure of BNs can systematically present information
from different sources and at different levels of abstraction (Li & Ji,
2005a), and can also capture probabilistic relationships. BNs have a
much more diverse and flexible representation than some formalized
mathematical approaches. Second, a BN is not only a computational model
but also a form of knowledge representation. The training of BNs can
identify the meaningful relationships that underlie model predictions.
Studying mutual information between evidence and hypothesis nodes
reveals the most influential evidence underlying model predictions (Guhe
et al., 2005a). Unlike other data mining approaches, such as support
vector machines, BNs reveal the relationships that generate the model
predictions. Third, BNs can handle situations with missing data (Li &
Ji, 2005a). The certainty of the hypothesis will change according to
BNs' reasoning, which incorporates new data using a probabilistic
dependence network when new evidence is added. Because of these
advantages, BNs are applicable to human-behavior modeling and have been
used to detect affective state (Li & Ji, 2005a), fatigue (Ji et al.,
2004a), lane change intent during driving (Kumagai & Akamatsu, 2006a),
and pilot workload (Guhe et al., 2005a). Despite these advantages,
creating a correct and stable BN model requires extensive computational
capability and a large amount of training data.

Although BNs have been used in many studies of human behavior, no
research has applied this technique to real-time detection of driver
distraction. This study presents a BN approach to identifying driver
cognitive distraction non-intrusively and in real-time, using eye
movement and driving performance data from a driving simulator
experiment. The objectives of this study were to learn what features of
BN models most influence the detection of driver distraction and to
explore the new insight into cognitive distraction that BN models
produce.

## Model Construction

The same data used to train and test the SVMs were used to train and
test the BNs.

### Training of BN models 

Following the results from a previous study (Liang, Reyes, & Lee, 2007),
IVIS drives and baseline drives were used to define driver cognitive
state as distracted and not distracted, respectively. The BN models used
this information to define the hypothesis node. The evidence used by the
BN models, hereafter called instances, were discretized values of eye
movement and driver behavior data summarized over a windows of 5, 10,
15, or 30 seconds. The raw eye data were transformed into fixation,
saccade, and smooth pursuit eye movements. From this basic eye movement
behavior, the following summary eye movement measures were calculated:
mean and standard deviation (SD) of fixation duration, horizontal and
vertical fixation location coordinates, pursuit duration, pursuit
distance, pursuit direction and pursuit speed, percentage of the time
spend on performing pursuit movements, and mean blink frequency.
Summarized driving performance data included standard deviation of
steering wheel position, mean steering error (Nakayama, Futami,
Nakamura, & Boer, 1999a), and standard deviation of lane position.
Finally, all 19 of the summarized measures were discretized, using
cutoff points for each measure that maximized the information gain for
the state of cognitive distraction. The splits for the measures used the
cutoff points that were the most representative of the two cognitive
states (Tan, 2005).

Several BN models were trained and tested for each of nine participants.
One participant's data was omitted because he failed to perform the
driving tasks as instructed. Following the typical data mining practice
(Tan, 2005), two-thirds of the instances (equivalent to approximately
one hour of driving) were chosen at random to form the training set, and
the remainder (about 30 minutes of driving) formed the testing set. The
training of BN models included structure learning and parameter
estimation. Structure learning identifies the possible connections
between nodes within a BN, and parameter estimation identifies the
conditional probabilities for these connections. These two kinds of
learning were processed in one learning run.

The BN models were trained using a Matlab toolbox (Murphy) and an
accompanying structure learning package (LeRay, 2005). The hypothesis
node represented driver state of cognitive distraction, which was
defined by the experimental conditions (IVIS drives -- distracted;
baseline drives -- not distracted). The evidence nodes included the 19
discrete measures of eye movements and driving performance. The hidden
node, which is not a hypothesis node in this study, represents an
abstraction and aggregation of the evidence, such as overall eye
scanning pattern or a general level of driving performance.

Model performance was compared for three factors (see Table 5.4): BN
model type, presence of a hidden node, and summarizing parameters for
training instances. The first factor, model type, compared SBN and DBN
models to assess how time dependencies affect model performance. Driving
involves a series of actions taken by drivers to control vehicles, and
these actions are sequentially related because changes in driver
performance and cognitive state are continuous and gradual. Taking
time-dependent relationships into account may help capture the overall
pattern of driver performance, and so the DBN models may detect driver
distraction more precisely than the SBN models.

The second factor considered whether including a hidden node improved
model performance compared to a model without a hidden node. Because
hidden nodes can group evidence in a meaningful way, they may be less
sensitive to noisy data for inferring hypothesis nodes. Hidden nodes can
also simplify inference by requiring less evidence at each step of the
inference. Luo and colleagues (Luo, Wu, & Hwang) proposed a hierarchical
DBN with five hidden nodes to successfully identify human motion from a
video sequence. In this study, some possible intermediate concepts, such
as eye movement pattern or driving performance, may aggregate observed
measures and provide a more stable indicator of driver cognitive
distraction than the observed measures themselves. Although hidden nodes
may improve model performance and provide meaningful interpretation,
they also increase computational complexity and uncertainty in training.

The last factor was the summarization of the parameters used to create
the instances for training and testing. Specifically, the various window
sizes over which data were summarized were compared. The aim was to find
the summarization of data that balances noise reduction and the washout
of the distraction signal associated with a longer window. For DBNs,
window size also defined the time steps. That is, evidence at time step
t was the summarized measures across a window ending at time t; evidence
at time step t+1 was summarized across the next window, which started at
t and ended at t + window size. Liang et al. (Liang et al.) found that
longer windows improved model performance. Since the total quantity of
training data was fixed (approximately one hour of driving), the number
of training instances decreased with increasing window size. Thus, the
performance of models with longer window sizes might deteriorate because
they were trained with fewer instances. More importantly, large windows
could impose a delay in detecting distraction. A second summarization
parameter, sequence length, was considered for DBNs. A sequence
contained multiple, consecutive training instances. It was thought that
longer sequences might provide more comprehensive and stable
time-dependent relationships between driver distraction and performance,
and could be expected to train better-performing models. The sequence
lengths were 30, 60, and 120 seconds.

[]{#__RefHeading___Toc173654938 .anchor}Table 5.4. The Comparisons with
Different Characteristics.

  ----------------------- --------------- ---------------- ---------------
  FACTORS                 LEVELS                           

  BN Types                SBNs            DBNs             

  Number of Hidden Node   0, 1            0, 1             

  Summarization           Window Size     Sequence Length  Window Size
  Parameters                                               

                          5, 10,15, 30 s  30 s             5, 15 s

                                          60 s             5, 10, 15 s

                                          120 s            5, 10, 15, 30 s
  ----------------------- --------------- ---------------- ---------------

The structures of the BNs were constrained so that the training
procedure was computationally feasible and the trained models could be
reasonably explained. When SBNs had a hidden node, the hypothesis node
(distracted/not distracted) was connected to the hidden node, and the
hidden node to the evidence nodes. This meant that there were
conditional relationships between the hypothesis and hidden nodes, and
between the hidden and evidence nodes, like the arrows between nodes in
Figure 5.16. There were no direct connections between the hypothesis and
evidence nodes. When SBNs had no hidden nodes, the hypothesis node was
connected with the evidence nodes directly. For DBN models (see Figure
5.17), the links within a time step, called intra-links, were present
only in the first time step. After the first step, nodes were linked
only with those in the previous time step. These links between time
steps were called inter-links. The inter-links connected the hypothesis
node at a previous time step to the hidden node at the present time
step; the hidden node at a previous time step was connected to the
evidence nodes at the present time step when the DBN models had a hidden
node (see Figure 5.17). When DBN models had no hidden node, the
hypothesis node at the previous time step was connected to the evidence
nodes at the present time step. For DBNs, two structures, an
intra-structure and an inter-structure, were trained, while only one
structure was trained for the SBNs.

![](media/image21.wmf)

[]{#__RefHeading___Toc173654927 .anchor}Figure 5.17. Constrained DBN
Structure, where solid arrows represent intra-links and dotted arrows
represent inter-links.

In total, 234 BN models were trained. Each participant had 26 BN models:
8 SBN models (2 levels of hidden nodes crossed with 4 levels of window
size) and 18 DBN models (2 levels of hidden nodes crossed with 9
combinations of summarization parameters). The trained models included
estimates of the beliefs of distracted/not distracted situations for the
testing cases. The resulting networks were assessed with experimental
conditions (IVIS drives/baseline drives) to evaluate their ability to
identify distraction.

### Model performance measures

Model performance was evaluated with the same measures used for SVM in
the previous section.

### Mutual information

Normalized mutual information was also used to assess the strength of
dependencies between performance measures and cognitive distraction.
Mutual information, I(X;Y), describes the information shared by two
random variables, X and Y (Guhe et al., 2005a). That is, it measures how
much uncertainty of Y is reduced by knowing X. The higher the level of
mutual information, the closer the connection between the two variables.
Based on the trained model, the normalized mutual information of each
driver behavior measure and cognitive distraction can be calculated
according to (3)\
![](media/image22.png){width="2.0395833333333333in"
height="0.42916666666666664in"}...................................................\..............(3)\
where X represents a performance measure, Y represents cognitive
distraction, H(Y) is the entropy of cognitive distraction, and H(Y\|X)
is the entropy of cognitive distraction given the particular performance
measure. The entropy measures the uncertainty of a variable. This
normalized mutual information describes the degree to which uncertainty
in identifying distraction is reduced by knowing the measure. The higher
this value, the more indicative the measure is of driver distraction.

To compare the mutual information of performance measures, we grouped
the model inputs according to their correlation, because correlated
measures may have a similar association with the level of driver
cognitive distraction. A factor analysis with varimax rotation, which
transforms the factors so that they are orthogonal to each other, was
conducted on 19 discrete performance measures. The results showed that
52.3% of the variation in the 19 measures could be accounted for by six
common factors. These six factors included fixation duration (consisting
of mean and SD of fixation duration and percentage of smooth pursuit),
fixation horizontal distribution (mean and SD of fixation horizontal
coordinates), fixation vertical distribution (mean and SD of fixation
vertical coordinates), pursuit duration (mean and SD of pursuit duration
and distance, and percentage of smooth pursuit), pursuit speed (mean and
SD of pursuit speed), and driving performance (SD of steering wheel
position, mean of steering error, and SD of lane position). An
additional category containing only blink frequency was added because
the six factors explained little of the variance of blink frequency.
Average normalized mutual information was calculated and compared for
the groups of fixation duration, fixation horizontal distribution,
fixation vertical distribution, pursuit duration, pursuit speed, driving
performance, and blink frequency.

## Results

The analyses were conducted in two parts. The first part evaluated the
overall BN model performance, and then examined the effects of the three
model characteristics by comparing testing accuracy and two SDT
measures. T-tests and the mixed linear model with subject as a repeated
measure were used for these analyses. Post-hoc comparisons were
performed using the Tukey-Kramer adjustment for multiple comparisons.
The second part of the analyses examined the relationships between model
structure and performance and mutual information. The mutual information
of six common factors was also compared intuitively to identify the
strength of the relationships between performance measures and driver
cognitive distraction.

### Comparison of model performance

Mean accuracy of BN models was 80.1% (SD = 10.0%); sensitivity, 2.22 (SD
= 1.17); and response bias, -0.52 (SD = 2.95). Accuracy and sensitivity
were both far better than the chance performance described by 50%
accuracy and zero sensitivity (accuracy: t(8) = 9.05, p \< 0.0001;
sensitivity: t(8) = 5.71, p = 0.0004). Overall, the BN models favored
"distraction," leading to a relative high false alarm rate (0.37) and
high hit rate (0.87). That is, the models correctly detected distraction
87% of the time when distraction was present, but also incorrectly
detected distraction 37% of the time when distraction was not present.

For the first factor, model type, which compared SBNs and DBNs, we found
that, on average, DBN models were more sensitive than SBN models (F(1,8)
= 21.2, p = 0.0017), but there were no significant differences in
accuracy (see Figure 5.18) and response bias. This result shows that
considering time-dependent relationships increases the ability to detect
distraction. The presence of a hidden node had a significant influence
on all three performance measures (accuracy: F(1,8) = 30.33, p = 0.0006;
sensitivity: F(1,8) = 9.86, p = 0.014; response bias: F(1,8) = 15.02, p
= 0.0047). The models without a hidden node recognized distracted
situations more accurately, and with greater sensitivity, and tended to
err in identifying distraction when it was not present, compared to
those with one hidden node.

There was a marginally significant interaction between model type and
the presence of a hidden node for accuracy (F(1,8) = 5.12, p = 0.052)
but not for sensitivity or response bias (sensitivity: F(1,8) = 3.01, p
= 0.12; response bias: F(1,8) = 0.58, p = 0.47). As seen in Figure 5.18,
the decrease in accuracy when a hidden node was present was greater for
the DBNs than for the SBNs. When the models had no hidden node, the
accuracy of DBNs was significantly greater than that of SBNs (t(8) =
2.4, p = 0.043), but DBN models did not show a significant advantage in
accuracy compared to SBN models (t(8) = -0.83, p = 0.43) when there was
a hidden node.

![](media/image4.jpeg){width="5.992361111111111in"
height="3.298611111111111in"}

[]{#__RefHeading___Toc173654928 .anchor}Figure 5.18. The Comparisons of
Model Type, Number of Hidden Nodes and the Interaction.

The summarization parameter, window size, had no significant effect on
the performance of the SBNs, as measured by accuracy, sensitivity, and
response bias. Similar results were obtained for DBNs, with no
significant effects of window size for the three model performance
measures. Longer training sequences produced more sensitive models
(F(2,16) = 12.26, p = 0.0006), but accuracy and bias were not affected.
No interaction between window size and sequence length was found. The
most accurate predictions of distraction occurred with DBNs and a
120-second training sequence.

### Analysis of mutual information

In addition to the analyses on model performance, mutual information
between evidence nodes (performance measures) and hypothesis node
(driver distraction) was calculated and compared for the SBN and DBN
models without a hidden node. Mutual information showed a correlation
with model performance. The total mutual information carried in a model
had a moderately positive relationship with its accuracy (r = 0.48,
p\<0.0001) and sensitivity (r = 0.51, p\<0.0001). As window size
increased, the number of links in a model decreased (SBNs: F(3,24) =
135.0, p\<0.0001; DBNs: F(3,24) = 470.3, p\<0.0001) and the average
mutual information per link, as calculated by total mutual information
in a model divided by the number of links, increased (SBNs: F(3,24) =
20.7, p\<0.0001; DBNs: F(3,24) = 66.8, p\<0.0001). These results suggest
that the larger windows more clearly identify critical performance
indicators of driver distraction.

Figure 5.19 shows the averaged normalized mutual information of all
seven categories of measures for the trained SBN, and DBN intra- and
inter-structures. Blink frequency played the most important role in
identifying driver cognitive distraction, as shown by the high level of
normalized mutual information. When mutual information was averaged
across SBN, DBN intra-, and DBN inter-structures, knowing blink
frequency reduced 37% of uncertainty in the detection. Fixation
measures, especially fixation duration and vertical distribution, also
helped to distinguish driver states because the average normalized
mutual information of fixation duration and vertical distribution were
10% and 12%, respectively. However, pursuit eye movement and driving
measures had only a weak connection with cognitive distraction, as
reflected in their normalized mutual information of less than 2%. The
SBN, and DBN intra- and inter-structures had similar patterns of mutual
information distribution across these seven categories.

![](media/image23.png){width="6.498611111111111in"
height="3.0527777777777776in"}

[]{#__RefHeading___Toc173654929 .anchor}Figure 5.19. The Mutual
Information of Seven Categories of Performance Measures.

Mutual information also revealed individual differences. Table 5.5 shows
the mutual information distributions for the four measures with the
highest mutual information values. Although the average mutual
information for blink frequency is 37% across all participants, this
value is less than 10% for three participants. The other groups of
performance measures also show inconsistent mutual information across
participants, suggesting the need to estimate separate BN structures and
parameters for each individual.

[]{#__RefHeading___Toc173654939 .anchor}Table 5.5. The
Mutual-Information Distribution of Nine Participants across the Most
Predictive Variables (dark shading indicates average normalized mutual
information greater than 30%; dark grey indicates between 30% and 10%;
grey indicates between10% and 5%; white indicates less than 5%).

  -------------------------------------- --- --- --- --- --- --- --- --- ---
  Participant                            1   2   3   4   5   6   7   8   9

  Blink frequency                                                        

  Fixation duration                                                      

  Fixation horizontal distribution                                       

  Fixation vertical distribution                                         
  -------------------------------------- --- --- --- --- --- --- --- --- ---

## Discussion

The performance of BN models confirmed that they are a useful and
promising technique for identifying driver cognitive distraction.
Drivers behaved differently when they were cognitively loaded and BNs
were able to detect these differences. Eye movement and driving
performance measures, especially those having high mutual information
with distraction, were useful in predicting drivers' cognitive state.
The best predictors of distraction were DBNs using 120-second training
sequences. These models produced an accuracy of 86.4%, with a
sensitivity of 3.90. DBNs produced more sensitive models, suggesting
that time-dependent data contains information that indicates
distraction.

The presence of a hidden node, particularly for the DBN, diminished
model performance. This was surprising in that a hidden node represents
a free parameter that should enable the model to fit the data more
precisely. One interpretation of this result is that including an
intermediate state in the model does not accurately describe the
mechanisms underlying cognitive distraction. The model structure without
a hidden node is a more accurate representation of the relationship
between driver performance and cognitive state. Another possibility is
that, because hidden nodes introduce more uncertainty in the model
learning process, using equal amounts of training data for the models
both with and without hidden nodes may have caused the hidden node
models to be less accurate. This second possibility would also explain
why accuracy decreased from no hidden node to one hidden node more for
the DBNs than for the SBNs; the learning process was more complex for
DBNs. To determine which explanation is correct, another study with is
needed to ensure that training can produce accurate parameter estimates.

Window size did not affect model performance for either the SBNs or
DBNs, which means that evidence summarized across different periods of
time did not present the distraction signal differently, and that the
definition of time step did not affect model performance of DBNs. These
results conflict with those from Liang et al. (Liang et al.), where
larger windows improved the ability of SVM to detect cognitive
distraction. The conflicting results may reflect important differences
between the two methods. As with the hidden node comparison, the models
that use longer window sizes were trained with fewer instances because
the length of total data was fixed, meaning that large windows produced
fewer training instances. The effect of sequential length of training
examples conforms to this explanation, with sequential length of
training examples improving DBNs model performance. Longer sequences
supply more repeated cases from which DBNs can extract the stable
relationship between performance and cognitive distraction.

The analysis of mutual information showed that blink frequency had the
strongest relationship with cognitive distraction. Another study used
blink measures to study driver vigilance (Bergasa et al., 2006), and
found that when drivers were alert, they blinked less frequently. We
found a similar pattern of behavior; during baseline drives, drivers
blinked at 0.31Hz, but when they interacted with the secondary task,
they blinked at 0.49 Hz (F(2,18) = 361.4, p\<0.0001). Such an increase
in involuntary eye movements may disrupt the consolidation of visual
information (Strayer et al., 2003b). The time-dependent changes in
fixation horizontal distribution, pursuit speed, and driving measures
are all more predictive of distraction than when such changes are
considered as static measures, because their mutual information in DBN
inter-structure (5%, 3%, and 2%, respectively) was higher than that in
SBN structure (2%, 1%, and 0.5%, respectively). The mutual information
analysis also shows that individual differences render unrealistic a
uniform model structure that fits all people.

Interestingly, larger window sizes decreased the number of links and
increased the mutual information per link in the trained models. This
suggests that larger window sizes make important dependent relationships
between performance measures and distraction more prominent, but do not
necessarily improve model performance. One possible reason is that
training instances decrease as window size increases.

Although this BN approach aims at real-time detection, there remains an
unavoidable lag from when the driver's cognitive state changes until the
model recognizes the change. As with the SVM algorithm, the lag results
largely from three sources: sensor delay, model computational delay, and
the delay caused by summarizing the measures across a window. Sensor
delay is the time required for data acquisition and reduction, and model
computational delay is the computational time of BN models. The
magnitudes of the first two kinds of delay are on the order of seconds.
The third source reflects the need to summarize the measures across a
window, which is expected to be approximately half of window size. That
is, a 5-second window produces about a 2.5-second delay, and 30-second
windows produce about 15-second delays. As a consequence, increasing
window size also increases model lag. The cost of the lag needs to be
balanced against the prediction accuracy needed to support the specific
distribution mitigation strategy. For the BNs in this study, the greater
number of training instances seemed to balance the less precise state
estimates associated with small time windows. Further analysis of larger
datasets with more training instances are needed to assess this
possibility and the tradeoff between prediction accuracy and prediction
lag.

Although using BN to identify driver distraction represents a promising
approach, implementation in production vehicles will depend on the
development of sensors. For example, economical eye tracking devices
that are robust to road surface, lighting conditions, eye color, and eye
glasses are only now becoming a reality. The outcomes of this study
suggest that as eye tracking and other driver-state sensors become more
available, real-time measurement of driver distraction may be feasible.
BN models that identify driver distraction provide critical input to
distraction mitigation strategies that might include warning a
distracted driver to attend to the road, diminishing the threshold of
collision warnings, or recording instances of distraction for later
review (Donmez et al., 2003a).

## Conclusion

BNs provide a viable means of detecting cognitive distraction in
real-time. Compared with SBNs, DBNs produce more sensitive detection
models, suggesting that considering time-dependent relationships is
useful in estimating the cognitive state of drivers. To train DBN
models, longer training sequences are needed in order to produce
sensitive models. Among the performance measures used in this study,
blink frequency, fixation duration and fixation horizontal and vertical
distribution played more important roles in identifying distraction than
did smooth pursuit and driving performance measures. These data were
limited, as they were collected in a driving simulator with relatively
homogenous traffic and roadway scenarios. On-road data in a more diverse
set of conditions are needed to assess the generality of the results.

# Combining Algorithms to Detect Distraction

Although the previous sections considered SVMs and BNs as separate
alternatives for detecting distraction, it may be useful to consider how
they complement each other and how they might be combined to detect
distraction. Detecting cognitive distraction is a complex procedure,
which requires a robust data fusion system. Unlike visual and manual
distraction, the challenge of detecting cognitive distraction is to
integrate multiple data streams, including eye movements and driving
performance, in a logical manner to infer the driver's cognitive state.
One way to address this challenge is by using data fusion. Data fusion
systems can align data sets, correlate relative variables, and combine
the data to make detection or classification decisions (Waltz, 1998).
One benefit of using a data fusion perspective to detect cognitive
distraction is that data fusion can occur at different levels of
abstraction. For instance, sensor data are aggregated to measure driver
performance at the most concrete level. These performance measures can
then be used to characterize driver behavior at higher levels of
abstraction, such as at the level of maneuver. This hierarchical
structure can be used to logically organize the data and inferences, and
to reduce parameter space in the detection procedure. The fusion systems
also can continuously refine the estimates made at each level across
time, which enables a real-time estimation of cognitive distraction.

There are two general approaches to implementing a data fusion system:
top-down and bottom-up. The top-down approach identifies the targets
based on known characteristics, such as shape and kinematic behavior. In
the detection of cognitive distraction, the top-down approach uses the
behavioral responses of drivers under high levels of cognitive load,
reflecting existing theories of human cognition such as Multiple
Resource Theory (Wickens, 2002) and ACT-R (Salvucci & Macuga, 2002). The
limitation of the top-down approach is that it makes it impossible to
implement data fusion without a complete understanding of the underlying
process---something that is lacking in the area of driver distraction.

The bottom-up approach overcomes this limitation, and uses data mining
methods to extract the characteristics of the targets from the data
directly. Data mining includes a broad range of approaches able to
search large volumes of data for unknown patterns, using techniques such
as decision trees, evolutionary algorithms, support vector machines, and
Bayesian networks. These methods are associated with multiple
disciplines (e.g., statistics, information retrieval, machine learning,
and pattern recognition) and have been successfully applied in business
and health care domains (Baldi & Brunak, 2001; Tan, 2005). In the
driving domain, decision tree, Support Vector Machines (SVMs), and
Dynamic Bayesian Networks (DBNs) have successfully captured the
differences in behavior between people driving normally and when
distracted, and produced promising results in terms of detecting
cognitive distraction (Liang, Reyes, & Lee, In press-a, In press-b;
Zhang, Owechko, & Zhang., 2004).

The strategies for constructing data fusion systems include using the
top-down approach alone, the bottom-up approach alone, or a mixed
approach, which combines both top-down and bottom-up strategies. The
choice of strategies depends on the availability of domain knowledge, as
shown in Table 5.6. When the targets are very well understood, a data
fusion system can be constructed using the top-down approach only.
Currently, most data fusion systems use this strategy. Nevertheless, the
lack of domain knowledge imposes an important constraint on this
top-down-alone strategy in some domains, such as the detection of
cognitive distraction. The bottom-up-alone and mixed strategies overcome
the limitation. Oliver and Horvitz (2005) have demonstrated the
effectiveness of these two strategies. They successfully used Hidden
Markov Models (HMMs) and DBNs to construct a layered data fusion system
able to recognize office activities by learning from sound and video
data.

[]{#__RefHeading___Toc173654940 .anchor}Table 5.6. Matrix of data fusion
strategies and the availability of domain knowledge.

  ------------- -------------------- ------------ ------------ ------------
                                     Data fusion               
                                     strategies                

                                     Top-down     Mixed        Bottom-up
                                     approach     approach     approach

  Domain        All available        √            √            √
  knowledge                                                    

                Partially available              √            √

                Not available                                √
  ------------- -------------------- ------------ ------------ ------------

Detecting cognitive distraction requires a bottom-up data mining
strategy because the effects of cognitive demand on driving are not
clearly understood. Although some theories of human cognition can help
explain driver behavior, most theories only aim to describe, rather than
predict, human performance, and thus cannot be used to detect cognitive
distraction. Some theories, like ACT-R, represent promising approaches
that are beginning to make predictions regarding distraction and driver
behavior. On the other hand, various data mining methods have been used
to detect cognitive distraction. Zhang et al. (2004) used a decision
tree to estimate driver cognitive workload from glances and driving
performance. In two other studies, Support Vector Machines (SVMs) and
Bayesian Networks (BNs), successfully identified the presence of
cognitive distraction from eye movements and driving performance (Liang
et al., 2007, In press-a, In press-b; Zhang et al., 2004). Thus,
strategies using bottom-up and mixed approaches are suitable data fusion
strategies for detecting cognitive distraction.

The procedure for detecting cognitive distraction can be formulated in
two sequential stages---feature refinement and state refinement---as
shown in Figure 5.20. Feature refinement uses the top-down approach and
transforms sensor data (such as eye and driving performance raw data)
into performance measures based on an understanding of what measures may
be most sensitive to distraction. The sensor data are collected at a
high frequency (e.g., 60 Hz) and include eye tracking systems and
measures of vehicle speed and driver steering inputs. Feature refinement
transforms raw data into eye movements described as fixations, saccades,
and smooth pursuits according to the speed and dispersion
characteristics of these movements. Various eye movement measures (such
as fixation duration and saccade distance) are then calculated to
describe drivers' scanning activity. Indicators of cognitive
distraction, such as the standard deviation of gaze, are used as inputs
for the next stage. State refinement then fuses these measures to infer
a driver's cognitive state. In this stage, data mining methods are
applied to train detection models from the data.

![](media/image24.wmf)

[]{#__RefHeading___Toc173654930 .anchor}Figure 5.20. Data fusion that
transforms raw driving and eye movement data into estimates of cognitive
distraction.

## Data mining techniques to assess cognitive state

Different data mining methods produce different models. Function-based
and probabilistic-based models represent two of the most promising
classes of models for assessing cognitive state. Typical function-based
models include SVMs, linear regression, and polynomial fitting, while
typical probabilistic-based models include decision trees, naïve
Bayesian classifiers, and BNs. Each of these classes has advantages in
detecting cognitive distraction in real time. Most function-based models
have mature techniques for training and testing, as well as fewer
computational difficulties compared to probability-based models. On the
other hand, probability-based models explicitly represent the
relationships between drivers' cognitive state and performance, which
helps summarize knowledge from resultant models. SVMs and BNs are the
representatives of function-based and probability-based models,
respectively.

Like SVMs, BNs can model complex, non-linear relationships. However, BNs
aim to identify relationships between variables, and to use these
relationships to generate model prediction. BNs can explicitly present
the relationships learned from data. As a consequence, studying trained
BNs helps identify cause-effect links between variables, and the
hierarchical structure of a BN provides a systematic representation of
these relationships. BNs are applicable to human-behavior modeling and
have been used to detect affective state (Li & Ji, 2005b), fatigue (Ji,
Zhu, & Lan, 2004b), lane change intent during driving (Kumagai &
Akamatsu, 2006b), pilot workload (Guhe et al., 2005b), and driver
cognitive distraction (Liang et al., In press-a). One disadvantage of
BNs is that they become computationally inefficient when models have a
large number of variables (i.e., 20). Another disadvantage is that
training techniques for BN are less robust than those available for
SVMs.

## Development of SVMs and BNs

Although both SVMs and BNs have been successfully applied to the task of
detecting cognitive distraction, it is not clear which method is more
effective or how they might differ from each other in carrying out this
task. Here, we compare the effectiveness of the SVM, SBN and DBN methods
in detecting driver cognitive distraction from eye movements and driving
performance. Nineteen performance measures were used to detect cognitive
distraction. The training data were randomly selected from experimental
data collected in a simulated driving environment. Testing used accuracy
and two signal-detection-theory measures. DBNs, which consider
time-dependent relationships, were expected to have the best
performance. Of the two methods that only consider the relationship at
single time point, it was expected that SVMs would perform better than
SBNs because SVMs have fewer computational difficulties in terms of
training and testing. The data for these comparisons were the same as
those used to develop the SVMs and BNs discussed in previous sections.

### Model training 

For each participant, three kinds of models were trained with the
randomly-selected data and the best model settings obtained from the
previous studies (Liang et al., 2007, In press-a, In press-b). Testing
accuracy and the signal detection theory measures of sensitivity and
response bias were used to assess the models.

The best parameter settings for each kind of model were selected. IVIS
drives and baseline drives were used to define each driver's cognitive
state as "distracted" or "not distracted." These became the prediction
targets. The 19 performance measures---including 16 eye movement
measures and 3 driving performance measures that were summarized across
a window (5, 10, 15, or 30 seconds long)---were used as predictive
evidence. SVM models used a continuous form of the measures, while BN
models used a discrete form.

As in the previous description of SVMs, the Radial Basis Function
(RBF),$K\left( x_{i},x_{j} \right) = e^{- \gamma\left| x_{i} - x_{j} \right|^{2}}$,
was chosen as the kernel function, where $x_{i}$and $\ x_{j}$ represent
two data points and γ is a pre-defined positive parameter. The RBF is a
very robust kernel function. Using the RBF, it is possible to implement
both non-linear and linear mapping by manipulating the values of γ and
the penalty parameter C, a pre-defined positive parameter used in the
training calculation (Hsu et al., 2006). In training, we searched for C
and γ in the exponentially growing sequences ranging from 2^-5^ to 2^5^,
using 10-fold-cross-validation to obtain good parameter settings (Chang
& Lin, 2006). "LIBSVM" Matlab toolbox (Chang & Lin, 2006) was used to
train and test the SVM models.

BN training included structure learning, which identified conditional
dependencies between variables, and parameter estimation, which
determined the strength of these dependencies. With 19 measures, the
structures of the BNs were constrained so that the training procedure
was computationally feasible. For SBNs, the direction of the arrows with
the target node---"distraction" or "not distraction"---was from the
target node to performance nodes. The performance nodes could connect
with one another. For DBN models (see Figure 5.21), the arrows within a
time step were present only in the first time step and constrained as
SBNs. After the first step, the arrows were only from the nodes in the
previous time step to the current one. The BN models were trained using
a Matlab toolbox (Murphy, 2004) and an accompanying structure learning
package (LeRay, 2005).

![](media/image25.wmf)

[]{#__RefHeading___Toc173654931 .anchor}Figure 5.21. Constrained DBN
Structure, where solid arrows represent relationships within the first
time step, dotted arrows represent relationships across time steps, and
H and E presents the predictive target and performance measures,
respectively.

Three types of models were trained with randomly selected data for each
participant. The training data took about 2/3 of the total data. The
other 1/3 was used as testing data. The training data for SVM and SBN
models were time-independent instances, and the training data for DBN
models were 120-second sequences. In total, there were 108 models
trained, 36 models (9 participants x 4 window size) each of DBNs, SBNs,
and SVMs.

### Model evaluation 

Model performance was evaluated using the same measures used for the SVM
and BN models in the previous sections.

## Model comparison

We conducted a 3x4 (model types: DBN, SBN, and SVM by window size: 5,
10, 15, and 30 seconds) factorial analysis on three model-performance
measures using a mixed linear model with participants as a repeated
measure. We then performed post hoc comparisons using the Tukey-Kramer
method with SAS 9.0.

DBN, SBN and SVM models differed in terms of accuracy and sensitivity
(testing accuracy: F(2,16) =6.6, p=0.008; sensitivity: F(2,16) =32.5,
p\<0.0001. As shown on the left in Figure 5.22, DBNs and SVMs were more
accurate than SBNs (DBNs: t(16) =3.4, p=0.003; SVMs: t(16) =2.77,
p=0.01. The DBN and SVM models had similar accuracy, t(16) =0.66, p=0.5.
On the right side of the figure, the DBN models are shown to be
significantly more sensitive than the SVM and SBN models, SBN:
t(16)=7.7, p\<0.0001; SVM: t(16) =6.1, p\<0.0001. The SVM and SBN models
had similar sensitivity, t(16)=1.6, p=0.13. These comparisons indicate
that DBNs can predict driver distraction more precisely than SBNs and
SVMs. Although the SVM and SBN models showed similar sensitivity, the
SVM models had an advantage in terms of testing accuracy, perhaps due to
their robust learning technique.

![](media/image26.jpeg){width="3.0131944444444443in"
height="1.7618055555555556in"}
![](media/image27.jpeg){width="3.1173611111111112in"
height="1.8069444444444445in"} Testing accuracy Sensitivity

[]{#__RefHeading___Toc173654932 .anchor}Figure 5.22. Comparisons of
testing accuracy and sensitivity.

The decision bias was marginally different for the three models,
F(2,16=2.8, p=0.09). The DBN models were more liberal than the SBN and
SVM models (DBN: -1.85; SBN: -0.47; SVM: -0.55) with marginal
significance, SBN: t(16) =2.1, p=0.051; SVM: t(16) =2.0, p=0.06). The
SBN and SVM models had similar response biases, t(16) =0.1, p=0.9, which
were not different from the neutral model that is characterized by zero,
SBN: t(16)=1.1, p=0.3; DBN: t(16) =1.3, p=0.2. These results help
explain the discrepancy in the comparisons of the DBNs' and SVMs'
testing accuracy and sensitivity. Although less sensitive than the DBN
models, the SVM models achieved accuracy similar to the DBN models by
using a more neutral strategy. Nevertheless, to explain how SBN and SVM
models resulted in different accuracy levels given their similar
sensitivity and response bias, the analyses of hit and false alarm rates
were needed.

Figure 5.23 shows show that DBNs had higher hit rates---and SVMs had
marginally higher hit rates---compared to SBNs, F(2,16)=4.8, p=0.02;
DBN: t(16) =3.1, p=0.008; SVM: t(16)=2.0, p=0.06. False alarm rates for
the three types of models were similar, F(2,16)=1.1, p=0.4. This
indicates that the DBN and SVM models reached higher accuracy than the
SBN models by generating a greater hit rate.

The effect of window size interacts with model type to affect the false
alarm rate, F(3, 24)=3.0, p=0.052; interaction: F(6, 46)=2.0, p=0.08.
Figure 5.24 disaggregates the false alarm data in Figure 5.23 to show
how false alarms were affected by window size. False alarms increased
with window size from 5 to 15 seconds, and then decreased at 30 seconds,
particularly for the DBN (see solid line in Figure 5.24). All the models
followed this trend, but the magnitude of the change was much more
dramatic for the DBN than for the SBN and SVM models. No main effect or
interaction was found for testing accuracy, sensitivity, response bias,
or hit rate. ![](media/image28.jpeg){width="5.571527777777778in"
height="3.2604166666666665in"}

[]{#__RefHeading___Toc173654933 .anchor}Figure 5.23. Comparisons of hit
and false alarm rates.

![](media/image29.wmf){width="5.822916666666667in" height="4.21875in"}

[]{#__RefHeading___Toc173654934 .anchor}Figure 5.24. Comparison of false
alarm rate for DBNs, SBNs, and SVMs for window sizes of 5 to 30 seconds.

In summary, DBNs produced more sensitive models than SBNs and SVMs. The
DBN and SVM models were more accurate and had higher hit rates than the
SBN models. However, the effects of response bias on the three types of
models were only marginally significant. Window size and its interaction
with model type did not affect testing accuracy, sensitivity, response
bias, or hit rate, but marginally affected false alarm rate.

## Discussion

Compared to SBN and SVM models, the DBNs, which model time-dependent
relationships between drivers' behavior and cognitive state, produced
the most accurate and sensitive models. This indicates that changes in
drivers' eye movements and driving performance over time are important
predictors of cognitive distraction. At the same time, the SVM models
detected driver cognitive distraction more accurately than the SBN
models. This suggests that the SVM learning technique has advantages
over the BN technique. The cross-validation learning process seems to
have resulted in better parameters for the SVMs. We used a 10-fold
cross-validation to search for the best C and γ values in the range of
2^-5^ to 2^5^. One possible result of selecting good parameters may be
evident in the marginally increased hit rates and relatively lower false
alarm rates of the SVM models, although the difference in false alarm
rate was not significant. In addition, SVMs have fewer computational
difficulties than BNs. It took less than one minute for SVMs to train a
model, but approximately a half hour for SBNs, and even longer for DBNs
using an equal number of training data.

Real-world distraction detection systems will need to both accurately
detect driver cognitive distraction and minimize the number of false
alarms to promote acceptance of the system. An interesting finding of
this paper is that window size had a marginal effect on false alarm
rate, but did not affect the other four measures. This effect was
particularly salient for DBNs. It means that either a small (5 s) or
large (30s) window size used to summarize drivers' performance measures
will decrease false alarms without affecting overall model performance.
However, as shown in Figure 5.24, the false alarm rates are still
relatively high. Reducing false alarm rate is an important issue that
future studies need to address.

Based on these comparisons, a hybrid learning algorithm that combines
the time-dependent relationship and SVM learning technique could result
in even better-performing models for detecting driver cognitive
distraction from eye movements and driving performance. Possible ways to
integrate SVMs and DBNs include bagging and paralleling. Bagging
describes using multiple models to make predictions on one target. It
involves first training multiple (an odd-number of) SVM and DBN models
with different training datasets to form a set of models. Then, each
model makes a prediction for the same datum (or case). The final
prediction of cognitive state for this datum ("distracted" or "not
distracted") depends on the vote of all models in the set. This method
can reduce the variance of prediction and avoid overfitting.

Paralleling involves connecting two models sequentially. For example, if
some aggregated descriptions of drivers' behavior, such as eye scanning
patterns, are demonstrated to be essential for identifying cognitive
distraction, we can first use SVMs to build models to identify the eye
scanning patterns from eye movement measures, and then use DBN models to
infer cognitive states from the patterns identified. One such an
approach combined a Bayesian Clustering by Dynamics and SVM model to
forecast electricity demand (Fan, Mao, Zhang, & Chen, 2006).

Models developed using data mining methods can be reciprocal with the
top-down theories. For example, the relationship identified from BNs may
help uncover evidence to support a current theory or to create a new one
regarding the cognitive processes that underlie cognitive distraction.
Such theories related to human cognition can, in turn, provide top-down
constraints that can be integrated into bottom-up models, such as the
initial structure and structural constraints of BN models. Data mining
and cognitive theories can directly cooperate to identify cognitive
distraction. For instance, data mining methods can be used to summarize
driver performance into an aggregated characteristic of driving
behavior, such as seriously-impaired driving behavior or diminishment of
the visual scanning field. A model based on a cognitive theory can then
take the characteristic as an input to identify a driver's cognitive
state. Thus, a combination of top-down and bottom-up models may provide
more comprehensive prediction of cognitive distraction than either
alone.

We have discussed how to detect visual and cognitive distraction as if
they occur in isolation. In the real driving environment, cognitive
distraction often occurs together with visual and manual distraction. To
obtain comprehensive evaluation of driver distraction, the detection
needs to cover all kinds of distractions. Two models---one that detects
visual and one cognitive distraction---need to work together in future
IVIS. To simplify detection, the procedure can begin by checking the
level of visual and manual distraction, because such distraction is much
easier to detect than cognitive distraction. If visual distraction is
detected, it may not then be necessary to check cognitive distraction.

# References