Human Factors: The Journalof the Human Factors and
Ergonomics Society
http://hfs.sagepub.com/

Cognitive Neurophysiologic Synchronies : What Can They Contribute to the Study ofTeamwork?
Ronald H. Stevens, Trysha L. Galloway, Peter Wang and Chris Berka
Human Factors: The Journal of the Human Factors and Ergonomics Society 2012 54: 489 originallypublished online 12 December 2011
DOI: 10.1177/0018720811427296
The online version of this article can be found at:
http://hfs.sagepub.com/content/54/4/489

Published by:
http://www.sagepublications.com

On behalf of:

Human Factors and Ergonomics Society

Additional services and information for Human Factors: The Journal of the Human Factors and Ergonomics
Society can be found at:
Email Alerts: http://hfs.sagepub.com/cgi/alerts
Subscriptions: http://hfs.sagepub.com/subscriptions
Reprints: http://www.sagepub.com/journalsReprints.nav
Permissions: http://www.sagepub.com/journalsPermissions.nav>> Version of Record - Jul 13, 2012
OnlineFirst Version of Record - Dec 12, 2011

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

What is This?

SPECIAL SECTION: Methods for the Analysis of Communication

Cognitive Neurophysiologic Synchronies: What
Can They Contribute to the Study of Teamwork?
Ronald H. Stevens, Trysha L. Galloway, and Peter Wang, UCLA IMMEX
Project, Culver City, California, and Chris Berka, Advanced Brain Monitoring,Inc., Carlsbad, California

Objective: Cognitive neurophysiologic synchronies(NS) are low-level data streams derived fromelectroencephalography (EEG) measurements that canbe collected and analyzed in near real time and in realistic
settings. The objective of this study was to relate theexpression of NS for engagement to the frequency of
conversation between team members during Submarine
Piloting and Navigation (SPAN) simulations.
Background: If the expression of different NS
patterns is sensitive to changes in the behavior of teams,
they may be a useful tool for studying team cognition.
Method: EEG-derived measures of engagement(EEG-E) from SPAN team members were normalized
and pattern classified by self-organizing artificial neuralnetworks and hidden Markov models. The temporal
expression of these patterns was mapped onto teamevents and related to the frequency of team members’
speech. Standardized models were created with pooleddata from multiple teams to facilitate comparisonsacross teams and levels of expertise and to provide aframework for rapid monitoring of team performance.
Results: The NS expression for engagement shifted
across task segments and internal and external taskchanges.These changes occurred within seconds and wereaffected more by changes in the task than by the person
speaking. Shannon entropy measures of the NS data stream
showed decreases associated with periods when the teamwas stressed and speaker entropy was high.
Conclusion: These studies indicate that expressionof neurophysiologic indicators measured by EEG may
complement rather than duplicate communication
metrics as measures of team cognition.
Application: Neurophysiologic approaches may
facilitate the rapid determination of the cognitive
status of a team and support the development of novel
adaptive approaches to optimize team function.
Address correspondence to Ronald H. Stevens, UCLAIMMEX Project, 5601 W. Slauson Avenue #272, Culver
City, CA 90230; e-mail: immex_ron@hotmail.com.
HUMAN FACTORS
Vol. 54, No. 4, August 2012, pp. 489-502
DOI:10.1177/0018720811427296
Copyright © 2012, Human Factors and Ergonomics Society.

Keywords: team neurodynamics, neurophysiologic
synchrony, artificial neural networks, EEG

Introduction
Much of recent teamwork research has used
externalized events focusing on who is a member of the team, how they work together, andwhat they do to perform their work. There havebeen fewer studies looking at the when of teamwork interactions, although the dynamics ofteam function are known to be complex (Marks,
Mathieu, & Zaccaro, 2001; Mathieu, Maynard,
Rapp, & Gilson, 2008). One framework forstudying the when of teams is macrocognition(Warner, Letsky, & Cowen, 2005), defined as theexternalized and internalized high-level mental
processes employed by teams to create newknowledge. External processes are those associated with observable actions and measurable in aconsistent, reliable, repeatable manner. Internalized
processes are indirectly approached throughqualitative metrics such as think-aloud protocols
or surrogate quantitative metrics (pupil size, EEG
metrics, galvanic skin responses).
Speech provides a detailed and dynamic representation of teamwork. When team members
interact, their communication streams contain
information about knowledge, uncertainty,
awareness of the situation, stress, and othercognitive states (Cooke, Gorman, & Kiekel,2008). Speech has structure in the content ofwhat is being said, flow, relating to who is
speaking along with specific speech acts suchas questioning, answering, making a statement,
and so on. Speech is also sequential, temporal,
and relational as people tend to speak one after
another, and what is currently being said has
temporal antecedents (Gorman, 2005; Markset al., 2001).
Communication streams are central forstudying teamwork, yet additional measureswould be useful that are relevant, unobtrusive,

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

490		

August 2012 - Human Factors

and obtained in real time and can be practically
implemented (Salas, Cook, & Rosen, 2008).
Neurophysiologic approaches may provide
such measures.
Entrainment of human rhythms by stimuli inthe environment is common, spanning timescales of milliseconds to days (Buzaki, 2006).
For instance, brain activity in individuals(within brain) can be synchronized by visual orauditory streams where different brain rhythms
become entrained by the frequencies of thestimuli (Will & Berg, 2007). Similarly, the neural synchronization of guitarists playing duets
can become entrained by external auditory signals (i.e., a metronome; Lindenberger, Li,
Gruber, & Muller, 2009).
We earlier hypothesized that as team members performed their duties, each would exhibit
fluctuations in, and perhaps entrainment of,cognitive components such as attention, workload, or engagement and the levels of thesecomponents might reflect aspects of teamwork.
As a result, we collected the simultaneous
expression of EEG-derived cognitive measuresfrom three-person teams to begin to constructneurophysiologic models of teamwork (Stevens,
Galloway, Berka, & Sprang, 2009; Stevens,
Galloway, Berka, & Behenman, 2010a, 2010b).
The measures developed, termed neurophysiologic synchronies (NS), are low-level data
streams representing the second-by-second
quantitative coexpression of the same neurophysiologic/cognitive measure by different
team members. The cognitive measures modeled included engagement and workload derived
from EEG data streams (Berka et al., 2007).
These studies provided a proof of concept forthe modeling approach and began to position
NS into the broader context of teamwork. They
established that NS were not uniformly
expressed during all portions of the task andalso that they showed significant associations
with speech flow (i.e., who was speaking, but
not who was speaking to whom) and speechacts (i.e., questioning, responding, and making
a statement).
The goal for the current study was to extend
NS research to teams operating in real-world,complex situations. Three hypotheses were
proposed:

1. NS models can be created that are sensitive to longterm (minutes/hours) and short-term (seconds/
minutes) changes in the task.
2. NS models can be used for comparing NS dynamics across teams and training sessions.
3. The dynamics of NS expression relate to someestablished aspects of team cognition, yet contribute something new.

MethodTasks

For this study we used navigation training
tasks that are integral components of theSubmarine Officer Advanced Course (SOAC)
at the U.S. Navy Submarine School, Groton,
Connecticut. Submarine Piloting and Navigation(SPAN) is a high-fidelity simulation containing
dynamically programmed situation events suchas encounters with approaching ship traffic,changing weather conditions, and instrument
failure. There are task-oriented cues to guidethe mission, team member cues providing information on how other members of the team are
performing/communicating, and adaptive behaviors that help the team adjust.
The teams contain 11 or 12 members in positions officer on deck (OOD), navigator (NAV),
assistant navigator (ANAV), contact coordinator(CC), fathometer (FATH), helm (HELM), quartermaster on watch (QMOW), radar operator(RAD), recorder (REC), periscope operator(SCOPE), and captain (CAPT) and/or instructor(INST). The simulations require a mixture oftask work and teamwork. For example, the task
work for the RAD would be adjusting the range
and bearing line on the radarscope, whereas the
teamwork would be appropriately conveying
this information to the CC in case of a new contact. Although we have collected EEG teamwork
data from 21 SPAN sessions, the data reported
here were derived from a subset of 12 of thosesessions selected as (a) the persons in thesame six crew positions were monitored by
EEG, (b) there were no role or membershipshifts in the teams across training sessions, and(c) there were three SOAC teams and threeexperienced submarine navigation teams thateach performed two SPAN simulations. SOAC
teams and sessions are designated with a T for

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Cognitive Neurophysiologic Synchronies

E2S1
E2S2

- 1 Min ‘til Next Round- 30 Sec. ’til Next Round- Standby for a Round- Mark a Round- End Round

E4S2
T5S5
T4S2
TIME

Figure 1. Mapping the periodic updating of thesubmarine position with the recorders’ speech.
During Submarine Piloting and Navigation therecorder uses a five-step countdown (shown to theleft) to the moment when a round is taken. The
timing of these five steps is shown for five teams
where each row represents one of the steps. The teamdesignations beginning with an E are experienced
submarine navigation teams; those with a T are
student teams.

491

expertise. For all teams there were periods when
the rhythm of rounds was broken, which wasoften indicative of stressful conditions.
Interleaved with these deterministic events
are situations arising from new ship traffic,
increased proximity to hazards, equipment malfunctions, or reduced visibility. In contrast to theregular updating of the submarine’s position,these events are more perturbations to the regularfunctioning of the team. Some are rapid such as aman overboard, whereas others evolve over 5 to
10 min and may be based on previous decisions.
The speech patterns of the team in response to
the evolving situation are much less regular thanwere those of the REC during rounds (Figure 2).
The debriefing is the most structured segment of
the training with team members reporting inorder, beginning with the NAV. The task times
ranged from 75 to 120 min, and the proportion oftime allocated to the scenario and debriefing was
variable depending on the team.
Electroencephalography

the team and S for the session (e.g., T4S1); forexpert teams, E is substituted for the team designation (e.g., E1S1). Each SPAN session begins
with a briefing outlining the mission goals andproviding information on position, contacts,
weather and sea state. The scenario segment ismore dynamic and contains easily identified
processes of teamwork along with others which
are less well defined. One regular process is the
updating of the ship’s position termed rounds.
Here, three navigation points are chosen, usuallyvisually, and the bearing of each from the boat ismeasured and plotted on a chart. This process
occurs every 3 min with a countdown from the
1-min mark. The REC counts down to the fix
and logs the data. The regularity of this process
is shown by the speech patterns of the REC forfive SPAN sessions (Figure 1).
The two expert sessions, E1S1 and E1S2,
mostly show complete five-step rounds countdowns. The patterns were less regular for SOAC
teams T4S2 and T5S5 where steps were omitted
and occasionally fixes were missed. Anotherexpert team E4S2 began the scenario with four
effective fixes and then began having difficulties conducting regular rounds. This example isshown as it indicates there are likely levels of

The ABM B-Alert® system contains an easily applied wireless EEG system that includes
intelligent software designed to identify andeliminate multiple sources of biological andenvironmental contamination and allow realtime classification of cognitive state changes
even in challenging environments. The ninechannel wireless headset includes sensor site
locations F3, F4, C3, C4, P3, P4, Fz, Cz, POz ina monopolar configuration referenced to linked
mastoids. ABM B-Alert software acquires thedata and quantifies alertness, engagement, andmental workload in real time using linear andquadratic discriminant function analyses withmodel-selected power spectral density (PSD)
variables in each of the 1 Hz bins from 1 to 40
Hz, ratios of power bins, event-related power,
and/or wavelet transform calculations.
The data processing uses eye-blink decontaminated EEG files containing second-bysecond calculations of the probabilities of highEEG-engagement (EEG-E), which is related toprocesses involving information gathering, visualscanning, and increased attention (Berka et al.,2004, 2007). This measure is generated from 22
PSD variables obtained from electrode combinations FzPOz and CzPOz over 1 to 40 Hz bins.

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

492		

August 2012 - Human Factors

ANAV
XO / INST
CC
FATH
HELM
NAV
OOD
QMOW
RAD
REC

E2S1

ANAV
XO / INST
CC
FATH
HELM
NAV
OOD
QMOW
RAD
REC

T5S5

Figure 2. Speech patterns during sample Submarine Piloting and Navigation (SPAN) team performances.
The dynamics of speech are shown for an experienced submarine navigation team (labeled E2S1), and one
junior officer navigation team that was midway through their required SPAN training (labeled T5S5). The
demarcations between the briefing, scenario and debriefing segments are shown by arrows. ANAV = assistantnavigator; XO/INST = commanding officer/instructor; CC = contact coordinator; FATH = fathometer;
HELM = helm; NAV = navigator; ODD = officer on deck; QMOW = quartermaster on watch; RAD =radar operator; REC = recorder.

The neuropsychological tasks used to build
the algorithm and subsequently used to individualize the algorithm’s centroids were presented using proprietary acquisition software.
The algorithm was trained using EEG datacollected during the OSLER maintenance of
wakefulness task (OSLER; Krieger & Ayappa,
2004), eyes closed passive vigilance (EC),eyes open passive vigilance (EO), and threechoice active vigilance (3CVT) tasks todefine the classes of sleep onset (SO), distraction/relaxed wakefulness (DIS), lowengagement (LE), and high engagement (HE),respectively.
Simple baseline tasks are used to fit the EEGclassification algorithms to the individual so the
cognitive state models can be applied to increasingly complex task environments. These methods have proved valid in EEG quantification of
drowsiness–alertness during driving simulation,in simple and complex cognitive tasks, and in

military, industrial, and educational simulation
environments (Berka et al., 2004, 2007; Stevens
et al., 2007).
Layered Models of
Neurophysiologic Synchronies

The first modeling step (Figure 3a) normalizes the second-by-second EEG-E measures toa team member’s average levels. This normalization identifies when a particular team member was experiencing above or below averagelevels of EEG-E and, by comparing across teammembers, whether the team as a whole wasexperiencing above or below their individual
average levels for the session. In this normalization the EEG-E levels are partitioned into theupper 33%, the lower 33%, and the middle
33%, and these are assigned values of 3, –1, and1, respectively, values chosen to facilitate further processing and enhance visualizations(Stevens et al., 2010a; Figure 3).

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Cognitive Neurophysiologic Synchronies

493

Figure 3. Data normalization and modeling. The top panel (A) shows the raw levels of EEG-E being
normalized into training vectors for neural network classification (Panel B). ANN = artificial neuralnetwork; QMOW = quartermaster on watch; NAV = navigator; ODD = officer on deck; ANAV = assistantnavigator; CC = contact coordinator; RAD = radar operator.

The next step (Figure 3b) combines thesevalues at each epoch for each team member
into a vector representing the EEG-E levels
for the team as a whole; these vectors areused to train unsupervised artificial neuralnetworks (ANN) to generate different NS_E
patterns of the team (Stevens et al., 2010a). In
this process EEG-E training vectors arerepeatedly (2,000–4,000 times) presented to a
1 × 25 node ANN where the output neurons
were organized in a linear architecture. Pilot
studies with architectures varying from 16 to
100 nodes indicated that the 25 node architecture provided a balance of speed and sensitivity. The ANN acts as a classifier similar to
K-means clustering with the advantage thateach node competes with its neighbors to theleft and right for the training vector and a
topology develops where vectors most similar to each other become closer and moredisparate vectors are pushed away. The outputof this training is 25 histogram patterns, termed
NS patterns, showing the relative levels of
EEG-E for each team member on a second-bysecond basis. Profiles of three NS_E patterns
are shown in Figure 4. Each pattern contains
six histograms, one for each team member, andthe height shows the relative levels of engagement. NS_E Pattern 1 represents a team where
Members 2 and 4 had average levels of engagement, Team Member 6 showed above average
engagement levels, and Team Members 1, 3,
and 5 were below average. NS_E Pattern 10
represents a team where Members 1, 4, and 6
had above average levels of engagement andthe other team members were below average.
NS_E Pattern 24 was where all team members
had above average levels of engagement
except Team Member 5, who showed average
engagement.

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Team Member 5High
Average
Low

Team Member 3

August 2012 - Human Factors

Team Member 1494		

Figure 4. Examples of neurophysiologic synchrony pattern profiles. The pattern designations above eachfigure are the same as in Figure 5. The high, average and low EEG-E designations are derived from thecoding of the training vectors in Figure 3a.

Mapping NS_E Patterns to NS_E States
State 1

3
2
1
0
–1

Patterns 1–5
Patterns 6–10
Patterns 11–15
Patterns 16–20
Patterns 21–25

3
2
1
0
–1
3
2
1
0
–1

State 2

State 3

State 41 2 3 4 5 6State 5

3
2
1
0
–1
3
2
1
0
–1

Team Order: QMOW NAV OOD ANAV CC RAD
Figure 5. NS_E pattern and state classifications. The NS_E patterns are numbered 1–5,
6–10, etc. row-wise from the left to right. Each of the six histograms in each patternrepresents the EEG-E levels of a team member. The order of team members is shown
below the figure. The surrounding state squares with the 25 boxes show the most frequent
associations of particular NS_E patterns with each NS_E state. QMOW = quartermasteron watch; NAV = navigator; ODD = officer on deck; ANAV = assistant navigator; CC =contact coordinator; RAD = radar operator.

To enable comparisons across teams, ANN
models were generated using pooled data from 8
six-person teams. This resulted in a training set
of 31,450 team vectors (∼9 hr of teamwork).
The resulting 25 NS_E patterns obtained following modeling are shown in Figure 5. NS_E
Patterns 1 through 4 represented times whenmost team members had low EEG-E levels,
whereas NS_E Patterns 13–15 and 22–24 represented times when most team members hadhigh EEG-E levels.
Autocorrelation studies have suggested that
there may be a temporal component to NS pattern expression, a hypothesis supported

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Cognitive Neurophysiologic Synchronies

by adding an additional modeling step using
hidden Markov modeling (HMM; Stevens,
Galloway, Berka, & Behenman, 2010c). This
process models temporal associations between
different symbols that are the 25 NS patterns
resulting from the ANN modeling. Like the
ANN training process, HMM requires a training step. The input data for this training are 120
epoch-long segments of NS patterns obtainedby segmenting the NS pattern data stream.
HMM generally requires an estimate of thenumber of states to model into, and we haveused five based on prior work and pilot studies(Soller & Stevens, 2007). The outputs of thisHMM modeling are termed NS states, and themapping of the different NS_E patterns to NS_E
states is shown in Figure 5.
What do the different NS_E states represent?
Unfortunately there is no simple answer. We
have conducted HMM modeling dozens oftimes but have failed to develop simple patterns
such as “team is fully engaged” or “fully unengaged.” There is always heterogeneity of NS_E
patterns in each state, which is indicative of anunderlying dynamics of NS data streams(Stevens & Gorman, 2011).

495

T1S1 NS_E Patterns - From ANN Clustering
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Debrief

Scenario

Brief

T1S1 NS_E States - From HMM Modeling
1
2
3
4
5

500

1000

1500

T4S2 NS_E States

2000

Epoch

2500

3000

3500

1
2
3
4
5

1000
1
2
3
4
5

2000

T5S5 NS_E States

3000

Epoch

4000

5000

500 1000 1500 2000 2500 3000 3500 4000
Epoch

E2S1 NS_E States

Results

Hypothesis 1: NS models can be created thatare sensitive to long-term (minutes/hours) andshort-term (seconds/minutes) changes in thetask. NS_E patterns and states can be visualized
on a second-by-second basis or binned overmultiple epochs for statistical analysis. Figure 6
illustrates a second-by-second mapping of the
NS_E patterns and states for SPAN team T1S1.
A consistent feature observed in all SPAN training sessions is the shift in NS_E expression at
the scenario–debrief junction (indicated by thearrow). Here team T1S1 showed an increased
expression of NS_E Patterns 1–5 and 17–21,
which were previously expressed at low levels.
The shift was more obvious for the HMMderived NS_E states where most of the debriefing was dominated by NS_E State 3. Similar
shifts at the brief–scenario junction are less pronounced, which is not surprising, as one briefing component is the determination of the ship’s
starting position which is similar to rounds.
Similar NS_E state shifts at the task segment

1000

2000

3000

Epoch

4000

5000

Figure 6. Dynamic expression of NS_E patterns andstates during Submarine Piloting and Navigation(SPAN) training sessions. The top figure shows
the second-by-second expression of each of the
NS_E patterns expressed by SPAN team T1S1, andimmediately below it is the expression of the five
NS_E states. The expressions of NS_E states forthree other teams are shown for comparison. The
briefing, scenario, and debriefing segment junctions
are shown by the arrows.

junctions are also shown for three other teams.
These shifts illustrate that NS_E expression issensitive to major task changes.
Another important feature illustrated in
Figure 6 is that NS pattern and state expressions
are not homogeneous but often punctuated byrecurrent blocks that span shorter time scales.

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

496		

August 2012 - Human Factors

A
State

1
2
3
4
5

00
26

00
27

00
28

00
29

00
30

00
31

00
24

00
25

00
32

Epoch

B
State

1
2
3
4
5

00
20

00
21

00
22

00
23

00
26

Epoch

Figure 7. Changes in NS_E expression during pauses
in two simulations. These figures focus on two ∼10
min segments during which the skippers of two boats
paused the simulation to address their navigation
teams. Panel A is team E2S1, and Panel B is team E3S2.
The gray areas highlight the periods of the pauses.

These 30- to 60-s state recurrences provide useful landmarks for relating NS expression withshort-term simulation events, team responses,
or team speech. Figure 7 shows two experienced navigation teams where the boat’s skippers paused the simulation midscenario toaddress the crews. For the first team (E1S2,
Panel A) prior to Epoch 2890 the team was
mainly expressing NS_E States 1, 2, and 3.
Within seconds of the CAPT beginning to speak
they transitioned through NS_E State 4 to NS_E
State 5. The team remained in this state until the
scenario resumed when they returned to prior
NS_E state expressions. For the second team
(E3S2, Panel B) the team was mainly expressing NS_E State 2, which then switched to NS_E
States 3 and 4 while the skipper was speaking.
Hypothesis 2: NS models can be used forcomparing NS dynamics across teams andtraining sessions. The next study compared thefrequency distributions of the five NS_E statesacross four teams that each performed two
SPAN sessions (Figure 8). As expected from the

Figure 8. NS_E state frequencies for Submarine
Piloting and Navigation (SPAN) sessions segments.
The frequency percentage of each of the five NS_E
states was calculated for the scenario (top) briefing(middle) and debriefing (bottom) segments for eight
SPAN sessions.

shifts in NS_E pattern expression at task junctions as shown in Figure 5, there was a differential expression of NS_E states during the
scenario and debriefing with NS_E State 2 being
overexpressed in the scenario and NS_E State
5 underexpressed (χ2 = 1326, df = 4, p < 0.01).
For most teams NS_E States 1 and 2 weremost frequent during the scenario. These represented periods when many of the team members
were highly engaged (refer back to the NS pattern/state classification diagram in Figure 5).
The exception was Team 1, where NS_E State 4
predominated; this state represents periods
when many of the team members had below
average engagement. There were no significant
state frequency differences between SPAN
Sessions 1 and 2 for any of the teams. Therewas more across-team and across-session heterogeneity during the briefing and debriefing
segments, and as expected from the data in
Figure 6, the distributions in the briefing weremore similar to those in the scenario than were
the distributions in the debriefing.
Hypothesis 3: The dynamics of NS expression relate to some established aspects of

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Cognitive Neurophysiologic Synchronies

Pause
Simulation

Begin
Debrief

CC Report

497

OOD Report

Begin MOB
Discussion
Capt.

Simulation
E States
12345

NAV Report

Capt.

Capt.

Capt.
Closing

3100 3200 3300 3400
34000 3500 3600 37000 3800
39000 4000
40000 4100 4200 4300 4400
34400 4500
4446000 4700 4800
poch
EEpochFigure 9. Mapping discussion units to NS_E state expression. Initially the NAV summarized the simulationand illustrated and discussed areas for improvement. Next, the CC provided an alternative navigation solution
to an issue that was raised by the NAV (“My personal comfort level would have been to shoot in between thetwo inbound merchants and have them be our front and back line backers,” etc.). In the next section (Epochs
3776–3922) the CAPT contributed general comments (“My main point is don’t wait until the last minute andthen call that guy after you just messed with his own radar picture,” etc.). The debriefing continued along
similar lines changing topics with each NS transition. OOD = officer on deck; MOB =man overboard.

teamwork, yet contribute something new. The
content and flow of communication are often
data sources for studying teamwork, and messages are generally regarded as a fundamental
unit for analysis. The debrief sections of SPAN
simulations were initially chosen to explore
linkages between NS_E expression and speech,
as there is a regular pattern of discussion where
only one person speaks at a time. The dialog during the debriefing from one SPAN session wastranscribed, time coded, and aligned with NS_E
expression (Figure 9). The gray bands emphasize
the reports of different team members.
From this example it appears that the major
NS state transitions occurred around blocks ofideas or discussion units, that is, when closureof a topic was achieved and a new topic began.
The NS_E state shifts were not linked to a
specific speaker as multiple team members contributed to each discussion unit. Also, although

the team was predominately in NS_E State 5
when the CAPT spoke, there was a long period
at the end of the debrief where the CAPT was
speaking and the team was in NS_E State 1.
Although the preceding approach may provide
useful information regarding associations between
NS_E expression and speech, the process was not
optimal for studying SPAN teamwork. First, most
speech is highly asynchronous during the scenario, with intermixed conversations of the different members of the navigation team. Second, thisapproach required a time-consuming analysis,
decreasing its usefulness as a rapid response monitor of the team to changing SPAN events.
The next studies addressed these challenges
by combining speech and neurophysiologic
analyses with entropy calculations and expert
performance analysis.
The markers for speech frequencies and the
NS_E patterns are both symbolic data, and for

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

498		

August 2012 - Human Factors

1000

Epoch

2000

3000

4000

5000

NULL
ANAV
CC
CAPT
FATH
HELM
INST
NAV
OOD
QMOW
RADAR

B
1

Events
2

5
3 4

6

7

8
- 1 Min ‘til Next Round- 30 Sec. ’til Next Round- Standby for a Round- Mark a Round- End Round

4.5

3.0

NS_E Entropy

Speaker Entropy

REC
SCOPE
UNK

RoundsSequences

Speaker

A

3.4

0

C

1000

2000

3000

4000

1000

5000

Epoch

D

2000

3000

4000

5000

Epoch

Figure 10. Linking NS_E entropy with team speech and performance events. Panel A uses bar symbols to markthe second-by-second speech of the different members of a Submarine Piloting and Navigation (SPAN) team
(T4S2). Panel B shows the rounds sequence for this team as described in Figure 1, and above it are markers
referring to different events that are further described in Figure 11. Panel C shows the entropy profile for the
speakers during the simulation, and Panel D shows a similar entropy profile for the NS_E.

quantitative comparisons it would be useful tohave numeric metrics. One transformation is tocalculate the Shannon entropy of the symbolic
speech and NS_E data streams (Shannon,
1951). This metric is derived from information
science and measures the level of uncertainty or“amount of mix” in a symbol stream. The idea
was that as teams organized themselves aroundsignificant task events, there may be changes inthe entropies of NS_E patterns or speech reflecting this cognitive reorganization.
Entropy is expressed in terms of bits calculated from a data stream. Entropy was calculated at each epoch using a 90-s sliding windowof the prior history. As 25 NS_E patterns areavailable for modeling, the maximum entropy
that we could expect from the 25 NS_E patterns
would be log2 (25), or 4.64. For comparison, if
a 90-s sequence of the data stream contained
only 12 of the 25 patterns, then the entropy
would drop to 3.6, and if the data stream contained only a single NS_E pattern, the entropy
would be 0. Similar calculations were applied tospeech. The entropy for speech was calculated
by first assigning a numeric code to each of the

16 speakers in the SPAN; the extra 5 speakersymbols added to the 11-man team included asecond instructor and/or evaluator, the technician as well as other speakers who could not beclearly identified from the audio stream. Thesesymbols were substituted into the speech log,
and then the entropy was calculated as describedfor NS_E.
Throughout the simulation there were significant speaker and NS_E entropy fluctuations,only some of which were associated withchanges in the task boundaries. During the scenario there were periods when the team was
relatively quiet, interspersed with periods when
there was extensive speech by multiple teammembers. The speech of the REC provided evidence of when the team was not functioning
smoothly as indicated by periods when theround was missed or marked multiple times. To
complement these data, a lieutenant who was an
instructor at the Submarine Learning Centersummarized the teams’ stress level and performance from an audio recording of the session.
From these data, a summary of the teams’ performance and dynamics was created (Figure 11),

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Cognitive Neurophysiologic Synchronies

Event

Epoch
1 0-700

499

LT Evaluation

Summary

The team is quiet andfocused.
and the simulation begins with a Round and proposed coursechange.

1 750-1000

Slight stress
Discussion of a potentially close Closest Point of Approach (CPA),
particularly to NAV and and future problems to deal with; failure of Virtual Mapper
OOD.
(VM2) and all contacts lost.

2 1050 – 1300

Stress elevating tomoderate,

2 1300 – 1400

More contact discussion; planning entrance to the Race (aonce through. Speed up to All Ahead Full and there is a crossing
situation with a tug, a Round is marked and a crew members
states “we’re going to get around this guy, right?”
Contact B is detected. This is additional stress to CC while
managing current contacts, and then the military GPS is lost.
Slowing in the race to allow Double Eagle to pass ahead, future
CPA to Double Eagle is zero!

4 1925 – 2050

Heavy stress situation
with multiple contacts.
Very stressful, little tono recommendations
from team to OOD.
Relief in stress
OOD arranges passing arrangements with Double Eagle; team
refocuses on safe plan
Moderate stressConfusion on contact picture; moderate stress while in the Race

5 2065-2230

Moderate stress3 1700 - 1750

3 1750 -1850

5 2245-2350
6 2365-2530
7 2541-2830

Preparing for turning out of the race; projected close CPA toevent B, CC info to OOD breaking down
Very stressful, loud and Turning out of race; close CPA to contact B, 100yds!
busy teamStress reduces
Past CPA and opening from event B; stress reduces, teamSevere stress andconfusionpassing info and trying to get on the same page. Trawlers havestill in reduced visibility.
Man overboard (Epoch 2847). “Believe it or not, I think a man
overboard drops the stress level and focuses the team. Here is
why: prior to the man overboard, the team was focused on avery dynamic problem of navigating through multiple moving
trawlers in reduced vis. Once the man falls overboard, the teamis relieved of that dynamic problem and can focus on this one
problem for which they have structured responses and have

8 2845-3358

Figure 11. Summary of team events and stress levels of team T4S2 in Figure 10. The event column refers to thesteps in Panel B of Figure 10. The lieutenant (LT) evaluation column is the LT’s assessment of the stress levelof the team, and the summary column provides an overview of the events occurring at different times. OOD =officer on deck; CC = contact coordinator)

with the event numbers relating to Panel B in
Figure 10.
Discussion

Three hypotheses were proposed in thisstudy, which if supported would help to better
position team neurodynamics research within thegeneral framework of teamwork. Hypothesis 1
postulated that NS expression would be sensitive to long- and short-term task changes.
Support for this hypothesis resulted in part fromthe nature of the task. SPAN contains task segments that differ significantly in their teamwork
requirements. For all 12 SPAN sessions in thisDownloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

500		

August 2012 - Human Factors

study there were major shifts in NS_E patternand state expression at the scenario–debriefing
junction (four examples were shown). Support
for shorter term changes came from two situations where the skipper of the sub put the simulation in pause midscenario. In both cases there
were rapid and significant shifts in NS_E state
expression. Less direct support was derived by
the linking of team speech with the content of
the discussion in Figure 9. Future studies on
short-term changes in the task will benefit fromthe measurement of the entropy in NS_E pattern streams as described below. Although EEG
metrics are often viewed as being useful overtime scales of milliseconds to seconds, theshifts and more prolonged recurrences of NS_E
states are extending this window to minutes.
There was also support for Hypothesis 2,
which postulated that NS models could be
developed that would enable comparisonsacross teams. Our initial studies used a singletrial approach for developing NS models, thatis, the data from a single performance wereused for deriving the ANN models for that performance. As new models were created for eachtask, comparisons across teams or levels of
experience were difficult as the ANN designations changed because of the probabilistic
assignment of vectors to specific nodes. In thisstudy we pooled the EEG-E data from 8 of the
12 SPAN sessions selected for this study as
described earlier, and all 12 teams were subsequently tested on these generic models. The
NS_E state frequency distributions shown in
Figure 7 are important as they emphasize thelong-term changes at task boundaries and also
begin to document the neurodynamics of SPAN
teams during the scenario.
Across sessions, most teams preferentiallyexpressed NS_E State 2 during the scenario,
which represented a moderately engaged team.
This NS_E state, perhaps along with NS_E
State 1, can be viewed as the normal operating
mode(s) of the team because they were poorly
expressed in the briefing and debriefing segments. In addition to the state transitions at thetask junctions, there were 30- to 50-s periods
during all scenarios studied where only one
NS_E state was expressed. Most often this

NS_E state was not the normal operating mode,and their expression coincided with external
perturbations to the task or periods when significant events were occurring (such as a man
overboard or near collision).
The development of generic NS models also
provides a possible framework for the rapid
reporting of events of significance to the team.
The data that could be reported are of several
types. First, the periods when particular NS_E
states persist, such as the skipper break in
Figure 7, can be reported as changes in the state
recurrence frequencies (Zbilut, Giuliani, &Webber, 1998). Alternatively, the entropy measures for NS_E patterns shown in Figure 10
could be calculated. Entropy in particular could
be important as preliminary studies have indicated that expert teams have fewer of theserecurrent states than SOAC teams and overall
higher NS_E entropy levels during the scenario(Stevens & Gorman, 2011). These measures
may therefore provide a metric for following
training progress over time.
Support for Hypothesis 3, where we began
linking NS_E expression with speech, is lessobvious. From the limited number of mappings
we have performed linking NS_E state expression with the speech content during the debriefing segments (i.e., Figure 9), there appears to be
an association with the changing dynamics ofthe conversation. Speech during the scenario,
however, is much more dynamic than during thedebriefing, and the data in Figure 10 and similar
studies with other teams suggest that the NS_E
entropy is lowest when the speech entropy ishighest; that is, the team is more cognitively
organized when the communication across teammembers appears the least organized.
Combining performance data (rounds), evaluation data (lieutenant’s notes), and the frequency and diversity of speech with thechanging levels of NS_E entropy suggests that
periods of decreased entropy represent times
when the team is experiencing stress. One possibility is that the team has lost its flexibility
and has locked itself into a more restrictive cognitive state. Alternatively, the decreased entropy
may represent the increased organization of theteam in response to a difficult situation.

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

Cognitive Neurophysiologic Synchronies

Acknowledgments
We extend a special thanks to Capt. Kenneth
Swan, Thomas Wohlgemuth, Lt. Robert Buckles,
and Dr. Jerry Lamb at the Submarine Learning
Center, Groton, Connecticut, for logistics support,
and Adrienne Behneman and Veasna Tan for the datacollection. A very special thanks to Oli D. for support and encouragement. Approved for public
release, distribution unlimited. The views expressed
are those of the authors and do not reflect the official
policy or position of the Department of Defense or
the U.S. Government. This is in accordance with
DoDI 5230.29, January 8, 2009. This work was supported by NSF SBIR Award 0822020, Office ofNaval Research Award N00014-11-M-0129, and anaward from the Defense Advanced Research Projects
Agency (DARPA) under contract numbers
NBCHC070101, NBCHC090054.

Key points•• Cognitive neurophysiologic synchronies forengagement change rapidly in response tochanges in the task and environment; the entropyof the NS data stream provides a quantitative
measure of these changes.
•• NS_E expression is not closely associated with
who is speaking but rather with “blocks of ideas”
in the conversation.
•• Generic models of NS_E pattern and state expression can be used for comparing across teams andsessions.
•• Expression of neurophysiologic indicators measured by EEG may complement rather than duplicate communication metrics as measures of teamcognition.

References
Berka, C., Davis, G., Johnson, R., Levendowski, D., Whitmoyer, M.,
Fatch, R., Murray, S., Ensign, W., Yanagi, M., & Olmstead, R.
(2007). EEG correlates of task engagement and mental workload in vigilance, learning and memory tasks. Aviation Spaceand Environmental Medicine, 78, B231–B244.
Berka, C., Levendowski, D. J., Cvetinovic, M. M., Petrovic, M. M.,
Davis, G., Lumicao, M. N., Zivkovic, V. T. (2004). Real-time
analysis of EEG indexes of alertness, cognition, and memoryacquired with a wireless EEG headset. International Journal of
Human-Computer Interaction, 17, 151–170.
Buzaki, G. (2006). Rhythms of the brain. Oxford, UK: Oxford University Press.
Cooke, N. J., Gorman, J. C., & Kiekel, P. A. (2008). Communicationas team-level cognitive process. In M. P. Letsky, N. W. Warner,& C. A. P. Smith (Eds.), Macrocognition in teams: Theories andmethodologies (pp. 51–64). Farnham, UK: Ashgate.

501

Gorman, J. C. (2005, July). The concept of long memory for assessing the global effects of augmented team cognition. Paper
presented at the 11th International Conference on HumanComputer Interaction, Las Vegas, NV.
Krieger, A. C., & Ayappa, I. (2004). Comparison of the maintenance of wakefulness test (MWT) to a modified behavioral test
(OSLER) in the evaluation of daytime sleepiness. Journal of
Sleep Research, 13, 407–411.
Lindenberger, U., Li, S.-C., Gruber, W., & Muller, V. (2009).
Brains swinging in concert: Cortical phase synchronization
while playing guitar. BMC Neuroscience, 10, 22–34.
Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally
based framework and taxonomy of team process. Academy ofManagement Review, 26, 356–376.
Mathieu, J., Maynard, M. T., Rapp, T., & Gilson, L. (2008). Teameffectiveness 1997–2007: A review of recent advancements
and a glimpse into the future. Journal of Management, 34,
410–476.
Salas, E., Cook, N. J., & Rosen, M. A. (2008). On teams, teamwork, and team performance: Discoveries and developments.
Human Factors, 50, 540–547.
Shannon, C. E. (1951). Prediction and entropy of printed English.
Bell System Technical Journal, 30, 50–64.
Soller, A., & Stevens, R. H. (2007). Applications of stochastic analyses for collaborative learning and cognitive assessment. In
G. Hancock & K. Samuelson (Eds.), Advances in latent variable mixture models (pp. 217–254). Greenwich, CT: Information Age.
Stevens, R. H., Galloway, T., & Berka, C. (2007). EEG-relatedchanges in cognitive workload, engagement and distraction
as students acquire problem-solving skills. In C. Conati, K.
McCoy, and G. Paliouras (Eds.), Lecture Notes in Computer
Science, User Modeling 2007 (187–196). Heidelberg, Germany: Springer.
Stevens, R., Galloway, T., Berka, C., & Behenman, A. (2010a).
Identification and application of neurophysiologic synchronies
for studying team behavior. In Proceedings of the 19th conference on behavior representation in modeling and simulation(pp. 21–28). Centerville, OH: BRIMS Society.
Stevens, R. H., Galloway, T., Berka, C., & Behneman, A. (2010b,
December). A neurophysiologic approach for studying teamcognition. Paper presented at the Interservice/Industry Training Simulation and Education Conference, Orlando, FL.
Stevens, R. H., Galloway, T., Berka, C., & Behneman, A. (2010c,
September–October). Temporal sequences of neurophysiologic
synchronies can identify changes in team cognition. In Proceedings of the Human Factors and Ergonomics Society 54th annual
meeting (pp. 190–194). Santa Monica, CA: Human Factors and
Ergonomics Society.
Stevens, R. H., Galloway, T., Berka, C., & Sprang, M. (2009).
Can neurophysiologic synchronies provide a platform foradapting team performance? In D. D. Schmorrow, I. V. Estabrooke, & M. Grootjen (Eds.), Foundations of augmented
cognition (Vol. 5638, pp. 658–667). Heidelberg, Germany:
Springer.
Stevens, R. H., & Gorman, J. (2011, July). Mapping cognitive
attractors onto the dynamic landscapes of teamwork. Paper
presented at the 14th International Conference on HumanComputer Interaction, Orlando, FL.
Warner, N., Letsky, M., & Cowen, M. (2005, September). Cognitive model of team collaboration: Macro-cognitive focus.
Paper presented at the 49th Human Factors and Ergonomics
Society annual meeting, Orlando, FL.

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012

502		

August 2012 - Human Factors

Will, U., & Berg, E. (2007). Brain wave synchronization andentrainment to periodic acoustic stimuli. Neuroscience Letters,
424, 55–60.
Zbilut, J. P., Giuliani, A., & Webber, C. L., Jr. (1998). Recurrence
quantification analysis and principle components in the detection of short complex signals. Physics Letters A, 237, 131–135.

Ron Stevens, PhD, is professor of microbiology, andmember of the Brain Research Institute at the UCLA
School of Medicine. He is the director of the Internetbased IMMEX problem-solving project which has
engaged more than 150,000 students and teachers incomputational education and professional development activities that span elementary school throughmedical school. He received the Foundations ofAugmented Cognition award from the Augmented
Cognition Society. His current interests are the useof machine learning tools and electroencephalography (EEG) to model the acquisition of scientificproblem-solving skills.
Trysha Galloway directs the EEG studies for theInteractive Multi Media Exercises (IMMEXTM) laboratory and is coauthor on 14 peer-reviewed publishedstudies. Her research interests blend the populationbased advantages of probabilistic performance modeling with the detection of neurophysiologic signals to

help personalize the learning process in complex
education and training activities.
Peter Wang is the programming director for the
IMMEXTM project. During the past 20 years he
has jointly developed hundreds of online scientificproblem-solving suites.
Chris Berka, CEO and cofounder of Advanced Brain
Monitoring, has more than 25 years’ experiencemanaging clinical research and developing and commercializing new technologies. She received her BA
with distinction in psychology/biology at Ohio State
University and completed graduate studies in neuroscience at UCSD. She is coinventor of seven patented and seven patent-pending technologies and is
the principal investigator or coinvestigator for grants
awarded by the National Institutes of Health,
DARPA, ONR, and NSF. She has 10 years’ experience as a research scientist with publications on theanalysis of the EEG correlates of cognition in
healthy subjects and patients with sleep and neurological disorders.
Date received: January 14, 2011
Date accepted: August 25, 2011

Downloaded from hfs.sagepub.com at HFES-Human Factors and Ergonomics Society on July 18, 2012