ARL-TR-8267 • JAN 2018

US Army Research Laboratory

A Report on Applying EEGnet to Discriminate

Human State Effects on Task Performance

by Ashton Gauff, Humberto Muñoz-Barona,

Addison Bohannon, and Jean Vettel

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the

Army position unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official endorsement or approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
ARL-TR-8267 • JAN 2018

US Army Research Laboratory

A Report on Applying EEGnet to Discriminate

Human State Effects on Task Performance

by Ashton Gauff and Humberto Muñoz-Barona

Southern University and A&M College, Baton Rouge, LA

Addison Bohannon and Jean Vettel

Human Research and Engineering Directorate, ARL

Approved for public release; distribution is unlimited.
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704‐0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the

data needed, and completing and reviewing the collection information.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing

the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704‐0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202‐

4302.  Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a

currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD‐MM‐YYYY) 2. REPORT TYPE 3. DATES COVERED (From ‐ To)

4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT

NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

13. SUPPLEMENTARY NOTES

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

OF

ABSTRACT

18. NUMBER

OF

PAGES

19a. NAME OF RESPONSIBLE PERSON

a. REPORT b. ABSTRACT c. THIS PAGE 19b. TELEPHONE NUMBER (Include area code)

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18

Jan 2018 Technical Report

A Report on Applying EEGnet to Discriminate Human State Effects on Task

Performance

Ashton Gauff, Humberto Muñoz-Barona, Addison Bohannon, and Jean Vettel

ARL-TR-8267

Approved for public release; distribution is unlimited.

June - September 2017

US Army Research Laboratory

ATTN: RDRL-HRF-B

Aberdeen Proving Ground, MD 21005-5066

primary author’s email: <addison.w.bohannon.civ@mail.mil>.

In this project, we utilized optimization to discriminate brain data. Participants completed 2 cognitive tasks while ongoing brain

activity was recorded from electrodes on their scalp. Our analysis examined whether we could identify what task the participant

was performing from differences in the recorded brain time series. We modeled the relationship between input data (brain time

series) and output labels (task A and task B) as an unknown function, and we found an optimal approximation of that function

from among a family of functions. We employed stochastic gradient descent to minimize the estimation error known as the loss

function. The optimal function from among our family of approximate functions, EEGNet, successfully discriminated brain data

from a single participant with approximately 90% accuracy. Future research will apply EEGNet on data from more participants

as well as develop approaches to adapt its architecture for the non-Euclidean domains.

machine learning, deep learning, EEG, network neuroscience, sleep study

22

Addison W Bohannon

Unclassified Unclassified Unclassified UU (410) 278-8974

ii
Approved for public release; distribution is unlimited.

Contents

List of Figures iv

Acknowledgments v

1. Introduction 1

2. Technical Approach 2

2.1 Convolutional Neural Networks 3

2.1.1 2-Dimensional Convolution 3

2.1.2 Pooling 3

2.1.3 Batch Normalization 4

2.2 EEGNet 5

2.3 Learning Objective 6

2.4 Stochastic Gradient Descent 7

3. Implementation and Analysis Using EEGNet 8

4. Conclusion and Future Directions 9

5. References 10

List of Symbols, Abbreviations, and Acronyms 11

Distribution List 13

iii
Approved for public release; distribution is unlimited.

List of Figures

Fig. 1 2-dimensional discrete convolution ...............................................4

Fig. 2 2-dimensional max pooling operator ..............................................4

Fig. 3 Batch normalization ..................................................................5

Fig. 4 Layers of EEGNet model............................................................5

Fig. 5 Receiver’s operating characteristic curve for one random partition of the

data.......................................................................................9

iv
Approved for public release; distribution is unlimited.

Acknowledgments

This research was supported by mission funding to the US Army Research Laboratory (Jean Vettel and Addison Bohannon) and through a summer student internship

(Ashton Gauff) and a Faculty Fellowship from the Thurgood Marshall College Fund

(Humberto Muñoz-Barona). The views and conclusions contained in this report are

those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the US Government.

v
Approved for public release; distribution is unlimited.

INTENTIONALLY LEFT BLANK.

vi
Approved for public release; distribution is unlimited.

1. Introduction

As the amount of battlefield technology continues to increase, Soldiers are faced

with a daunting task of trying to integrate diverse information across numerous

devices. The growing information burden across devices has spawned a strong interest in “smart technology,” where algorithms strive to become a digital assistant

to streamline information processing for the end user. Unfortunately, users typically have as many examples of the technology making pervasive errors as examples where the technology provides helpful assistance, such as the numerous

auto-correct fail memes, comical questions incorrectly deduced by voice recognition software, or lane-keeping sensors on cars that alert for highway medians or

chain link fences. These failures often originate from a rigid application of an algorithm. We posit that the technology would become smarter if there was real-time

feedback from the user that could modify the algorithm. Thus, research at the US

Army Research Laboratory (ARL) targets the development of adaptive technology

based on the real-time detection of human state (e.g., engagement or fatigue) to

improve the integrated performance of humans and systems.

While the concept of human state is inherently nebulous, the intuition is that the

configuration of our physiology underlying our behavior is predictive of upcoming

performance. Take the example of driving while fatigued. When we are tired, we

may swerve out of a lane unexpectedly. If sensors could use real-time physiology

to detect a human state that is predictive of decreased vigilance, smart technology

could intelligently assist with lane-keeping technology. In contrast, on days where

the driver does not have physiological markers of decreased vigilance, the smart

technology would not interfere with human driving as the swerving in and out of

lanes may be necessary in a construction zone. This is just one example about how

knowledge of the human state could enable technology to adapt more intelligently

to the user’s real-time needs.

These adaptive technologies depend on a fundamental scientific achievement to reliably detect physiological states that are indicative of performance. Ongoing work

at ARL examines physiological signals across the brain and body, including how

well brain activity can predict upcoming task performance. In this research, brain

activity is often measured from the scalp using electroencephalogram (EEG) sensors that record ongoing electrical activity generated when different brain regions

1
Approved for public release; distribution is unlimited.

are communicating information in support of behavioral performance. Fluctuations

in the functional activity are then linked to variability in task performance, attempting to capture the physiological features of human state needed to predict real-time

human task needs. Based on the success of convolutional neural networks (CNN) in

extracting meaningful representations of data in image and speech processing, we

would like to be able to apply these methods to EEG.1 Currently, the need for lots

of training data poses the most significant obstacle to implementing these powerful

algorithms on EEG.

In my summer internship, I learned about ongoing work at ARL to overcome this

data challenge of EEG by using CNNs with minimal parameters. I used my background in mathematics to understand the algorithms involved in implementing EEGNet, an ARL-developed CNN for EEG data, and apply EEGNet to a new problem.2

This report summarizes that work. CNNs are powerful function approximators, and

in the first section of the technical approach, I review the functions that make up the

EEGNet convolutional neural network. To understand EEGNet, I had to learn about

maximum likelihood estimators that allow us to pose learning problems as optimization problems. I derive the EEGNet objective function from a maximum likelihood estimate in the second section. In the third section of the technical approach, I

describe stochastic gradient descent, the learning algorithm used with CNNs. Then,

I describe the implementation of EEGNet and its application to an ARL data set

that examines how human state changes following naturalistic sleep loss. The preliminary results effectively discriminate between human states, and this leads to

a discussion of future directions in which I propose using EEGNet to ask further

questions about physiological state changes and how they impact task performance.

2. Technical Approach

We wanted to discriminate between human states from individuals performing different cognitive tasks versus observing their EEG activity alone. For this particular

project, we used EEG recordings from individuals during either rest or while performing an attentional bias task. Our goal was to discriminate between the 2 tasks.

We used machine learning to discriminate between the states. We modeled the cognitive task as a Bernoulli trial from which we observed the EEG recording. This

made the state discrimination problem one of state detection. Then, we posed our

learning problem as one of posterior inference, in which we used EEGNet2for the

2
Approved for public release; distribution is unlimited.

inference model. Finally, we used stochastic gradient descent as our learning algorithm to fit EEGNet to the observed data.

In this section, we will briefly review CNNs, introduce EEGNet, derive the posterior

inference problem, and review stochastic gradient descent.

2.1 Convolutional Neural Networks

CNNs are powerful machine learning algorithms that have been successfully applied to image processing, automatic speech recognition, speech translation, and

natural language processing.1 More recently, they have even been applied to the

analysis of EEG data.2 CNNs are a subset of algorithms known as deep learning.

Deep learning essentially implies the composition of multiple nonlinear functions.

f(x) = (fn ◦ · · · ◦ f0)(x). (1)

CNNs comprise particular functions: 2-dimensional convolutions, pooling, and batch

normalization.

2.1.1 2-Dimensional Convolution

Convolutions are a common tool in signal processing. These are linear transformations that filter a signal to accentuate or attenuate particular features. In CNNs, we

use the 2-dimensional generalization of the convolution ∗ : R

n × Rm → Rn×m:

(f ∗ g)(x, y) = Xn

i=1

Xm

j=1

f(i, j)g(x − i, y − j). (2)

Here, f is the signal and g is the filter. During the learning process, we are selecting

the filters that capture the most important features of the signal. In images, these

features are often edges and blobs. A visualization of 2-dimensional convolution is

shown in Fig. 1.

2.1.2 Pooling

Pooling, or downsampling, reduces the dimensionality of the signal and decreases

the sensitivity of the algorithm to noise in the signal. Pooling is a dimensionality

reduction technique:

P : R

n→ R ρ

n

.

(3)

3
Normally, this operation is either the average or maximum of adjacent elements of

the signal. A visualization of pooling is shown in Fig. 2.

N

2.1.3 Batch Normalization

Batch normalization is a technique for improving the learning process. It normalizes the features of signals in subsequent functions within the CNN, enforcing a

multivariate Gaussian distribution of features. Figure 3 shows the distribution of

observations before and after batch normalization.

Let X1

(k)

, . . . , X

(k)

be a subset of the observations where k corresponds to the batch.

More generally, these could be the input to any function fi

in the CNN. We would

like to model these observations to be nearly independent and identically distributed

(IID) Gaussian samples X

(k)∼ N (x; μ = 0, σ

2= 1). In practice, we can estimate

Approved for public release; distribution is unlimited.

4

Fig. 1 2-dimensional discrete convolution

Figure reprinted from https://mlnotebook.github.io/post/CNN1/.3

Fig. 2 2-dimensional max pooling operator

Figure reprinted from https://mlnotebook.github.io/post/CNN1/.3

i
Approved for public release; distribution is unlimited.

the mean and variance of each batch with the sample mean x ̄

(k)

and sample variance

(s

2

)

(k)

and normalize the observations accordingly as in a z-score.

z

(k)

i = (s

2

)

(k)x

(k)

i + x ̄

(k)

. (4)

Learning parameters for the sample mean β ≈ s

2

and sample variance γ ≈ x ̄ make

this procedure more computationally efficient and the learning more smooth.4

Fig. 3 Batch normalization

2.2 EEGNet

EEGNet is a particular instance of a CNN with a set of functions optimized for EEG

observations. The functions that compose EEGNet are shown in Fig. 4. It is divided

into 4 layers, in which most have a 2-dimensional convolution, pooling operation,

and batch normalization. The first layer can be interpreted as a nonlinear spatial

filter. The second layer can be interpreted as a nonlinear temporal filter. Then, the

third layer aggregates the global features of the signal.

Fig. 4 Layers of EEGNet model

Table reproduced from Lawhern, et al.2

5
Approved for public release; distribution is unlimited.

2.3 Learning Objective

Following the maximum likelihood formulation of Duda et al. in Pattern Classification,5 we modeled the cognitive task to be distributed according to a Bernoulli

distribution conditioned on the associated realization of EEG recording. Our goal

was then to infer the cognitive task given the EEG realization, a statistical inference

task. We used EEGNet to estimate the Bernoulli parameter for each EEG realization

so that we could pose the inference problem as a parameter estimation problem.

This allowed us to use a maximum likelihood estimate to formulate the learning

problem as an optimization problem.

Let X1, . . . , XN , be IID observations of EEG recordings and Y1, . . . , YN to be the

associated cognitive task. As mentioned above, let Yi|Xi ∼ Bern(p(Xi)):

fy|x(yi|xi) = p(xi)

yi



1 − p(xi)

1−yi

. (5)

We will consider the log likelihood of all EEG observations:

lnY

N

i=1

fy|x(yi|xi) = lnY

N

i=1

h

p(xi)

yi



1 − p(xi)

1−yi

i

(6)

=

X

N

i=1

ln hp(xi)

yi

(1 − p(xi))1−yi

i

(7)

=

X

N

i=1

yiln p(xi) + (1 − yi) ln (1 − p(xi)). (8)

Now, we want to maximize the log likelihood. We can consider the joint probability

distribution f(x, y) for the random variables X, Y and maximize the expected value

Ex,y:

max

p

EX,Y ∼f [y ln p(x) + (1 − y) ln (1 − p(x))]. (9)

If p = p(X; ω) is a function of both the EEG observation and parameters ω ∈ Ω,

then we have the more commonly known binary cross-entropy loss:

L(ω) = min

ω∈Ω

E[−y ln p(x; ω) − (1 − y) ln (1 − p(x; ω))]. (10)

6
Approved for public release; distribution is unlimited.

2.4 Stochastic Gradient Descent

Since we were able to pose our state detection problem as an optimization problem,

we selected the commonly used stochastic gradient descent algorithm for our learning algorithm.6 This is a first-order method, which scales well to large data sets and

complex models.

The gradient descent algorithm is a powerful unconstrained optimization method

with convergence guarantees for strongly convex functions. It is an iterative technique that uses only knowledge of the gradient of the function. Let f be a function

of ω and 0 < η < 1 be the learning rate. The algorithm is as follows:

ωk+1 ← ωk − ηk∇ωf(ω). (11)

For objectives that have the form L(ω) = EXf(X; ω), gradient descent can be

computationally expensive. An alternative approach is stochastic gradient descent,

which uses Monte Carlo sampling to estimate the gradient:

EXf(X; ω) ≈

X

N

i=1

f(xi; ω), (12)

where xi ∼ fX. Therefore, we can estimate the gradient of our maximum likelihood

objective as follows:

∇ωEX,Y ∼f [y ln p(x; ω) + (1 − y) ln (1 − p(x; ω))]

≈ ∇ω

1

m

Xm

i=1

yiln p(xi

∂

∂ω; ω) + (1 − yi) ln (1 − p(xi

; ω)), (13)

which yields the following iterative algorithm:

ωt ← ωt−1 + ηt∇ω

1

m

Xm

i=1

yiln p(xi; ω) + (1 − yi) ln (1 − p(xi; ω)). (14)

7
Approved for public release; distribution is unlimited.

3. Implementation and Analysis Using EEGNet

Using Python, we implemented EEGNet in Keras7 with Tensorflow8 back-end according to Fig. 4. We used the default parameters for the Adam9 optimizer (a variant

of stochastic gradient descent). We trained the algorithm for 10 epochs.

Lawhern et al.2 used EEGNet on data from standard brain-computer interface (BCI)

paradigms (i.e., rapid serial visual presentation, motor imagery, and visually evoked

potentials). In this project, we have examined whether EEGNet can be extended to

detect more nebulous cognitive states that capture the influence of naturalistic sleep

loss on task performance.

We tested the approach on data from a previously collected ARL study called Cognitive Resilience and Sleep History (CRASH). Participants (N=29) provided sleep

history over an 18-week time period, including objective measurements of sleep duration and quality from actigraph wrist watches and subjective measurements from

daily web-based questionnaires. They came into the laboratory every 2 weeks for

a 4-h experimental session where brain data was collected while they performed

5 cognitive tasks and a resting state scan. In this novel data set, we have 8 brainbehavior sessions to assess the impact of naturalistic sleep loss on task performance

over an 18-week timeframe.

To test the extension of EEGNet to state detection, we compared EEG data between the resting state, where the participant was able to mind wander as desired,

and the attentional bias task, where the participant had to discriminate letters that

followed emotional faces. We selected one subject from the CRASH data set and

included observations from all 8 recording sessions. We separated the data into nonoverlapping epochs of 500 ms. The data were previously processed using ARL’s

standard pipeline.10

We randomly partitioned our data into training (80%) and testing (20%) 10 times

and reported the performance in Area Under the Receiver’s Operating Characteristic Curve (AUC). The average AUC observed over the 10 iterations was 90% (see

Fig. 5 for the results). This performance is significantly above change and indicates that EEGNet shows promise for our extension of the method to discriminate

cognitive states beyond those of standard BCI paradigms.

8
Fig. 5 Receiver’s operating characteristic curve for one random partition of the data

4. Conclusion and Future Directions

In my summer internship project, I learned how recent work at ARL has made it

possible to apply CNNs to EEG data. Following the work of Lawhern et al.,

2

I

implemented EEGNet, a CNN with relatively few parameters, and used it to classify

human states from EEG recordings in a single subject from the CRASH data set.

To complete this project, I learned how CNNs compose convolutions and pooling

functions to build representations of data, how to use maximum likelihood estimators to pose learning problems as optimization problems, and how to use stochastic

gradient descent to solve optimization problems. Our preliminary results indicate

that EEGNet successfully discriminates between cognitive tasks.

In future work, I would like to use EEGNet to detect the presence of more complicated states. As described in the introduction, state detection will facilitate adaptive

technologies that can respond to changes in the user’s state. In this preliminary

work, we only discriminated the performance of one cognitive task from a rest

state. However, the CRASH data set has recordings from several sessions for each

subject. I would like to use EEGNet to discriminate between sessions for the same

subject. Because the sessions differ by sleep history, the ability to discriminate by

session would indicate that EEGNet could discriminate sleep history of a user. This

could be used in future adaptive technologies to detect user fatigue and likely poor

performance.

Approved for public release; distribution is unlimited.

9
Approved for public release; distribution is unlimited.

5. References

1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–

444.

2. Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ.

EEGNet: a Compact Convolutional Network for EEG-based Brain-Computer

Interfaces. arXiv preprint arXiv:1611.08024; 2016.

3. Machine Learning Notebook. https://mlnotebook.github.io/; [accessed 2017-

11-25].

4. Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training

by reducing internal covariate shift. In: Bach F, Blei D, editors. Proceedings of

the 32nd International Conference on Machine Learning; Vol. 37; Lille, France:

PMLR; 2015. p. 448–456.

5. Duda RO, Hart PE, Stork DG. Pattern classification. New York (NY): John

Wiley & Sons; 2012.

6. Bottou L. Large-scale machine learning with stochastic gradient descent. In:

Proceedings of COMPSTAT’2010: 19th International Conference on Computational Statistics; Lechevallier Y, Saporta G, editors. Heidelberg: PhysicaVerlag HD; 2010 Aug. p. 177–186.

7. Chollet F. Keras: deep learning for Python. https://github.com/fchollet/keras;

[accessed 2017-11-25].

8. Abadi M et al. TensorFlow: large-scale machine learning on heterogeneous

systems [whitepaper]. http://tensorflow.org/; 2015.

9. Kingma D, Ba J. Adam: A method for stochastic optimization. In: Proceedings

of the 3rd International Conference for Learning Representations; San Diego,

CA: ICLR 2015; 2015.

10. Garcia JO, Brooks J, Kerick S, Johnson T, Mullen TR, Vettel JM. Estimating

direction in brain-behavior interactions: Proactive and reactive brain states in

driving. NeuroImage. 2017;150(Supplement C):239–249.

10
Approved for public release; distribution is unlimited.

List of Symbols, Abbreviations, and Acronyms

ARL US Army Research Laboratory

AUC area under the receiver’s operating characteristic curve

BCI brain-computer interface

CNN convolutional neural network

CRASH cognitive resilience and sleep history

EEG electroencephalogram

EEGNet ARL-developed CNN for EEG data

IID independent and identically distributed

11
Approved for public release; distribution is unlimited.

INTENTIONALLY LEFT BLANK.

12
1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIR ARL

(PDF) IMAL HRA 

RECORDS MGMT

RDRL DCL

TECH LIB

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 ARL

(PDF) RDRL HRB B

T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 

35898-7290

8 ARL

(PDF) SFC PAUL RAY SMITH CENTER

RDRL HRO COL H BUHL

RDRL HRF J CHEN

RDRL HRA I MARTINEZ

RDRL HRR R SOTTILARE

RDRL HRA C A RODRIGUEZ

RDRL HRA B G GOODWIN

RDRL HRA A C METEVIER

RDRL HRA D B PETTIT

12423 RESEARCH PARKWAY

ORLANDO FL 32826

1 USA ARMY G1

(PDF) DAPE HSI B KNAPP

300 ARMY PENTAGON 

RM 2C489

WASHINGTON DC 20310-0300

1 USAF 711 HPW

(PDF) 711 HPW/RH K GEISS

2698 G ST BLDG 190 

WRIGHT PATTERSON AFB OH 

45433-7604

1 USN ONR

(PDF) ONR CODE 341 J TANGNEY

875 N RANDOLPH STREET

BLDG 87 

ARLINGTON VA 22203-1986

1 USA NSRDEC

(PDF) RDNS D D TAMILIO

10 GENERAL GREENE AVE 

NATICK MA 01760-2642

1 OSD OUSD ATL

(PDF) HPT&B B PETRO

4800 MARK CENTER DRIVE

SUITE 17E08

ALEXANDRIA VA 22350 

ABERDEEN PROVING GROUND

13 DIR ARL

(PDF) RDRL HR

J LOCKETT

P FRANASZCZUK

K MCDOWELL

K OIE

RDRL HRB

D HEADLEY

RDRL HRB C

J GRYNOVICKI

RDRL HRB D

C PAULILLO

RDRL HRF A

A DECOSTANZA

RDRL HRF B

A EVANS

A BOHANNON

J VETTEL

RDRL HRF C

J GASTON

RDRL HRF D

A MARATHE
Approved for public release; distribution is unlimited.

INTENTIONALLY LEFT BLANK.

14