ORIGINAL RESEARCH
published: 22 December 2021
doi: 10.3389/fnhum.2021.785562

Multi-Parameter Physiological State
Monitoring in Target Detection Under
Real-World Settings
Yang Chang 1,2 , Congying He 1,2 , Bo-Yu Tsai 2,3 and Li-Wei Ko 1,2,3,4*
1

Institute of Bioinformatics and Systems Biology, National Yang Ming Chiao Tung University, Hsinchu, Taiwan, 2 Center
for Intelligent Drug Systems and Smart Bio-Devices (IDS2B), College of Biological Science and Technology, National Yang
Ming Chiao Tung University, Hsinchu, Taiwan, 3 Department of Electrical and Computer Engineering, National Yang Ming
Chiao Tung University, Hsinchu, Taiwan, 4 Drug Development and Value Creation Research Center, Kaohsiung Medical
University, Kaohsiung City, Taiwan

Edited by:
Gianluca Di Flumeri,
Sapienza University of Rome, Italy
Reviewed by:
Marius Klug,
Technical University of Berlin,
Germany
Pietro Aricò,Sapienza University of Rome, Italy*Correspondence:
Li-Wei Ko
lwko@nctu.edu.tw
Specialty section:
This article was submitted toBrain-Computer Interfaces,
a section of the journal
Frontiers in Human Neuroscience
Received: 29 September 2021
Accepted: 03 December 2021
Published: 22 December 2021
Citation:
Chang Y, He C, Tsai B-Y andKo L-W (2021) Multi-ParameterPhysiological State Monitoringin Target Detection Under Real-World
Settings.
Front. Hum. Neurosci. 15:785562.
doi: 10.3389/fnhum.2021.785562

Mental state changes induced by stimuli under experimental settings or by daily events inreal life affect task performance and are entwined with physical and mental health. In thisstudy, we developed a physiological state indicator with five parameters that reflect the
subject’s real-time physiological states based on online EEG signal processing. These
five parameters are attention, fatigue, stress, and the brain activity shifts of the left andright hemispheres. We designed a target detection experiment modified by a cognitive
attention network test for validating the effectiveness of the proposed indicator, as suchconditions would better approximate a real chaotic environment. Results demonstrated
that attention levels while performing the target detection task were significantly higher
than during rest periods, but also exhibited a decay over time. In contrast, the fatigue
level increased gradually and plateaued by the third rest period. Similar to attention
levels, the stress level decreased as the experiment proceeded. These parameters
are therefore shown to be highly correlated to different stages of the experiment,suggesting their usage as primary factors in passive brain-computer interfaces (BCI).
In addition, the left and right brain activity indexes reveal the EEG neural modulationsof the corresponding hemispheres, which set a feasible reference of activation for an
active BCI control system, such as one executing motor imagery tasks. The proposed
indicator is applicable to potential passive and active BCI applications for monitoring
the subject’s physiological state change in real-time, along with providing a means ofevaluating the associated signal quality to enhance the BCI performance.
Keywords: BCI, EEG, physiological state, attention, fatigue, stress

INTRODUCTION
Brain-computer interfaces (BCI) are methods that offer direct communication pathways between
the human brain and external devices which have attracted much attention in various fields. They
provide great potential for controlling machines, especially through a hands-free approach. To
translate electroencephalographic (EEG) signals into actual commands they depend on users togenerate EEG patterns recognizable to the system. These signals can be improved by appropriate
training (Neuper and Pfurtscheller, 2009), however, even if with such training, the inevitable mental

Frontiers in Human Neuroscience | www.frontiersin.org

1

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

drowsiness (Lin et al., 2010; Bajaj et al., 2020), fatigue (Monteiro
et al., 2019; Xu et al., 2019), and stress (Xia et al., 2018;
Asif et al., 2019). To assess the changes of physiological states
exploiting EEG signals, five parameters are employed to develop
the proposed pentagonal physiological state indicator: attention,
stress, fatigue, and left and right brain activity levels.
Attention has been one of the most discussed cognitive
functions in the field of neuroscience. It is an ability of anindividual to selectively concentrate on relevant information
while filtering irrelevant information. The attentional networks
composed of alerting, orienting, and executive attention
proposed by Posner and Petersen have been widely studied interms of functional and anatomical perspectives (Posner andPetersen, 1990; Raz, 2004; Xuan et al., 2016). The attentional
network test (ANT) is a behavioral task designed to evaluate thethree stages of attention (Fan et al., 2002). The experiment thatwas developed to validate the proposed indicator also utilized
ANT as a reference. Studies based on EEG indicate that attention
can be estimated by detecting changes in alpha power activity,
especially in the frontal lobe which is known to be responsible
in regulating concentration (Quilodran et al., 2008; Doesburg
et al., 2016). Research showed that the power spectra of theta andalpha increase when the subject is going into drowsiness (Junget al., 1997; Lin et al., 2005); consistent experimental evidence has
also demonstrated that mental engagement is inversely correlated
to theta and alpha activities in the frontal region (Pope et al.,
1995; Szafir and Mutlu, 2012). Our previous study has also
demonstrated a real-time alertness detection system using thisapproach (Lin et al., 2010). Furthermore, research has applied
similar approaches to determine attentional states during BCI
experiments, providing a means to improve BCI performance(Lee et al., 2015).
Fatigue is a state that follows excessive physical or cognitive
activity, thus it is typically partitioned as physical fatigue or
mental fatigue (Jason et al., 2010). Physical fatigue is associatedwith body or muscle exhaustion, which leads to unsatisfactory
physical performance. In contrast with physical fatigue, mental
fatigue generally denotes a feeling of sleepiness or drowsiness,and it manifests as decreased attentional capacities and prolonged
reaction times (Boksem et al., 2005), resulting in reduced
efficiency. In a simulated driving experiment designed to study
fatigue by Ma et al. (2019) subjects were allowed limited sleep
and performed driving on tedious paths. It was found thatalpha and beta activities in parietal and occipital lobes werestrongly associated with the driver’s fatigue level (Ma et al., 2019).
Another experiment exploiting simulated driving by Huang et al.
(2016) demonstrated a closed-loop fatigue mitigation system that
detected fatigue from the alteration of theta and alpha power in
occipital components of recorded EEG. Similarly, experimental
results of an ICA-based BCI study by Chuang et al. (2013) showedthat increasing power in theta and alpha ranges were highly
correlated with poor task performance and fatigue.
With the rapid pace of modern lifestyles, people race against
time and experience different kinds of stressors. It is a wellknown fact that stress causes negative effects on our health,emotions, and cognitive functions (McEwen and Sapolsky, 1995;
Pearlin et al., 2005; Lazarus, 2006). However, mild-to-moderatestate changes of the individual during BCI usage still impact
performance. Effects of attention and fatigue are regarded as themain causes of variation in the performance of BCIs (Myrden
and Chau, 2015). Research has found that task engagement andattentional processes may impact the performance of several BCI
techniques, such as P300 (Riccio et al., 2013) and motor imagery(MI) (Zhang et al., 2016). Mental fatigue produced by prolonged
sequences of a cognitive task affects the accuracy of BCIs, as it
decreases the separability of EEG signals (Talukdar et al., 2019).
Furthermore, in a recent study by Zhang et al. (2020), it has beenfound that resting EEG modulation induced by stress is one of
the key factors to signal-to-noise ratio and BCI performance.
To address such impact of mental state changes on active BCIs,
passive BCI has been proposed to assess the user’s cognitive
state during ongoing BCI tasks, allowing the improvement in theinteraction of human-machine system (Zander and Kothe, 2011).
On the other hand, people encounter various situations andevents in daily life that elicit different emotions and physiologicalstates than those in laboratory settings. Beyond their influenceson BCI systems, some physiological states, such as stress andfatigue, are responsible for certain physical and mental healthproblems. For example, stress accounts for physical problems
such as high blood sugar, high blood pressure, high cholesterol,obesity (Clark et al., 2011), and mental problems, e.g., depressiveor anxiety disorders (Virtanen et al., 2007). Fatigue leads todecreased attention, which is the main cause of transportation
accidents (Roets and Christiaens, 2019), and is reported to affectseveral psychological and physiological aspects of health (Locket al., 2018). Besides stress and fatigue, other physiological statesaffect human life in various ways. Since physiological states playsuch important roles in both BCI and human life, how to measureand visualize them both within and beyond experimental settingsis crucial and valuable.
A popular approach of deriving physiological states is fromphysiological signals, including EEG, electrocardiography (ECG)(Lampert, 2015; Hsu et al., 2017; Brás et al., 2018; Huang et al.,2018), electromyography (EMG) (Wijsman et al., 2013; Luijcks
et al., 2014; Van Cutsem et al., 2017; Kehri et al., 2019), respiration(Widjaja et al., 2013; Grassmann et al., 2016), and galvanic
skin response (GSR) (Villarejo et al., 2012; Fernandes et al.,2014; Krupinski et al., 2015; Kurniawan et al., 2016; Liu et al.,2016). Different physiological states regulate these autonomic
physiological signals which naturally respond to changes in thecentral and peripheral nervous systems. Compared with otherphysiological approaches, EEG more directly reflects responses tothe environment and external stimuli present in the processing ofthese events by the central nervous system, thus yielding morereliable outcomes. One of the most common methods for theinvestigation of brain dynamics is to identify the EEG spectral
characteristics under varied conditions. Typically, EEG power
spectra are quantified and divided by frequency ranges, suchas delta (1–3 Hz), theta (4–7 Hz), alpha (8–12 Hz), beta (13–31 Hz), and gamma (32–50 Hz) bands (Tatum, 2014), with theseoscillations within different frequency bands representative of
various brain activations and conditions. Several publications inrecent years have appeared documenting that EEG is a robust
approach to evaluate physiological states, including alertness,

Frontiers in Human Neuroscience | www.frontiersin.org

2

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

effectiveness of the indicator. The signal processing flowchart is
illustrated in Figure 1.

stress has been found to be salutary, with this relationshipbetween stress and performance described as the “inverted-U
theory” (Sapolsky, 2015). The frontal lobes of the brain havebeen proposed as moderators of emotional regulation; extensive
research has shown that frontal alpha asymmetry, representing
the homologous differences between right and left frontal sites(typically F4–F3), of the is associated with affective processing(Harmon-Jones and Gable, 2018; Reznik and Allen, 2018).
The asymmetry score is derived by subtracting natural logtransformed left hemisphere alpha power from the right, withpositive asymmetry scores indicating stronger relative activity(Pfurtscheller et al., 1996). Specifically, it has been demonstrated
that positive asymmetry scores are linked to positive emotions,whereas negative asymmetry scores are associated with stressresponses and negative emotions (Meyer et al., 2014; Quaedflieg
et al., 2015; Zhang et al., 2018a).
Motor imagery has been a widely discussed techniquein BCI applications, and acts as an endogenous BCI, as
it relies on spontaneous brain activities, as opposed tobrain dynamics elicited by an external stimulus. The signals
employed in MI-based BCI are based on principles of
event-related desynchronizations and synchronizations, whichcorrespond to changes of EEG power in the mu rhythm
during either mental imagination of movement or actual motor
execution (Pfurtscheller and Neuper, 1994; Pfurtscheller and
Neurophysiology, 1997). Previous studies revealed electrodepositions C3 and C4 are the most important ones fordifferentiating different motor imagery tasks by acquiring murhythm activities (Ramoser et al., 2000; Pfurtscheller et al., 2006).
To investigate the effectiveness of the proposed physiologicalstate indicator, a prolonged experiment based on target detection
was conducted. As acquiring EEG signals outside of experimental
settings is impractical, a portable multi-channel EEG device wasincorporated into the system development, as the long typicalsetup time and procedure needed for preparation of a standard
off-the-shelf EEG system could affect the results in a study
focusing on a time dependent effect like fatigue. In this study,
a physiological state indicator with five parameters that reflect
the physiological state changes based on EEG dynamics wasdeveloped, specifying the real-time fluctuations of the proposedparameters during the usage of BCI. These five parameters
are attention, fatigue, stress, and the brain activities of the left
and right hemispheres. The incorporated physiological states
have been well-studied, with each having significant evidence
for adopting specific EEG channels and frequency bands toderive them. The proposed experiment was designed to explore
the neurophysiological changes using the indicator during a
prolonged, attention-related task. With the proposed indicator,
BCI usage can be evaluated according to a user’s physiologicalstate changes, yielding improved BCI performances.

Participants
Forty college students (30 males, 10 females) aged from 18 to
30 were recruited for participation in the study. No subjects
reported histories of psychiatric or neurological disorders. All
the participants were instructed to perform the task using theirright hand. The experiments were administered at National Chaio
Tung University (Hsinchu, Taiwan). The study was approvedby the institutional review boards (IRB) of National Chaio
Tung University.

Experimental Paradigm
In this study, we designed a target detection experiment thatsimulates a battlefield scenario to elicit state changes assessedby the proposed indicator (He et al., 2021). The experiment wascomposed of a 30-s EEG baseline recording and five blocks of 12min sessions along with a 5-min rest session. First, the participant
was seated in front of a 23.6” monitor with a relaxed posture.
Then, the wireless EEG device was placed on the participant’shead and adjusted to fit. The participant was asked to fixate at
the monitor showing a cross-mark for 30 s to establish a baseline
recording. Afterward, the participant underwent a 68-min task
to induce attention, fatigue, and stress level changes. In each trial,
the participant was shown a cue representing in an asterisk on theleft or right, indicating the probable location that a target would
appear. The cue was followed by the presentation of a target,
which was a figure of a soldier holding a rifle, accompanied withfive other figures as distractors. The participants were required topress the button on the keyboard according to the direction of
the target as soon as possible (Figure 2). Congruent conditions
occurred if the direction of the target and the cue were thesame and incongruent if they were opposite. To mimic more
realistic circumstances, a condition with an absence of the target
represented a no-target condition, for which the participant was
required to press the “up” button. As the reaction time (RT) of
the no-target condition is assumed to be longer, the criterion of
maximal possible reaction time before a trial to be considered
as fail is set at 2 s. Within a 12-min session, each of the three
conditions were distributed between the 180 trials, resulting in 60
trials of each condition presented randomly during the session.
Each session is followed by a 5-min rest. The total duration of
the experiment was approximately 1 h, which was designed tobring about a mental state of fatigue. During the rest period,
subjects were instructed to relax in order to ameliorate someof the neurological effects resulting from prolonged tension or
discomfort produced during the active tasks; resting data fromthe period after the first 2 min of rest were used so as to minimize
signal artifacts carrying over from the previous active stage. The
combination of session and rest repeated four times, comprising
the whole experiment. The attention parameter was assessed inboth active and resting periods, while the fatigue was assessedand validated in the rest periods because other physiologicalstate changes could interference the measurement of the fatigue
parameter. To further investigate the effect of fatigue caused bycontinuous cognitive performance, the subjects were inquired to

MATERIALS AND METHODS
The parameters in the indicator were derived by EEG signal
processing. We used portable wireless EEG equipment toacquire the EEG signals during the experiment to validate the

Frontiers in Human Neuroscience | www.frontiersin.org

3

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

FIGURE 1 | Signal processing for parameters. The EEG signals were acquired through a 32-channel wireless EEG device and preprocessed for artifact removal.
Then, the signals of the particular sites of interest were decomposed and normalized to yield five parameters.

FIGURE 2 | Experimental paradigm based on target detection. The experiment was composed of five sessions and five resting periods, with each session consistingof 180 trials. Each subject was instructed to respond to the target, and the conditions of the stimulus referred to the relationship between the cue and the target (Heet al., 2021). To induce the effect of fatigue, the experiment lasted about 1 h.

experiment. The references were located at the earlobes via two
ear clips, and the sampling rate was set at 500 Hz. To obtainthe features for physiological states, the acquired EEG signal
was preprocessed for artifact removal and feature extraction. The
signal underwent a 1–50 Hz bandpass filter for the preliminary
removal of artifacts. The artifact removal from the EEG signal was
carried out using the artifact subspace reconstruction (ASR) in
EEGLAB toolbox, which is a component-based method capableof removing large-amplitude noise or artifacts in real-time, andis therefore practical for embedding in the software of portabledevices (Chang et al., 2018). The ASR process comprises threesteps: (1) Extracting reference, (2) determining threshold, (3)rejecting and reconstructing. To implement online ASR, thereference data were extracted using the baseline before theexperiment onset. The reference data was applied to define
the rejection threshold. Therefore, the ASR could reject artifact
components and reconstruct cleaned data in near real-timeusing the established threshold (Chang et al., 2019). The signal
was then processed via fast Fourier transform to obtain powerrate their self-evaluated fatigue level they perceived during task
performances on a 10-point rating scale according to the method
described by Karrer et al. (2004).

Electroencephalographic Signal
Acquisition and Preprocessing
To be compatible with the real-time state indicator, a wireless
32-channel EEG system called VEGA developed by Artise
Biomedical, Taiwan was used for EEG acquisition. It consists ofa Bluetooth module, miniature amplifiers, and semi-dry spongebased electrodes with 32 designated channels: Fp1, Fp2, AF3,
AF4, F7, F3, Fz, F4, F8, FT7, FC3, FCz, FC4, FT8, T7, C3,Cz, C4, T8, TP7, CP3, CPz, CP4, TP8, P7, P3, Pz, P4, P8,
O1, Oz, and O2. The sponge electrodes used in the experimentfunction by absorbing saltwater prior to use; their design makes
them flexible, adequate for hairy scalps, and usable for longterm EEG acquisition (Ko et al., 2019). The impedance of theskin-electrode interface was monitored constantly during the

Frontiers in Human Neuroscience | www.frontiersin.org

4

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

Alpha asymmetry in the frontal region, measured F4/F3 signal
ratios, was utilized as the stress parameter, defined as:

spectral density (PSD) in decibel (dB). The PSD from the baseline
was subtracted from the task periods. Finally, the signals went
through band decomposition, yielding four commonly usedfrequency bands, delta (1–3 Hz), theta (4–7 Hz), alpha (8–12 Hz),and beta (13–31 Hz).

Stress (t) = 5−

where P is the natural log-transformed alpha power and fC isstd(PF4 −PF3 ). Positive asymmetry scores represent stronger relative2
left than right activity as higher alpha band power reflects
lower brain activity.
Similar to the parameters that positively correlate to theamplitudes of EEG power, the parameters of the left and rightbrain activity also followed a normalized formula. Hence, theparameter for left brain activity is defined as:

Development of Physiological State
Indicator
The features used to develop the five parameters were predefined, yet their scales were not specified a priori. To obtaineasily understandable output parameters for each subject, theEEG power retrieved from 40 subjects was normalized to a usable
scale. The general transformation formula was defined based on
normalized t-score, but with the scale ranging from 0 to 10.
The average power for each parameter was derived by averaging
specific power across all subjects, as the power has been processed
through baseline removal. Considering the existing and future
outlier data, the normalization factor is modified to σ/2 insteadof σ. Therefore, it enables 98.75% of the derived physiologicalstate levels, i.e., y (t), to be distributed within 0–10 (µ ± 2.5σ).
Previous study has posited the feasibility of using this approachfor normalization of EEG power values (Thatcher, 1998). The
formula is defined as:

Left (t) = 5 +Right (t) = 5 +1 X PC (t) −PCnfC

where P is the EEG band power of the region of interest (ROI), c
is the channels included in ROI, n is the number of channels, andf is the normalization factor.
For a better, intuitive visualization, the magnitudes of thenormalized parameters were displayed in a pentagonal indicator(Figure 3). Attention is known to be inversely proportional to
the theta- and alpha-band amplitudes in the frontal region (Linet al., 2010; Szafir and Mutlu, 2012), which were the F3, Fz, and F4
sites according to the channel locations on the EEG equipment.
Therefore, the level of the attention parameter was defined as:13PFz (t) −PFz
PF4 (t) −PF4
PF3 (t) −PF3
+
+
fF3
fFz
fF4



RESULTS

where P is the EEG power includes theta and alphafrequency bands.
As previous studies have reported, fatigue level is positively
proportional to wide-band EEG power in the occipitalregion, including theta-, alpha-, and beta-band amplitudes,
corresponding to the O1, Oz, and O2 electrodes (Chuang et al.,2013; Huang et al., 2016; Ma et al., 2019). The level of the fatigue
parameter is defined as:
Fatigue (t) = 5 +

13PO1 (t) −PO1
POz (t) −POz
PO2 (t) −PO2
+
+
fO1
fOz
fO2

Behavioral Statistics
In the experiment, the stimuli are categorized as congruent,
with the direction of the cue matching the target, incongruent,
with the direction of the cue different from the target, or notarget, where the target is absent. To investigate the effects
caused by different conditions, the RT and accuracy of the users’
responses were recorded. The condition of the congruent stimuli
induced the shortest median RT of 634.7 ms (IQR 567.5–731.0).
The incongruent stimuli yielded a slightly longer median RTof 665.7 ms (IQR 613.7–765.9). Other than the left and rightstimulus correspondence choices, the subjects were required
to press the “up” button after they recognized the absence ofthe target, which led to the longest median RT of 868.0 mswhere P is the EEG power includes theta, alpha, and beta
frequency bands.

Frontiers in Human Neuroscience | www.frontiersin.org

PC3 (t) −PCz (t)fC

where P is the EEG power within mu rhythm (8–13 Hz), fC
is std(PC42−PCz ) for left brain activation and std(PC32−PCz ) for rightbrain activation, respectively.
Figure 3 depicts an informational flow diagram underlying
the state indicator. While utilizing a BCI accompanied by
the physiological state indicator, the acquired EEG signal wastransmitted to the computer through Bluetooth. The user data
would be recorded for 30 s to establish a baseline for removal,
also as the reference data for ASR, as previously described. The
recorded signal was then sent to a portable device by a computer
program based on C# via local area networks with a refresh rateof 1 s. Any EEG state changes of the user during BCI usage
would be converted into corresponding values of the proposedparameters based upon the mean and standard deviation of
EEG power acquired by the 40-subject dataset. The indicator on
the portable device displays the five parameters acquired fromthe connected computer on an easily accessible pentagonal plot
in near real time.

C 1

Attention (t) = 5 −

PC4 (t) −PCz (t)fCand the parameter of right brain activity is defined as:ny (t) = 5 ±

PF4 (t) −PF3 (t)fC

5December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

FIGURE 3 | System information flow. The EEG acquisition is conducted with the software in the computer through Bluetooth. The acquired signal could be applied
to a BCI system; at the same time, it is transmitted to the monitoring system running on portable devices. In the system, the signal is computed to easily assess theparameters which are displayed in a pentagon plot.

FIGURE 4 | Reaction time and accuracy of congruent, incongruent, and no-target stimulus. (A) The mean RT of the three conditions. The RT of no-target stimulus
was significantly longer than other conditions. (B) The accuracy of the three conditions. The incongruent stimulus induced the highest accuracy of 99.17% (IQR
98.75–100.00). *indicates that the difference between the two groups was statistically significant. The whiskers extend to the most extreme data points notconsidered outliers, and the outliers are plotted individually using the “+” marker symbol.

(IQR 789.5–983.0). The median accuracies of the congruent,incongruent, and no-target conditions were 98.75% (IQR 97.91–
99.17), 99.17% (IQR 98.75–100.00), and 98.54% (IQR 97.50–
99.58), respectively. The variables of RT and accuracy werecompared using Kruskal-Wallis test, as they were not normally
distributed. The p-values were adjusted for multiple comparisonsusing the Dunn and Sidák’s approach. A false discovery rate ofp < 0.05 was considered statistically significant represented by anasterisk (*) (Figure 4).

periods in the experiment for validation. To assess the attention
parameter, the whole experiment was included to obtain the
dynamics of attention throughout the four experimental sessions
and rests. The results demonstrated that attention levels during
the active sessions were significantly higher than their subsequent
rest periods, but also exhibited a decay over time (Figure 5). The
attention level during the rest periods also declined over timeand eventually stabilized by the final period. The rest periodsin the experiment were used to study the fatigue levels affectedby the long sessions during which the subjects were highly
focused. The experiment was intentionally lengthy, reachingnearly 1 h, with the intention of inducing fatigue. Consequently,
the fatigue levels increased gradually and plateaued by thethird rest period (Figure 6). In addition to the assessment
by EEG signals, self-evaluated fatigue levels were recorded for

Validation of Parameters
The five parameters within the proposed physiological state
indicator were developed using particular combinations of brain
regions and EEG frequency bands. Each parameter, respectively,
represents a distinct mental state, therefore, we utilized different

Frontiers in Human Neuroscience | www.frontiersin.org

6

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

FIGURE 5 | Attention level fluctuations during the whole experiment. This parameter was developed using theta and alpha-band power at F3, Fz, and F4. (A) The
average attention level across the four session and rest periods. (B) The attention level during the sessions decreased over time and was significantly higher thanduring the following rest periods in pairs 2 and 3. *indicates that the difference between the two groups was statistically significant. The whiskers extend to the mostextreme data points not considered outliers, and the outliers are plotted individually using the “+” marker symbol.

FIGURE 6 | Fatigue level fluctuations during the rest periods. This parameter was developed using theta, alpha and beta-band power at O1, Oz, and O2. (A) The
average fatigue level across the four rest periods. (B) The fatigue level increased from rest 1 to rest 3 and mitigated in rest 4. (C) The self-evaluated fatigue level overfour rest periods. *indicates that the difference between the two groups was statistically significant. The whiskers extend to the most extreme data points notconsidered outliers, and the outliers are plotted individually using the “+” marker symbol.

and stress parameters were compared using the repeated measure
ANOVA with post-hoc test using Dunn and Sidák’s approach. The
alpha level of 0.05 was used for statistical hypothesis testing.

comparison (Figure 6C). The stress parameter was intended torespond to the stress that the users experience across different
sessions. It illustrated that the stress levels were at their highest in
the beginning of the experiment, and then significantly decreased
over the four session periods (Figure 7). Since the participants
were instructed to perform the experiment with their righthand, hypothetically the derived right brain activity levels should
be higher than the left brain activity level during all session
periods and approximate to the left brain activity level in all rest
periods. However, the results revealed that the left and right brain
activities were close to each other, and did not appear to exhibit
significant trends (Figure 8). The levels of the parameters of eachsubject were averaged per period for statistical analysis, yielding
one average value of the physiological state level per subject in one
period (session or rest). The comparable distributions passed theKolmogorov-Smirnov test for normality. The average levels of theattention parameter between each pair of session and rest werecompared using the paired t-test. The average levels of the fatigue

Frontiers in Human Neuroscience | www.frontiersin.org

DISCUSSION
In this study, we developed a physiological state indicator
integrated with wireless EEG equipment, enabling the assessmentof physiological states for potential passive and active BCI
applications. Prior active BCI studies have documented theeffects of physiological state changes on BCI performance(Riccio et al., 2013; Myrden and Chau, 2015; Talukdar et al.,2019; Zhang et al., 2020). During long-term BCI performance
users’ physiological states change spontaneously, which lead tounsatisfactory outcomes. However, passive BCIs may facilitate
the use of traditional, active BCIs by providing an estimationof the underlying user state variations (Millán et al., 2010).

7

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

FIGURE 7 | Stress level fluctuations during the active sessions. This parameter was developed using log-transformed alpha-band power at F3 and F4. (A) The
average stress level across the four session periods. (B) The stress level significantly decreased over time. *indicates that the difference between the two groups was
statistically significant. The whiskers extend to the most extreme data points not considered outliers, and the outliers are plotted individually using the “+” marker
symbol.

Parameters that were reported to affect BCI performance,
including attention, fatigue, and stress were adopted in the
proposed system, and the left and right brain activity parameters
were designed specifically for MI-based BCI, yielding a collectionof five parameters for BCI assessment. The experiment basedon ANT not only induced attentional alterations, the prolonged
duration also caused fatigue in the subjects and the moderatedifficulty allowed subjects to adapt, producing transitions
in stress levels.
To induce attention differences between active sessions andrest periods, but not produce excessive fatigue over the 1h duration, we designed an experiment that was based ontarget detection and ANT, somewhat modified to be closer
to real-world scenarios. With this experiment, the relationshipamong these three conditions and the physiological state changesof the users can be evaluated. Generally, incongruent conditions
introduce “conflict” that requires longer information processing
times, resulting in longer RTs. In our experiment, the target
in the no-target condition is absent, hypothetically causing
even longer RTs than the incongruent conditions. The notarget condition demonstrated a lower accuracy of 98.54% anda significantly longer median RT of 868.0 ms, compared tothe congruent and incongruent conditions. The conflict in theincongruent condition did bring about slightly longer medianRT, however, the had the highest accuracy among the conditions.
This may have been caused by the relatively simple experimental
design compared to similar cognitive experiments, such as theclassical ANT protocol. As the result, the median accuracies
for all conditions exceeded 98%. Nevertheless, the experimental
procedure provided the necessary factors to elicit physiologicalstate changes, especially for the adopted parameters. Although
the participants were instructed to respond as soon as possible,
the maximal possible reaction time before a trial was considered
as failed was set at 2 s because the experiment included theno-target condition. This was assumed to require a longer
reaction time, resulting a ceiling effect for performance. However,
several participants performed relatively poorly, and there
were significant differences between congruent/incongruent andincongruent/no-target conditions, indicating that the three
conditions could elicit the variations in performance within
participants, despite the ceiling effect.
A large body of prior literature has indicated thatneurophysiological signals contain information related tophysiological state changes, which were acquired via EEG in
our experiment (Lin et al., 2010; Xia et al., 2018; Asif et al.,2019; Monteiro et al., 2019; Xu et al., 2019; Bajaj et al., 2020).
Some of the research in recent years has applied machine

FIGURE 8 | Left and right brain activity level fluctuations during the wholeexperiment. The parameters were developed using alpha-band power ofC4–Cz and C3–Cz. It shows that the left and right brain activities areapproximate to each other, and neither ascended nor descended significantly.

Frontiers in Human Neuroscience | www.frontiersin.org

8

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

performance sessions, progressively ascended and peaked in thethird period, indicating the subjects were experiencing maximal
fatigue by that time. Correspondingly, the self-evaluated fatigue
level in period 3 reached an average of 7.25 points out of 10, which
was also the highest among the four rest periods. The fatigue
level demonstrated a similar trend to the cluster of “strugglers”
proposed by Karrer et al. (2004), which presented a high level offatigue during a simulated driving test and subsequent reductionof fatigue toward the end of the driving test. We inferred thatthe perception of the impending end of the experiment resulted
in a reduction of fatigue level, possibly indicating a mental
relaxation in the subject.
Stress is a parameter that was not specifically elicited in
the experimental design, instead, the spontaneous responsesof the subjects during four sessions were recorded. Comparedwith the EEG characteristics that correlate to attention andfatigue, frontal alpha asymmetry is a relatively well-establishedelectrophysiological feature for stress assessment. In a study
proposed by Arpaia et al. (2020), frontal alpha asymmetry wasapplied to machine learning classifiers and achieved more than
a 90% accuracy in classifying 2-s EEG epochs using a wearable
EEG instrument. Similarly, the F4/F3 asymmetry was adopted inthis study to assess the stress that the subject experienced in the
successive performance of the cognitive task. As the experiment
proceeded, subjects’ stress levels exhibited a descending trend,
indicating that the subjects felt less anxious toward the lastsession, according to gradually increased F4-F3 asymmetryscores. Generally speaking, situations that are unpredictable,
ambiguous, or unfamiliar are more likely to elicit stress (Michie
and medicine, 2002). In our experiment, which did not include
induced stressors, the unfamiliarity of the first session, where thesubjects experienced the highest level of stress, was posited as themain factor of stress; this elevated stress level was mitigated as the
subjects habituated the experiment.
One limitation of the proposed indicator is that the advanced
algorithms of MI-BCIs were not included in the parameters of left
and right activity. The MI-BCI algorithms have been extensively
studied to distinguish the EEG patterns representative of themental imagination of different movements. Simply measuring
the mu rhythm power of the left and right motor cortices might
not be sufficient to differentiate motor executions based on the
results of the left and right brain activity parameters. Estimating
the subject’s event-related desynchronization/synchronization isfundamental to MI, however, additional algorithms utilizing theexisting parameters are required for accurate MI-BCI control.
In this study, the features for the five parameters wereadopted based on online signal processing using the findings ofprevious studies. The experiment intended to elicit the variationsin the parameters of the subjects, specify the EEG power
ranges of the parameters, and thus develop a real-time multiparameter physiological state indicator with proper scaling. In
the experimental design, the general attention and fatigue wereassessed, while the stress, left activity, and right activity werestudied through the spontaneous responses of the participants.
However, to investigate each of the parameters throughout,sophisticated experimental design for each of the parameters
would benefit the investigation of the physiological state changes.
Furthermore, the parameters have intricate interactions andlearning or deep learning methods to identify different stagesof physiological states (Jebelli et al., 2019; Ma et al., 2019;
Khessiba et al., 2021). However, the high computational powerthat deep learning demands does not necessarily provide betterclassification performance due to inter-subject variabilities.
For example, in studies that investigated human fatigue during
driving simulation experiments, one experiment deployed an
EEG classifier based on convolutional neural networks (CNN),
which demonstrated an average accuracy of 92.68% in theintra-subject test (Zeng et al., 2018). By contrast, it demonstrated
a reduced performance of 84.38% in an inter-subject binaryclassification, falling beneath the performance reported by
Ko et al. (2015), which employed a linear regression model
on a 1–30 Hz EEG power spectrum array. A review study byMonteiro et al. (2019) also demonstrated that linear models likefisher linear discriminant analysis (FLDA) could outperform
deep belief network (DBN) while using PSD to assess mental
fatigue in cross-subject classification. In consideration of thecomputational capacity necessary to process five parametersin real-time on portable devices, we implemented models that
determine physiological state levels using linear transforms of
EEG power amplitudes. Additionally, adaptive BCIs have been
proposed to deal with the ubiquitous non-stationarities in EEG
signals (Shenoy et al., 2006). Linear transformation models likeour indicator benefit the such adaptive models by providing
consecutive levels that correspond to user physiological statechanges rather than binary classification (Krol et al., 2016;
Zhang et al., 2018b).
To study and validate the indicator that was developed basedon cognitive state changes, a prolonged experiment incorporating
EEG signal recording was administered. In general, long-term
task performance results in increases in fatigue and reductions ingoal-directed attention (Boksem et al., 2005). In the validation of
the attention parameter, the attention levels in the active sessions
surpassed those in the subsequent rest periods, and fluctuations
indicated that the users’ mental engagements dropped in theresting periods, then ascended right after the task onset. In
addition, attention level exhibited a decay over time in bothactive sessions and rest periods, suggesting that attention reduced
throughout the experiment. In other words, the subject’s thetaand alpha band amplitudes in the frontal region increased overtime under both conditions. However, it is generally acceptedthat attention is categorized into a variety of types based upon
functional aspects. Sustained attention represents the attentional
functionalities that determine the cognitive capacity and efficacy
of selective and divided attention (Sarter et al., 2001), which isrepresented by the attention parameter, and reflects the vigilance,of an individual during long-term BCI usages.
The cognitive demands during the experiment cause mental
fatigue, which is assumed to increase over time. Trejo et al.
(2015) demonstrated that the EEG signs of developing mental
fatigue are present 15–30 min after task onset. In our paradigm,
physiological state changes other than fatigue could confoundthe measurement of the fatigue parameter. Therefore, the fatigue
was assessed and validated in the rest periods, as opposed tothe attention parameter which was assessed in both active andresting periods. The results showed that the estimated fatigue
levels during the rest periods, which followed 12-min cognitive
Frontiers in Human Neuroscience | www.frontiersin.org

9

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

neurophysiological information for parameter calibration. As
such, the dataset would be expanded and bring about moregeneralized results.
With the increasing quality of commercial EEG devices andadvancements in artifact removal techniques, using BCIs inparallel with physiological state assessment tools in daily living
has become more realistic. The validation results suggest that theneurophysiological manifestations of state changes are consistentacross a variety of cognitive processes, setting the stage forpractical applications of mental state indicators in passive BCIs.
The proposed multi-parameter indicator has since been applied
to a variety of fields in real-world environments to strengthen
the practicability of the system. In 2018, a study adopted thephysiological state indicator to aid professional and amateur Go
players in competing with smart machines (Lee et al., 2020;
Figure 9A). The wearable BCI system benefits machine-humanco-learning as it can be integrated with training methods orneurofeedback to monitor and regulate Go players’ physiologicalstates, helping them to maintain their attention and overcome
the competitive pressure experienced in the game. Later, thesystem was extended to the educational domain interfacingwith an intelligent agent for robotic edutainment in 2019 (Leeet al., 2019; Figure 9B). The experimental results showed thatthrough playing games, the system improved the interest andperformance of students in mathematical and language learning.
Currently, the proposed indicator is also being applied to a gamebased assessment for cognitive function in Kaohsiung Medical
Hospital (Figure 9C) and a training paradigm for military
soldiers using a first person shooting scenario in National Chung
Shan Institute of Science and Technology (Figure 9D). To sumup, by estimating physiological state changes precisely using

are not independent mechanisms. For instance, the impact of
fatigue on attention (Boksem et al., 2005; Faber et al., 2012), theinteractions between attention and stress (Chajut and Algom,
2003; Zhang et al., 2020), and even the effects of fatigue andstress on MI (Talukdar et al., 2019; Schlatter et al., 2020). Such
usage of a general indicator doesn’t warrant a detailed intraparametric analysis, though that could be an interesting direction
for generating a future indicator.
Mental workload is a parameter we would like to investigate
and incorporate into the system in the future. Despite different
causes, the physiological correlates of mental workload andstress manifest in rather similar ways (Parent et al., 2019).
Likewise, research by Käthner et al. (2014) reported that ahigh mental workload, as well as fatigue, affects event-related
potential (ERP) -based BCI performance. The results of ourprevious work reported in Zhang et al. (2020) also indicatedthat stress-dependent effects are similar to the outcome of
fatigue, leading to a potential conclusion that stress andfatigue both impact attention capacity and interact to affect
BCI performance.
All in all, future experiments should be designed and managed
carefully as the causes and consequences of the physiologicalstates including mental workload, attention, stress, and fatigue
can be easily confounded. It is worth mentioning that currently
the values of the parameters in the indicator are derived fromthe 40-subject EEG data. The indicator software on portabledevices can instantly reflect the physiological states of theuser based on the existing dataset. In addition, with accurate
labeling of corresponding physiological states, parameters infuture indicator can benefit from the acquired EEG signals
during potential experiments, as the collected data provide more

FIGURE 9 | Real-world applications of the proposed physiological states indicator. (A) A Go player using the system while playing Go with smart machines formachine-human co-learning. (B) Intelligent agent for robotic edutainment in mathematical and language learning. (C) Game-based assessment of cognitive
functions. (D) Training of soldiers using first person shooting scenario.

Frontiers in Human Neuroscience | www.frontiersin.org

10

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

artifact-free EEG signals, the indicator can advance applicationsin interdisciplinary fields.

Tung University. The patients/participants provided their written
informed consent to participate in this study.

CONCLUSIONAUTHOR CONTRIBUTIONS

To our knowledge, this is the first study presenting a systemthat indicates easily accessible metrics regarding multiple
physiological states via wireless EEG in real-time. While BCI
has been demonstrating great potential for controlling external
devices with our minds, the physiological state changes induced
by excessive stimulus under experimental settings or realconditions not only impact task performance and BCI accuracy,
but are also interconnected with physical and mental health.
Therefore, the concept of a passive BCI indicator is proposed
to address such conditions by identifying the users’ spontaneous
brain activities in real time. In this study, the parametersintegrated into the state indicator include attention, fatigue,
stress, and left and right brain activities, which were derived
according to the cognitive state changes gathered from EEG
signal processing. Furthermore, a prolonged experiment based ontarget detection was developed to validate the proposed indicator.
The experimental results demonstrated a feasible outcome that
corresponded to different stages during the experiment. The
proposed physiological state indicator is applicable to online
BCI applications, simultaneously allowing the state of the user
to be assessed, along with providing a means of continuously
evaluating the associated control signal quality.

L-WK initialized the research idea, designed the experimental
paradigm, and advised the methods of EEG data analysis.
L-WK, YC, and CH performed the data analysis of EEG signals
and drafted the manuscript. CH and B-YT performed theEEG data collection and provided advice on the manuscript
revisions. L-WK, YC, and B-YT developed the software of thephysiological state indicator. All authors agreed on the final
manuscript submission.

FUNDING
This work was supported by the Higher Education Sprout
Project of the National Yang Ming Chiao Tung University
and Ministry of Education (MOE), Taiwan, under Contracts
110W153, and in part by the Center for Intelligent Drug
Systems and Smart Bio-Devices (IDS2B) from the Featured
Areas Research Center Program within the framework of
the Higher Education Sprout Project by MOE in Taiwan,
and in part by the Artificial Intelligence to Precision Health:Integrating Dynamic Physiological Signals and EMR tobuild a Medical Digital Twins Platform sponsored by the
Taiwan Ministry of Science and Technology under Grants
MOST 110-2321-B-010-005-, MOST 110-2321-B-A49-003-,
MOST 110-2622-E-A49-009-, and MOST 108-2221-E-009-045MY3.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will bemade available by the authors, without undue reservation.

ACKNOWLEDGMENTS
We would like to thank Dr. Cory Stevenson for editing assistance,
and the Information and Communications Research Division,
National Chung-Shan Institute of Science and Technology forsupporting the research.

ETHICS STATEMENT
The studies involving human participants were reviewed andapproved by the Institutional Review Boards of National Chaio

REFERENCES

Chajut, E., and Algom, D. J. (2003). Selective attention improves under stress:
implications for theories of social cognition. J. Pers. Soc. Psychol. 85:231. doi:
10.1037/0022-3514.85.2.231
Chang, C. Y., Hsu, S. H., Pion-Tonachini, L., and Jung, T. P. (2018).
“Evaluation of artifact subspace reconstruction for automatic EEG
artifact removal,” in Proceeding of the 2018 40th Annual International
Conference of the IEEE Engineering in Medicine and Biology Society (EMBC),
1242–1245.
Chang, C.-Y., Hsu, S.-H., Pion-Tonachini, L., and Jung, T.-P. (2019). Evaluationof artifact subspace reconstruction for automatic artifact components removal
in multi-channel EEG recordings. IEEE Trans. Biomed. Eng. 67, 1114–1121.
doi: 10.1109/TBME.2019.2930186
Chuang, C.-H., Ko, L.-W., Lin, Y.-P., Jung, T.-P., and Lin, C.-T. (2013).
Independent component ensemble of EEG for brain–computer interface. IEEE
Trans. Neural Syst. Rehabil. Eng. 22, 230–238. doi: 10.1109/TNSRE.2013.
2293139
Clark, M. M., Warren, B. A., Hagen, P. T., Johnson, B. D., Jenkins, S. M.,
Werneburg, B. L., et al. (2011). Stress level, health behaviors, and quality of

Arpaia, P., Moccaldi, N., Prevete, R., Sannino, I., and Tedesco, A. (2020).
A wearable EEG instrument for real-time frontal asymmetry monitoring
in worker stress analysis. IEEE Trans. Instrument. Measure. 69,
8335–8343.
Asif, A., Majid, M., and Anwar, S. M. (2019). Human stress classification using
EEG signals in response to music tracks. Comput. Biol. Med. 107, 182–196.
doi: 10.1016/j.compbiomed.2019.02.015
Bajaj, V., Taran, S., Khare, S. K., and Sengur, A. (2020). Feature extraction method
for classification of alertness and drowsiness states EEG signals. Appl. Acoustics
163:107224. doi: 10.1109/IEMBS.2009.5334491
Boksem, M. A., Meijman, T. F., and Lorist, M. M. (2005). Effects of mental
fatigue on attention: an ERP study. Brain Res. Cogn. Brain Res. 25, 107–116.
doi: 10.1016/j.cogbrainres.2005.04.011
Brás, S., Ferreira, J. H., Soares, S. C., and Pinho, A. J. (2018). Biometric and emotion
identification: an ECG compression based method. J. Conf. Endorsement 9:467.
doi: 10.3389/fpsyg.2018.00467

Frontiers in Human Neuroscience | www.frontiersin.org

11

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

life in employees joining a wellness center. Am. J. Health Promotion 26, 21–25.
doi: 10.4278/ajhp.090821-QUAN-272
Doesburg, S. M., Bedo, N., and Ward, L. M. (2016). Top-down alpha oscillatory
network interactions during visuospatial attention orienting. IEEE Trans.
Biomed. 132, 512–519. doi: 10.1016/j.neuroimage.2016.02.076
Faber, L. G., Maurits, N. M., and Lorist, M. M. (2012). Mental fatigue affects visual
selective attention. PLoS One 7:e48073. doi: 10.1371/journal.pone.0048073
Fan, J., McCandliss, B. D., Sommer, T., Raz, A., and Posner, M. I. (2002). Testing
the efficiency and independence of attentional networks. J. Cogn. Neurosci. 14,
340–347. doi: 10.1162/089892902317361886
Fernandes, A., Helawar, R., Lokesh, R., Tari, T., and Shahapurkar, A. V. (2014).
“Determination of stress using blood pressure and galvanic skin response,”
in Proceeding of the 2014 International Conference on Communication and
Network Technologies, (IEEE), 165–168.
Grassmann, M., Vlemincx, E., von Leupoldt, A., Mittelstädt, J. M., and Van den
Bergh, O. (2016). Respiratory changes in response to cognitive load: a systematic
review. Neural Plasticity 2016, 8146809. doi: 10.1155/2016/8146809
Harmon-Jones, E., and Gable, P. A. (2018). On the role of asymmetric frontal
cortical activity in approach and withdrawal motivation: an updated review ofthe evidence. Psychophysiology 55:e12879. doi: 10.1111/psyp.12879
He, C., Chikara, R. K., Yeh, C.-L., and Ko, L.-W. (2021). Neural dynamics oftarget detection via wireless EEG in embodied cognition. Sensors 21:5213. doi:
10.3390/s21155213
Hsu, Y. L., Wang, J. S., Chiang, W. C., and Hung, C. H. (2017). Automatic ECGbased emotion recognition in music listening. IEEE Trans. Affect. Comput. 11,
85–99.
Huang, K.-C., Huang, T.-Y., Chuang, C.-H., King, J.-T., Wang, Y.-K., Lin, C.-T.,
et al. (2016). An EEG-based fatigue detection and mitigation system. Int. J.
Neural Syst. 26:1650018. doi: 10.1142/S0129065716500180
Huang, S., Li, J., Zhang, P., and Zhang, W. J. I. (2018). Detection of mental
fatigue state with wearable ECG devices. Int. J. Med. Inform. 119, 39–46. doi:
10.1016/j.ijmedinf.2018.08.010
Jason, L. A., Evans, M., Brown, M., and Porter, N. (2010). What is fatigue?
pathological and nonpathological fatigue. PM R. 2, 327–331. doi: 10.1016/j.
pmrj.2010.03.028
Jebelli, H., Khalili, M. M., and Lee, S. (2019). “Mobile EEG-based workers’ stressrecognition by applying deep neural network,” in Proceedings of the 35th CIB
W78 2018 Conference: IT in Design, Construction, and Management, (Springer),
173–180.
Jung, T.-P., Makeig, S., Stensmo, M., and Sejnowski, T. (1997). Estimating alertness
from the EEG power spectrum. IEEE Trans. Biomed. Eng. 44, 60–69. doi: 10.
1109/10.553713
Karrer, K., Vöhringer-Kuhnt, T., Baumgarten, T., and Briest, S. (2004). “The role ofindividual differences in driver fatigue prediction,” in Proceeding of the Third
International Conference on Traffic and Transport Psychology, (Nottingham:
Citeseer), 5–9.
Käthner, I., Wriessnegger, S. C., Müller-Putz, G. R., Kübler, A., and Halder, S.
(2014). Effects of mental workload and fatigue on the P300, alpha and theta
band power during operation of an ERP (P300) brain–computer interface. Biol.
Psychol. 102, 118–129. doi: 10.1016/j.biopsycho.2014.07.014
Kehri, V., Ingle, R., Patil, S., and Awale, R. (2019). “Analysis of facial emg signal
for emotion recognition using wavelet packet transform and svm,” in MachineIntelligence and Signal Analysis, eds M. Tanveer and R. Pachori (Springer),247–257.
Khessiba, S., Blaiech, A. G., Ben Khalifa, K., Ben Abdallah, A., and Bedoui, M. H.
(2021). Innovative deep learning models for EEG-based vigilance detection.
Neural Comput. Appl. 33, 6921–6937.
Ko, L.-W., Chang, Y., Wu, P.-L., Tzou, H.-A., Chen, S.-F., Tang, S.-C., et al. (2019).
Development of a smart helmet for strategical BCI applications. Sensors (Basel)
19:1867. doi: 10.3390/s19081867
Ko, L.-W., Lai, W.-K., Liang, W.-G., Chuang, C.-H., Lu, S.-W., Lu, Y.-C.,
et al. (2015). “Single channel wireless EEG device for real-time fatigue level
detection,” in Proceeding of the 2015 International Joint Conference on Neural
Networks (IJCNN), (IEEE), 1–5.
Krol, L. R., Freytag, S.-C., Fleck, M., Gramann, K., and Zander, T. O.
(2016). “A task-independent workload classifier for neuroadaptive technology:
preliminary data,” in Proceeding of the 2016 IEEE International Conference on
Systems, Man, and Cybernetics (SMC), (IEEE), 003171–003174.

Frontiers in Human Neuroscience | www.frontiersin.org

Krupinski, E. A., MacKinnon, L., and Reiner, B. I. (2015). “Feasibility ofusing a biowatch to monitor GSR as a measure of radiologists’ stress andfatigue,” in Proceeding of the Medical Imaging 2015: Image Perception, Observer
Performance, and Technology Assessment: International Society for Optics andPhotonics, 941613.
Kurniawan, H., Maslov, A. V., and Pechenizkiy, M. (2016). “Stress detection
from speech and galvanic skin response signals,” in Proceedings of the 26th
IEEE International Symposium on Computer-Based Medical Systems, (IEEE),
209–214.
Lampert, R. (2015). ECG signatures of psychological stress. J. Electrocardiol. 48,
1000–1005. doi: 10.1016/j.jelectrocard.2015.08.005
Lazarus, R. S. (2006). Stress and Emotion: A New Synthesis. New York: Springer
Publishing Company.
Lee, C.-S., Wang, M.-H., Ko, L.-W., Lee, Y.-H., Ohashi, H., Kubota, N., et al.
(2020). Human intelligence meets smart machine: a special event at the IEEE
international conference on systems, man, and cybernetics 2018. IEEE Syst.
Man Cybernetics Magazine 6, 23–31.
Lee, C.-S., Wang, M.-H., Tsai, Y.-L., Ko, L.-W., Tsai, B.-Y., Hung, P.-H., et al.
(2019). Intelligent agent for real-world applications on robotic edutainment andhumanized co-learning. J. Ambient Intelli. Hum. Comput. 11, 1–19.
Lee, Y.-C., Lin, W.-C., Cherng, F.-Y., and Ko, L.-W. (2015). A visual attention
monitor based on steady-state visual evoked potential. IEEE Trans. Neural Syst.
Rehabil. Eng. 24, 399–408. doi: 10.1109/TNSRE.2015.2501378
Lin, C.-T., Lin, F.-C., Chen, S.-A., Lu, S.-W., Chen, T.-C., and Ko, L.-W.
(2010). EEG-based brain-computer interface for smart living environment
auto-adjustment. J. Med. Biol. Eng. 30, 237–245. doi: 10.5405/jmbe.30.4.07
Lin, C.-T., Wu, R.-C., Jung, T.-P., Liang, S.-F., and Huang, T.-Y. (2005). Estimating
driving performance based on EEG spectrum analysis. J. Adv. Signal Proc. 2005,
1–10.
Liu, M., Fan, D., Zhang, X., and Gong, X. (2016). “Human emotion recognition
based on galvanic skin response signal feature selection and svm,” in Proceedingof the 2016 International Conference on Smart City and Systems Engineering(ICSCSE), (IEEE), 157–160.
Lock, A., Bonetti, D., and Campbell, A. (2018). The psychological and physiologicalhealth effects of fatigue. Occup. Med. 68, 502–511. doi: 10.1093/occmed/kqy109
Luijcks, R., Hermens, H. J., Bodar, L., Vossen, C. J., Os, J., and Lousberg, R. (2014).
Experimentally induced stress validated by EMG activity. PLoS One 9:e95215.
doi: 10.1371/journal.pone.0095215
Ma, Y., Chen, B., Li, R., Wang, C., Wang, J., She, Q., et al. (2019). Driving
fatigue detection from EEG using a modified PCANet method. Comput. Intelli.
Neurosci. 2019, 1–19. doi: 10.1155/2019/4721863
McEwen, B. S., and Sapolsky, R. M. (1995). Stress and cognitive function. Curr.
Opin. Neurobiol. 5, 205–216.
Meyer, T., Quaedflieg, C. W., Giesbrecht, T., Meijer, E. H., Abiad, S., and Smeets,
T. (2014). Frontal EEG asymmetry as predictor of physiological responses
to aversive memories. Psychophysiology 51, 853–865. doi: 10.1111/psyp.1
2230
Michie, S., and medicine, E. (2002). Causes and management of stress at work.
Occup. Environ. Med. 59, 67–72. doi: 10.1136/oem.59.1.67
Millán, J. D. R., Rupp, R., Mueller-Putz, G., Murray-Smith, R., Giugliemma, C.,
Tangermann, M., et al. (2010). Combining brain–computer interfaces andassistive technologies: state-of-the-art and challenges. Front. Neurosci. 4:161.
doi: 10.3389/fnins.2010.00161
Monteiro, T. G., Skourup, C., and Zhang, H. (2019). Using EEG for mental fatigue
assessment: a comprehensive look into the current state of the art. IEEE Trans.
Hum. Mach. Syst. 49, 599–610. doi: 10.1109/THMS.2019.2938156
Myrden, A., and Chau, T. (2015). Effects of user mental state on EEGBCI performance. Front. Hum. Neurosci. 9:308. doi: 10.3389/fnhum.2015.0
0308
Neuper, C., and Pfurtscheller, G. (2009). “Neurofeedback training for BCI control,”in Brain-Computer Interfaces. The Frontiers Collection, eds B. Graimann, G.
Pfurtscheller, and B. Allison (Berlin, Heidelberg: Springer), 65–78.
Parent, M., Peysakhovich, V., Mandrick, K., Tremblay, S., and Causse, M. (2019).
The diagnosticity of psychophysiological signatures: can we disentangle mental
workload from acute stress with ECG and fNIRS? Int. J. Psychophysiol. 146,
139–147. doi: 10.1016/j.ijpsycho.2019.09.005
Pearlin, L. I., Schieman, S., Fazio, E. M., Meersman, S. C., and Behavior, S.
(2005). Stress, health, and the life course: some conceptual perspectives.

12

December 2021 | Volume 15 | Article 785562

Chang et al.

Multi-Parameter Physiological State Monitoring

J. Health Soc. Behav. 46, 205–219. doi: 10.1177/0022146505046
00206
Pfurtscheller, G. J. E., and Neurophysiology, C. (1997). EEG event-related
desynchronization (ERD) and synchronization (ERS). Clin. Neurophysiol. 1:26.
Pfurtscheller, G., and Neuper, C. (1994). Event-related synchronization of murhythm in the EEG over the cortical hand area in man. Neurosci. Lett. 174,
93–96. doi: 10.1016/0304-3940(94)90127-9
Pfurtscheller, G., Brunner, C., Schlögl, A., and Da Silva, F. L. (2006). Mu rhythm(de) synchronization and EEG single-trial classification of different motor
imagery tasks. Neuroimage 31, 153–159. doi: 10.1016/j.neuroimage.2005.12.003
Pfurtscheller, G., Stancak, A. Jr., and Neuper, C. (1996). Event-related
synchronization (ERS) in the alpha band—an electrophysiological correlate ofcortical idling: a review. Int. J. Psychophysiol. 24, 39–46. doi: 10.1016/s01678760(96)00066-9
Pope, A. T., Bogart, E. H., and Bartolome, D. S. (1995). Biocybernetic systemevaluates indices of operator engagement in automated task. Biol. Psychol. 40,
187–195. doi: 10.1016/0301-0511(95)05116-3
Posner, M. I., and Petersen, S. E. (1990). The attention system of the human brain.
Ann. Rev. Neurosci. 13, 25–42.
Quaedflieg, C., Meyer, T., Smulders, F., and Smeets, T. (2015). The functional roleof individual-alpha based frontal asymmetry in stress responding. Biol. Psychol.
104, 75–81. doi: 10.1016/j.biopsycho.2014.11.014
Quilodran, R., Rothe, M., and Procyk, E. (2008). Behavioral shifts and action
valuation in the anterior cingulate cortex. Neuron 57, 314–325. doi: 10.1016/
j.neuron.2007.11.031
Ramoser, H., Muller-Gerking, J., and Pfurtscheller, G. (2000). Optimal spatial
filtering of single trial EEG during imagined hand movement. IEEE Trans.
Rehabil. Eng. 8, 441–446. doi: 10.1109/86.895946
Raz, A. (2004). Anatomy of attentional networks. Anat. Rec. B. New Anat. 281,
21–36. doi: 10.1002/ar.b.20035
Reznik, S. J., and Allen, J. (2018). Frontal asymmetry as a mediator and moderatorof emotion: an updated review. Psychophysiology 55:e12965. doi: 10.1111/psyp.
12965
Riccio, A., Simione, L., Schettini, F., Pizzimenti, A., Inghilleri, M., Olivetti
Belardinelli, M., et al. (2013). Attention and P300-based BCI performance inpeople with amyotrophic lateral sclerosis. Front. Hum. Neurosci. 7:732. doi:
10.3389/fnhum.2013.00732
Roets, B., and Christiaens, J. (2019). Shift work, fatigue, and human error: anempirical analysis of railway traffic control. J. Trans. Safety Security 11, 207–224.
Sapolsky, R. M. (2015). Stress and the brain: individual variability and the invertedU. Nat. Neurosci. 18:1344. doi: 10.1038/nn.4109
Sarter, M., Givens, B., and Bruno, J. P. (2001). The cognitive neuroscience ofsustained attention: where top-down meets bottom-up. Brain Res. Brain Res.
Rev. 35, 146–160. doi: 10.1016/s0165-0173(01)00044-3
Schlatter, S., Guillot, A., Faes, C., Saruco, E., Collet, C., Di Rienzo, F., et al. (2020).
Acute stress affects implicit but not explicit motor imagery: a pilot study. Int. J.
Psychophysiol. 152, 62–71. doi: 10.1016/j.ijpsycho.2020.04.011
Shenoy, P., Krauledat, M., Blankertz, B., Rao, R. P., and Müller, K.-R. (2006).
Towards adaptive classification for BCI. J. Neural. Eng. 3:R13. doi: 10.1088/
1741-2560/3/1/R02
Szafir, D., and Mutlu, B. (2012). “Pay attention! designing adaptive agentsthat monitor and improve user engagement,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 11–20.
Talukdar, U., Hazarika, S. M., and Gan, J. Q. (2019). Motor imagery and mental
fatigue: inter-relationship and EEG based estimation. J. Comput. Neurosci. 46,
55–76. doi: 10.1007/s10827-018-0701-0
Tatum, W. O. (2014). Ellen r. grass lecture: extraordinary eeg. Neurodiagn J. 54,
3–21.
Thatcher, R. W. (1998). Normative EEG databases and EEG biofeedback. Databases
Neurother. 2, 8–39.
Trejo, L. J., Kubitz, K., Rosipal, R., Kochavi, R. L., and Montgomery, L. D. (2015).
EEG-based estimation and classification of mental fatigue. Psychology 6:572.
Van Cutsem, J., Marcora, S., De Pauw, K., Bailey, S., Meeusen, R., andRoelands, B. (2017). The effects of mental fatigue on physical performance:

Frontiers in Human Neuroscience | www.frontiersin.org

a systematic review. Sports Med. 47, 1569–1588. doi: 10.1007/s40279-016-0
672-0
Villarejo, M. V., Zapirain, B. G., and Zorrilla, A. M. (2012). A stress sensor basedon galvanic skin response (GSR) controlled by ZigBee. Sensors 12, 6075–6101.
doi: 10.3390/s120506075
Virtanen, M., Honkonen, T., Kivimäki, M., Ahola, K., Vahtera, J., Aromaa, A.,et al. (2007). Work stress, mental health and antidepressant medication findings
from the health 2000 study. J. Affect. Dis. 98, 189–197. doi: 10.1016/j.jad.2006.
05.034
Widjaja, D., Orini, M., Vlemincx, E., and Van Huffel, S. (2013). Cardiorespiratory
dynamic response to mental stress: a multivariate time-frequency analysis.
Comput. Math Methods Med. 2013:451857. doi: 10.1155/2013/451857
Wijsman, J., Grundlehner, B., Penders, J., and Hermens, H. (2013). Trapezius
muscle EMG as predictor of mental stress. ACM Trans. Embed. Comput. Syst.
12, 1–20.
Xia, L., Malik, A. S., and Subhani, A. R. (2018). A physiological signal-based
method for early mental-stress detection. Biomed. Signal Proc. Control 46,
18–32.
Xu, X., Gu, H., Yan, S., Pang, G., and Gui, G. (2019). Fatigue EEG feature
extraction based on tasks with different physiological states for ubiquitous edge
computing. IEEE Access 7, 73057–73064.
Xuan, B., Mackie, M.-A., Spagna, A., Wu, T., Tian, Y., Hof, P. R., et al. (2016).
The activation of interactive attentional networks. Neuroimage 129, 308–319.
doi: 10.1016/j.neuroimage.2016.01.017
Zander, T. O., and Kothe, C. (2011). Towards passive brain–computer interfaces:
applying brain–computer interface technology to human–machine systems ingeneral. J. Neural Eng. 8:025005. doi: 10.1088/1741-2560/8/2/025005
Zeng, H., Yang, C., Dai, G., Qin, F., Zhang, J., and Kong, W. (2018). EEGclassification of driver mental states by deep learning. Cogn. Neurodyn. 12,
597–606.
Zhang, H. Y., Stevenson, C. E., Jung, T. P., and Ko, L. W. (2020). Stress-induced
effects in resting EEG spectra predict the performance of SSVEP-based BCI.
IEEE Trans. Neural Syst. Rehabil. Eng. 28, 1771–1780. doi: 10.1109/TNSRE.
2020.3005771
Zhang, T., Liu, T., Li, F., Li, M., Liu, D., Zhang, R., et al. (2016). Structuraland functional correlates of motor imagery BCI performance: insights fromthe patterns of fronto-parietal attention network. Neuroimage 134, 475–485.
doi: 10.1016/j.neuroimage.2016.04.030
Zhang, X., Bachmann, P., Schilling, T. M., Naumann, E., Schächinger, H., andLarra, M. F. (2018a). Emotional stress regulation: the role of relative frontal
alpha asymmetry in shaping the stress response. Biol. Psychol. 138, 231–239.
doi: 10.1016/j.biopsycho.2018.08.007
Zhang, X., Krol, L. R., and Zander, T. O. (2018b). “Towards task-independent
workload classification: shifting from binary to continuous classification,” inProceeding of the 2018 IEEE International Conference on Systems, Man, andCybernetics (SMC), (IEEE), 556–561.
Conflict of Interest: The authors declare that the research was conducted in theabsence of any commercial or financial relationships that could be construed as apotential conflict of interest.
Publisher’s Note: All claims expressed in this article are solely those of the authorsand do not necessarily represent those of their affiliated organizations, or those of
the publisher, the editors and the reviewers. Any product that may be evaluated inthis article, or claim that may be made by its manufacturer, is not guaranteed or
endorsed by the publisher.
Copyright © 2021 Chang, He, Tsai and Ko. This is an open-access article distributed
under the terms of the Creative Commons Attribution License (CC BY). The
use, distribution or reproduction in other forums is permitted, provided theoriginal author(s) and the copyright owner(s) are credited and that the original
publication in this journal is cited, in accordance with accepted academic practice.
No use, distribution or reproduction is permitted which does not comply withthese terms.

13

December 2021 | Volume 15 | Article 785562