# Exploring EOG markers of fatigue during motor imagery BCI use

Pauline Dreyer, Aline Roc, David Trocellier, Marc Welter, Raphaëlle N Roy, Fabien Lotte

<figure><img src="image_1.png" /></figure>

## To cite this version:

Pauline Dreyer, Aline Roc, David Trocellier, Marc Welter, Raphaëlle N Roy, et al. Exploring EOG markers of fatigue during motor imagery BCI use. INE - 9th Graz BCI Conference 2024, Sep 2024, Graz, Austria. [hal-04706552](https://hal.science/hal-04706552v1)

HAL Id: hal-04706552

https://hal.science/hal-04706552v1

Submitted on 23 Sep 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# EXPLORING EOG MARKERS OF FATIGUE DURING MOTOR IMAGERY BCI USE

Pauline Dreyer<fnref n="1" />, ², Aline Roc<fnref n="1" />, David Trocellier<fnref n="1" />, ², Marc Welter<fnref n="1" />, ², Raphaëlle N. Roy³, Fabien Lotte<fnref n="1" />, ²

¹Inria Center at the University of Bordeaux, Talence, 33405, France

²LaBRI (Univ. Bordeaux / CNRS/ Bordeaux INP) France

³Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, France

pauline.dreyer@inria.fr

## ABSTRACT

Brain-Computer Interface (BCI) performance suffer from various variability sources, including intra-subject factors such as mental fatigue. While frequently measured using subjective reports, mental fatigue can also be assessed via blink parameters extracted from electro-oculography signals. To our knowledge, no study has yet evaluated blink parameters during motor imagery (MI) BCI use to assess the potential development of mental fatigue. In this study, the blinks of 23 MI-BCI participants were analyzed concurrently with subjective reports and BCI performance. Our results showed that blink parameters were correlated with neither MI-BCI performance nor subjective reports. However, they revealed a positive correlation between time-on-task and both blinks number and mean duration. Similarly, subjective fatigue was correlated with time-on-task. This suggests that blinks parameters may be useful for BCI user monitoring, although their relationship with BCI performance and fatigue needs further studies. Altogether, this study paves the way towards a better understanding of mental fatigue during BCI use, and in finding solutions to mitigate it.

## INTRODUCTION

Intra-user Brain-Computer Interface (BCI) performance is known to fluctuate due to the interaction of several potential sources of variability including context, time and day, as well as user engagement and fatigue [1–6]. Mental fatigue<fnref n="1" /> has been long known to impact human performance and engagement in general [7], and has started to be studied regarding both active and passive BCI performance in the last decade [8, 9]. The measures that were used to evaluate user fatigue in those previous studies were mostly focused on electroencephalography (EEG) metrics, as well as subjective (i.e. questionnaires) and behavioral metrics.

Yet, to the best of our knowledge, the ocular behavior metrics that can be extracted from the electro-oculography (EOG) signal – and which are widely used

for mental fatigue and vigilance characterization [10, 11] – have never been used to study fatigue during active BCI operation. Such metrics include blink number, blink duration, opening and closing velocity, as well as opening and closing duration [12–14].

Given the lacks identified in the literature, in order to better understand the development of user fatigue during the execution of active BCI tasks, and more precisely during Motor Imagery (MI) BCI tasks, the present study investigates the evolution of blinks and their parameters across MI-BCI runs based on EOG signal analysis. To do so, a standard motor imagery BCI protocol was used, in which participants also had to answer questionnaires to gather subjective reports of fatigue. It was expected that (i) blink number and duration would increase with runs. It was also expected that (ii) blink number and parameters would correlate with BCI performance and subjective reports. The remainder of this paper presents the data used, the analysed performed, the results obtained and their interpretation.

## MATERIALS AND METHODS

### Participants:

Twenty-tree (23) participants completed the BCI experiment (10 women/13 men), aged 28.4 ± 6.2 y.o. Recruitment was limited to volunteer participants aged 18-60 years old, with no history of neurological or psychiatric disorders, normal (or corrected) vision and naive MI-BCI users, i.e. using a MI-BCI system for the first time. Before participating in each study, participants gave informed consent. The study has been approved and reviewed by Inria's ethics committee, the COERLE (Approval number: 2020-32).

### Protocol:

The experiment consisted of 3 experimental MI-BCI sessions (completed on 3 different days) per participant. A brief pre-session questionnaire was assessed at the beginning of each session to measure the participant alertness. However we did not use this questionnaire for

<footnote n="1">¹a.k.a., reduced alertness, which arises from growing time-on-task.</footnote>
this study. Participants were then asked to perform two short working memory tasks to serve as EEG training data for a future offline passive BCI study unrelated to the present paper. The MI-BCI training and use then started.

The training protocol used for this experiment follows the standard left and right hand MI-BCI protocol from TU Graz [15], which comprises two phases: (1) motor imagery practice to collect data for calibrating machine learning algorithms (runs 1–4), and (2) closed-loop user-training with real-time classifier feedback (runs 5–12). After each of the 12 runs, participants were instructed to rate their mental state using a 1-10 scale of selected items from the NASA Task Load Index (NASA-TLX) questionnaire [16] (mental demand, effort and frustration levels), as well as their subjective mental fatigue. Note that other items of the NASA-TLX were not used in order to keep the number of questions to a minimum. The general experiment workflow is illustrated in Figure 1.

<figure><img src="image_3.png" /><figcaption>Figure 1: Experimental protocol. The orange circles correspond to the minutes that each part takes. In green rectangles are the questionnaires, in pastel orange rectangles are the experimenter's instructions, in pink rectangles are the technical processes with the cap, in yellow are the EEG recordings.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 2: Organisation and timing of a single MI-BCI trial.</figcaption></figure>

During each run, participants performed 16 trials (8 per MI-task, presented in a random order) each trial lasting 8s. First a green cross appeared (t = 0s) on the screen, then an acoustic signal (t = 2s) announced the appearance of a red arrow (t = 3s). The arrow pointed towards the task to be performed. (e.g., towards the left for left hand MI) and remained displayed for 1.25s. From t = 4.25s, the visual feedback was continuously provided (a blue bar varying in length according to the classifier output). The feedback lasted for 3.75s and was updated at 16Hz, using a 1s sliding window. Positive feedback only was displayed.

Then the screen turned black again after 8 seconds until

the next trial begin, starting randomly between 1.5 to 3.5 seconds later. (see Figure 2). At the end of the session, participants filled-in the post-experiment NeXT-Q questionnaire [17] (around 5 min). Then, the cap was removed and participants were debriefed (around 8 min).

### EEG and EOG data acquisition:

Participants sat comfortably in a chair in front of a computer screen. EEG data was acquired using 42 active scalp electrodes (i.e., F3, Fz, F4, FC5, FC3, FC1, FCz, FC2, FC4, FC6, C3, C1, Cz, C2, C4, CP5, CP3, CP1, CPz, CP2, CP4, CP6, P3, Pz, P4, AF3, AFZ, AF4, FC7, FC8, C5, C6, TP9, TP7, TP8, TP10, PO7, POz, O1, Oz, O2, PO8, 10-20 system), referenced to the left earlobe, the ground electrode being placed in FPz position. The electrooculography (EOG) signals of one eye was recorded using three active electrodes. Two of them were located below and above the eye (EOG1 and EOG3) and one was located on the side of the left eye (EOG2). We also recorded the electromyographic (EMG) signals of both hands using two active electrodes located 2.5 cm below the skinfold on each wrist. Physiological signals were measured using two g.USBAmp amplifiers (g.tec, Austria), sampled at 512 Hz, and processed online using the open-source BCI platform OpenViBE [18]. The recording room was dimly-lit. The raw signals were recorded without any hardware filters.

### Online BCI Performances:

The metric used for quantifying BCI performances is the online Trial-wise Accuracy (TAcc), i.e. the default performance metric provided online in the MI-BCI scenarios of OpenViBE. TAcc measures the accuracy of trial classifications, with each trial categorized as either correctly or incorrectly classified. The classification outcome for each trial is computed by summing the signed classifier outputs over all epochs during the trial feedback period (from t = 4.25 s to t = 8 s of the trial). A trial is considered correctly classified if the sum sign matches the required trial label (negative for left hand MI and positive for right hand MI), otherwise, it is considered incorrect. For this experiment, online classification was performed using Common Spatial Pattern (CSP) (3 filter pairs) band power features in 8-30 Hz and a Linear Discriminant Analysis (LDA) classifier. TAcc for each run was calculated as the percentage of trials accurately classified using this methodology. Notably, this metric utilizes LDA outputs instead of discrete classification outputs for each epoch. Therefore, TAcc also reflects the length of the feedback bar participants observed, as it is proportional to the classifier output. Participants were instructed to train to achieve not only correct classifications but also maximize the length of this feedback bar. Thus, TAcc considers both aspects, providing a comprehensive assessment of BCI performance.

### EOG signal analysis:

EOG signals were analysed with MNE Python [19],
a popular EEG and EOG data analysis toolbox that provides extensive event detection and feature extraction capabilities, and with NeuroKit2 a Python toolbox for neurophysiological signal processing with advanced artefact detection and removal [20].

The processing of EOG signals, to detect blinks and extract their parameters, was the following, for each run:
(1) **Bipolar channel**: Creation of a bipolar EOG channel, EOG1-EOG3, focusing on vertical EOG signals which should capture blinks to enhance the blink peak detection with MNE.

(2) **Cleaning**: Cleaning the bipolar EOG channel using NEUROKIT `eog_clean()` function to prepare for eye blink detection.

(3) **Event detection**: Detecting EOG events using MNE: with the `preprocessing.find_eog_events()` function, to detect EOG events in the cleaned bipolar EOG channel. While this function typically enables precise identification of eye movements and blinks, its performance may be compromised by the presence of high amplitude artifacts, especially when the signal contains significant noise or bifurcated artifacts. Indeed, such function uses a threshold on EOG amplitude to detect blinks, this threshold being estimated according to the EOG minimum and maximum values. To address this challenge, we implemented a preprocessing step aimed at enhancing the robustness of EOG event detection.

(4) **Winsorization**: Specifically, we applied winsorization to the raw EOG signal, exclusively on runs that exhibited significant artifacts. Winsorization is a technique that limits the influence of extreme values (here higher than the 95 percentile of the EOG signal) by replacing them with less extreme values (here the 95 percentile), thereby reducing the impact of outliers on subsequent analyses. Doing so, we were able to accurately identify and quantify the overall number blink events within each run, as confirmed with visual analysis.

(5) **Additional epoching**: In addition to the overall number of blinks per run, we also epoched the runs to estimate the number of blinks occurring only during the motor imagery task. This thus provided us with the number of blinks during MI tasks per run.

(6) **Blink features**: Then we extracted blink features, describing the characteristics of the blinks, shedding light on underlying physiological processes and potential mechanisms contributing to fatigue. More precisely We used Neurokit `eog_features()` function to extract EOG-related features from each blink of the cleaned bipolar EOG channel. These features included the blink duration and the blink Velocity (of eyes closing velocity denoted as pAVR and eyes opening velocity denoted nAVR), i.e., the speed at which blinks occur.

*Statistical analyses to study the relationship between blinks parameters, time-on-task, MI-BCI performance and subjective mental states:*

Our goal was to study whether we could identify relationships between the parameters extracted from the blinks (number of blinks, duration and velocity) and MI-BCI performance, time-on-task (here measured as the run index, which increases with time-on-task) and/or subjective mental states, notably mental fatigue.

To assess these potential relationships, we used repeated measures correlation (rmcorr) analyses, to determine the common within-individual association for paired measures assessed multiple times for multiple individuals. [21]. Here the repeated measures per subject were the measures collected across all 8 feedback runs per session (repeated measures across runs). A total of 27 correlation analyses were performed: 4 between the number of overall blinks and the 4 mental states, 4 between MI-BCI performance and the 4 mental states, 4 between the number of blinks (overall and in MI tasks only) and time-on-task or MI-BCI performances, 6 between blinks duration and the 4 mental states, MI-BCI performances and time-on-task, and 8 between the mean blink velocity (pAVR and nAVR), and the 4 mental states and 1 between subjective fatigue and time-on-task. Thus, all p-values for these analyses are reported as corrected for multiple comparison with False Discovery Rate (FDR) across these 27 tests.

## RESULTS

*Relationship between subjective mental fatigue and time-on-task:*

There was a positive correlation between time-on-task (as measured by the run index) and the subjective mental fatigue, showing that participants tend to report being increasingly more tired as time-on-task with the BCI increases (see Figure 3).

<figure><img src="image_5.png" /><figcaption>Figure 3: Repeated measure correlation between time-on-task (measured using run index) and the mental fatigue per participant, r=0.2, p<0.00001 (one colour per participant).</figcaption></figure>

*Relationship between BCI performances and mental states:*

No significant correlation was observed between mental fatigue, mental demand, and effort with online performance measures. However, a slight negative correlation
was detected ($r = -0.138, p<0.005$) between the performance and the frustration per participant, indicating that participants with good performance tend to be less frustrated (see Figure 4).

<figure><img src="image_6.png" /><figcaption>Figure 4: Negative repeated measure correlation per participants between BCI performance in % and the frustration measured by NASA-TLX, $r=-0.1386, p<0.005$ (one colour per participant).</figcaption></figure>

*Relationships between number of blinks and BCI performance or mental states:*

No significant correlations were found between the number of blinks observed during the runs and online performance. Similarly, there were no significant correlations detected between the number of blinks and the four different subjective mental states (i.e., mental fatigue, mental demand, effort, and frustration) assessed.

*Relationships between blinks parameters and BCI performance or mental states:*

First, it should be noted that Neurokit was not able to extract automatically the blink parameters from all blinks. Out of 552 runs, Neurokit was able to extract all the blink parameters for all the blinks of 380 of those runs. The subsequent results are thus based on 380 runs.

Our correlation analyses revealed that there were no significant correlations between between blink velocity (mean pAVR and mean nAVR) per run and the various mental states. There was no significant correlation between the mean duration of blinks and the different mental state, except with frustration. There was a negative correlation between them ($r=-0.15, p<0.01$), suggesting that blink duration decreases when frustration increases (see Figure 5). There was also no correlation between the mean duration of blinks and MI-BCI performance.

*Relationship between blinks parameters and time-on-task:*

A significant positive correlation was observed between the overall number of blinks per run and time-on-task (i.e., with the run index) ($r = 0.178, p<0.0005$), suggesting that for each session, the number of blinks increased with time-on-task, i.e., with the number of MI-BCI runs completed (see Figure 6).

<figure><img src="image_7.png" /><figcaption>Figure 5: Repeated measure correlation between frustration and the mean duration of blinks, $r=-0.15, p<0.01$ (one colour per participant).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 6: Repeated measure correlation between the overall number of blinks per run and time-on-task (as measured by the run index) per participant, $r=0.178, p<0.0005$ (one colour per participant).</figcaption></figure>

Similarly there was a positive correlation between the number of blinks during the MI tasks per run and time-on-task (i.e., run index) ($r = 0.176, p<0.0005$), suggesting as well that the number of blinks during MI tasks per run increased with time-on-task, i.e., with the number of MI-BCI runs completed, in a given session (see Figure 7).

There was a positive correlation between the mean duration of blinks in a run and time-on-task (i.e., with the run index) ($r = 0.09, p<0.005$), suggesting that blinks are becoming increasingly longer with time-on-task, i.e., with the number of runs completed (see Figure 8).

DISCUSSION

Overall, contrary to our initial hypotheses, we did not find any significant correlation between the number of blinks or any of the blinks parameters (duration, opening and closing velocity) and neither online MI-BCI performance nor with subjective mental states, including fatigue.

However we could find a significant positive correlation between the subjective fatigue and time-on-task, suggesting that a BCI session is increasingly more tiring as time-
<figure><img src="image_9.png" /></figure>

Figure 7: Repeated measure correlation between the number of blinks during the motor imagery task per run, and time-on-task (as measured by the run index) per participant, *r*=0.176, *p*<0.0005 (one colour per participant).

<figure><img src="image_10.png" /></figure>

Figure 8: Repeated measure correlation between the duration of blinks during the motor imagery task and time-on-task (as measured by the run index) per participant, *r*=0.09, *p*<0.005 (one colour per participant).

on-task increases, which confirms subjective reports from our participants.

Interestingly enough, while the number of blinks or the mean blink duration were not correlated with subjective fatigue, they were both significantly correlated (although with a weak correlation) with time-on-task, as subjective fatigue did. In other words, both the number of blinks and the mean blink duration increased with time-on-task with a BCI. In the literature, both parameters have been related with mental fatigue [10, 11], although not only. However, the fact that they were not correlated with subjective fatigue nor with MI-BCI performance raises a number of questions. It may be that the number and mean duration of blinks are rather affected, in this context of MI-BCI, by visual fatigue (the current BCI protocol being based on visual cues and feedback) rather than by mental fatigue. Another possible interpretation could be that MI-BCI performance, blink numbers and duration and subjective fatigue have a more complex relationship, possibly non-linear, that is not captured by the linear correlation analyses we performed. Along these lines, it is interesting to note that subjective fatigue was also not linearly

correlated with MI-BCI performance, even though mental fatigue is known to affect EEG and thus possibly BCI performance [8]. Alternatively, maybe that subjective fatigue and more objective markers of fatigue such as those based on EOG as studied here significantly differ from each other. This will need to be studied in more details in the future.

An unexpected finding was that blink duration significantly decreased with increased subjective frustration during MI-BCI use. A possible interpretation could be that this frustration is most likely due to poor MI-BCI performances (since frustration and MI-BCI performance were also correlated), and that such poor performance motivates the (frustrated) users to focus more on the task, leading to shorter blinks.

## CONCLUSION

In this paper, our aim was to study whether blink parameters, such as their number, their duration or velocity, could be used to monitor fatigue during MI-BCI use, and to study whether they were related to subjective fatigue, MI-BCI performance and time-on-task during MI-BCI use. To do so, we analysed the data of 23 participants, who performed 3 sessions of 12 runs each (including 8 runs with real-time feedback) of MI-BCI training. We studied the (linear) relationships between these participants' blink parameters estimated from their EOG signals and their MI-BCI performance, subjective mental states (including fatigue, measured after each run) and time-on-task.

Altogether, our analyses did not reveal any significant correlation between these blink parameters and neither MI-BCI performance nor subjective mental states. They did reveal a positive correlation between time-on-task and both the number of blinks and the mean blink duration. Similarly, subjective fatigue significantly correlated with time-on-task.

Overall, while these blink parameters did not prove as accurate as expected to monitor mental fatigue during MI-BCI use, they do reflect time-on-task and may thus still be useful to consider for user monitoring during MI-BCI use. Additionally, the results obtained also call for further studies to better understand the link between MI-BCI performance, subjective measures of fatigue and EOG-related blink parameters.

Future works could consider additional markers of fatigue, e.g., measures of saccade, which may be more reliable for cognitive state monitoring than blinks [14], or non EOG markers, e.g., cardiac markers or even directly EEG markers of fatigue [8]. In conclusion, this study paved the way towards acquiring a better understanding of mental fatigue in the context of MI-BCI use, and therefore in finding solutions to mitigate such fatigue to increase user engagement and performance.
REFERENCES

[1] Grosse-Wentrup M, Schölkopf B. A review of performance variations in SMR-based brain-computer interfaces (BCIs). Brain-Computer Interface Research: A State-of-the-Art Summary. 2013:39–51.

[2] Dehais F et al. Monitoring pilot's mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors. 2019;19(6):1324.

[3] Podda J et al. Spatial constraints and cognitive fatigue affect motor imagery of walking in people with multiple sclerosis. Scientific Reports. 2020;10(1):21938.

[4] Saha S, Baumert M. Intra-and inter-subject variability in EEG-based sensorimotor brain computer interface: A review. Frontiers in computational neuroscience. 2020;13:87.

[5] Benaroch C et al. Long-term BCI training of a tetraplegic user: Adaptive riemannian classifiers and user training. Frontiers in Human Neuroscience. 2021;15:635653.

[6] Roy RN et al. Retrospective on the first passive brain-computer interface competition on cross-session workload estimation. Frontiers in Neuroergonomics. 2022;3:838342.

[7] Lal SK, Craig A. A critical review of the psychophysiology of driver fatigue. Biological Psychology. 2001;55(3):173–194.

[8] Roy RN, Bonnet S, Charbonnier S, Campagne A. Mental fatigue and working memory load estimation: Interaction and implications for EEG-based passive BCI. In: 2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2013, 6607–6610.

[9] Talukdar U, Hazarika SM, Gan JQ. Motor imagery and mental fatigue: Inter-relationship and EEG based estimation. Journal of computational neuroscience. 2019;46:55–76.

[10] Roy RN, Charbonnier S, Bonnet S. Eye blink characterization from frontal EEG electrodes using source separation and pattern recognition algorithms. Biomedical Signal Processing and Control. 2014;14:256–264.

[11] Kołodziej M et al. Fatigue detection caused by office work with the use of EOG signal. IEEE Sensors Journal. 2020;20(24):15213–15223.

[12] Caffier PP, Erdmann U, Ullsperger P. Experimental evaluation of eye-blink parameters as a drowsiness measure. European journal of applied physiology. 2003;89:319–325.

[13] Schleicher R, Galley N, Briest S, Galley L. Blinks and saccades as indicators of fatigue in sleepiness warnings: Looking tired? Ergonomics. 2008;51(7):982–1010.

[14] Bafna T, Hansen JP. Mental fatigue measurement using eye metrics: A systematic literature review. Psychophysiology. 2021;58(6):e13828.

[15] Pfurtscheller G, Neuper C. Motor imagery and direct brain-computer communication. Proceedings of the IEEE. 2001;89(7):1123–1134.

[16] Cegarra J, Morgado N. Étude des propriétés de la version francophone du NASA-TLX. In: EPIQUE 2009: 5ème Colloque de Psychologie Ergonomique. 2009, 233–239.

[17] Bismuth J, Vialatte F, Lefaucheur JP. Relieving peripheral neuropathic pain by increasing the power-ratio of low-β over high-β activities in the central cortical region with eeg-based neurofeedback: Study protocol for a controlled pilot trial (SMRPain study). Neurophysiologie Clinique. 2020;50(1):5–20.

[18] Renard Y et al. OpenViBE: An open-source software platform to design, test and use brain-computer interfaces in real and virtual environments. Teleoperators and Virtual Environments. 2010;19(1):35–53.

[19] Gramfort A et al. MEG and EEG data analysis with MNE-python. Frontiers in neuroscience. 2013;7:267.

[20] Makowski D et al. NeuroKit2: A python toolbox for neurophysiological signal processing. Behavior Research Methods. 2021;53(4):1689–1696.

[21] Bakdash JZ, Marusich LR. Repeated measures correlation. Frontiers in psychology. 2017;8:456.