<figure><img src="image_83.png" /><figcaption>**Figure 1.** Experimental design and performance evaluation. (a) A participant playing the Space Fortress game (in this example, the participant is equipped with a 4 × 1 HD montage). (b) Space Fortress in-game screenshot. (c) Timeline of the experimental protocol. GS*n* stands for Game Session number *n* across days. Short-term is the delta of performance between GS9 and GS1. Long-term is the delta of performance between GS11 and GS1. Retention is the delta of performance between GS11 and GS9.</figcaption></figure>

**Behavioural effects of the stimulation type.** Except for day 1, when the baseline was recorded, participants performed two game sessions (GS) each day, leading to a total of 11 GS. We computed the performance improvement as the difference of scores between the baseline score at day 1 (GS1) and the second game session of day 5 for the short-term delta score (GS9–GS1), and between GS1 and the second game session of day 15 for the long-term delta score (GS11–GS1). The retention delta score was computed as the difference between the long-term (day 15) and short-term (day 5) second game sessions (GS11–GS9). Figure 1 gives an overview of days and game sessions distribution over time, and illustrates the timeline for the delta scores.

The short-term delta score was significantly and positively influenced by the gaming experience ($F_{(1,57)} = 7.37, p = 0.009, \eta_p^2 = 11\%, b = 0.36, \text{power} = 0.76$) but not by the stimulation group ($F_{(2,57)} = 2.27, p = 0.112, \eta_p^2 = 7\%, \text{power} = 0.44$) (Fig. 4a).

The gaming experience effect faded with time to become non significant for both the long-term delta score ($F_{(1,57)} = 2.87, p = 0.096, \eta_p^2 = 5\%, b = 0.20, \text{power} = 0.38$) and the retention delta score ($F_{(1,57)} = 1.24, p = 0.270, \eta_p^2 = 2\%, b = -0.14, \text{power} = 0.19$). However, significant effects of the stimulation group were found for both long-term delta ($F_{(2,57)} = 4.21, p = 0.020, \eta_p^2 = 13\%, \text{power} = 0.72$) and retention delta ($F_{(2,57)} = 4.32, p = 0.018, \eta_p^2 = 13\%, \text{power} = 0.73$) (Fig. 4b,c).

**tRNS Adverse Effects Questionnaire.** The results of the tRNS Adverse Effects Questionnaire can be found in Figs. S2 and S3 in supplementary material. A Kruskal–Wallis test compared the three groups on the post session questionnaire and revealed a significant effect ($p = 0.034$). Post-hoc analyses showed that the HD-tRNS group had higher scores than the SHAM group ($p = 0.048$).
<figure><img src="image_84.png" /><figcaption>**Figure 2.** Mean total scores for Space Fortress game (with standard error bars) per group across game sessions. The fitted ln model across game sessions and for each group is presented in the lower right part. D day.</figcaption></figure>

<figure><img src="image_85.png" /><figcaption>**Figure 3.** Learning rate by stimulation group (Hypothesis 1). Bold: *p* < 0.05.</figcaption></figure>

Discussion

The present experiment is the first to investigate the effects of different tRNS montages (SD vs. 4 × 1 HD) on learning a complex video game during a one-week training. We hypothesized that both tRNS groups would have a better video game performance compared to the sham group, with an additional advantage for HD stimulation. More specifically, we excepted higher learning rates and improvements on the short-term (day 5 vs. day 1), the long-term (day 15 vs. day 1) and on the retention period (day 15 vs. day 5). Although no short-term effect was observed, our data partially support our hypotheses with a tendency for the learning rate, along with a significant long-term effect for HD compared to SD, but not to sham. In addition, the highest retention performance was observed for the HD-tRNS group, compared to both sham and SD groups this time. In contrast, the SD-tRNS montage showed no differences with the sham group, whatever the comparison. To our knowledge, our study
<figure><img src="image_86.png" /><figcaption>**Figure 4.** (a) Delta between short-term and baseline performance (Hypothesis 2.a.). (b) Delta between long-term and baseline performance (Hypothesis 2.b.). (c) Delta between long-term and short-term performance (Hypothesis 3.). GS Game Session. Bold: *p* < 0.05.</figcaption></figure>

is the first to provide evidence that an HD-tRNS montage may be more effective than an SD-tRNS montage to train complex task skills in healthy adults. This study also confirms that the right DLPFC is a valid target for neuromodulation and enhancement of high-level cognitive functions²³.

One important finding here is the positive effect of the neurostimulation, for the HD-montage only, during the retention period—10 days after the end of training and stimulation. The difference between the stimulation groups was indeed the strongest between day 5 and day 15, suggesting that HD-tRNS would have an outlasting effect on performance gain once the training is over⁵⁶. This effect was not abrupt, but rather it seems to have strengthened over time. This observation is supported by the absence of short-term effect, followed by a superiority of HD montage that became significant for the long-term (compared to SD only), and for retention period (compared to both sham and SD).

This delayed behavioral effect might be explained by the time necessary to modify the activity of the frontoparietal network, solicited while playing SF video game⁵⁷. Indeed, one of the main action principle of tRNS is supposed to occur through stochastic resonance¹. According to this hypothesis, by inducing noise, such stimulation would modify the discharge threshold of the neurons in the targeted brain networks. Simultaneous neuronal stimulation would induce subsequent synaptic strengthening according to the coincidence-detection rule proposed by Hebb⁵⁸ and Konorski⁵⁹. Long-term potentiation (LTP) has been proposed as one of the main consequences of this synaptic modification, to explain how the memory and the learning are stored in the brain⁶⁰,⁶¹ following cognitive training and brain stimulation. More specifically, it has been proposed that cellular changes might be responsible for the long-term effects of tRNS⁶². On a more marcroscopic view, it has been shown that tRNS combined with cognitive training induced more efficient neurovascular coupling²¹. These neural mechanisms take place as the cognitive training starts, but may be measurable with some delay. The low intensity (from
<figure><img src="image_87.png" /><figcaption>**Figure 5.** Stimulation montages and influence maps according to the montage type. Extracted from NIC2 software v.2.0.11.7.</figcaption></figure>

– .5 to .5 mA) of the tRNS and the few stimulation sessions (a total of 60 min) in our study, arguably limited the extent of such synaptic plasticity, both in time and intensity, which may have led to a delayed observation of these effects. It seems therefore important to extend the classical pre-post evaluation of cognitive training with additional long-term and retention assessments in future studies.

Another important result for future research in non-invasive neurostimulation is the superiority of the 4 × 1 HD-tRNS montage over the SD montage. Although the actual mechanisms of tRNS are still discussed¹, this difference might lie in the distribution of the current flow, as illustrated in Fig. 5. First, the stimulation is more focal in the HD montage; the target area is narrowed and, therefore, receives a greater amount of current that may induce higher stochastic resonance effect¹. Second, the HD stimulation only targeted the right hemisphere, while the SD targeted both left and right. The right and left DLPFCs have been described as crucial for multitasking³⁹,⁴⁰ and executive functions³⁷,³⁸ in general, but it has also been suggested that concomitant stimulation of two interconnected brain areas might be more disruptive than beneficial because of inter-hemispheric inhibition⁶³,⁶⁴. The rapid (i.e. up to 2 ms) alternating stimulation of left and right DLPFCs with the SD montage may have induced such inter-hemispheric inhibition in the present study, which could have lead to an absence of behavioral difference with the Sham group. This superiority of unilateral over bilateral montage has been previously observed in a tDCS methodological article⁶⁵. The authors measured a higher electrical dose within the targeted DLPFC and also within the connected network nodes (i.e. the ventral attention network, the limbic system and the executive control network).

Although the superiority of the HD montage was among our hypotheses, the non-significant difference between the SD-tRNS and the sham groups was not expected. From the few other studies that have used similar tRNS parameters (i.e. F3/F4 electrode position, frequency range, intensity range, stimulation duration), three did show significant effects of the stimulation²⁰,²¹,²⁶ and one did not²³. Several factors such as lack of robustness in experimental designs, absence of full details about stimulation parameters, suitability of statistical approaches, and publication in favor of positive results⁶⁶ have been proposed to explain the discrepancy of results²³. Specifically, the present study (stimulation parameters) was designed based on Snowball et al.²¹ who found an improved learning rate and long-term performance in an arithmetic task for the tRNS group compared to a sham group. However, our study differs in the following points: lower frequency range (100–500 vs. 100–600 Hz), shorter
<table><caption>**Table 1.** Participants' demographics (mean ± SD). <sup>a</sup>Education is measured on a scale from 1 to 20 (with 12 = high school degree). <sup>b</sup>Video Game experience (VGexp) is measured by a questionnaire and range on a scale from 0 to 10. ***ANOVA's were performed for Age and Education, and a Kruskal-Wallis was performed for VGexp (available in Supplementary Information). LH Left-handed</caption><thead><tr><th></th><th>Sham</th><th>SD-tRNS</th><th>HD-tRNS</th><th>p-value***</th></tr></thead><tbody><tr><td>N</td><td>19</td><td>21</td><td>21</td><td>NA</td></tr><tr><td>Gender</td><td>4 women</td><td>3 women</td><td>3 women</td><td>NA</td></tr><tr><td>Handedness</td><td>1 LH</td><td>1 LH</td><td>1 LH</td><td>NA</td></tr><tr><td>Age</td><td>24.0 ± 4.6</td><td>22.7 ± 3.1</td><td>24.1 ± 3.4</td><td>0.39</td></tr><tr><td>Education<sup>a</sup></td><td>16.2 ± 2.0</td><td>16.4 ± 1.6</td><td>16.3 ± 1.8</td><td>0.93</td></tr><tr><td>VGexp<sup>b</sup></td><td>4.8 ± 2.4</td><td>3.8 ± 1.5</td><td>5.0 ± 3.0</td><td>0.32</td></tr></tbody></table>

training protocol (3 consecutive days of coupling tRNS and cognitive training vs. 5 consecutive days) and a different experimental task (Space Fortress vs. arithmetic task). All or some of these factors may explain the absence of effect found in our SD-tRNS group. Meanwhile, taking into account the superiority of the HD montage, we would recommend to rather consider this montage option over the SD one for a similar investigation. Further studies will be necessary to determine the tRNS parameters, brain location and/or cognitive training types most sensitive to the SD montage.

As expected, video game experience had a significant role during SF training<sup>45,67</sup>. Participants who were more exposed to video games (e.g. number of played hours per week) were more likely to perform better on SF. However, one interesting result is that the more the participants trained to the SF task, the lower this effect; eventually it became non-significant in the last session. On the contrary, the stimulation effect became more important as training went by. As a result, the gaming experience was an important variable to control for, even though its impact diminishes with training.

On average, the responses of the stimulated groups on the tRNS adverse effect questionnaire were slightly higher than those from the sham group (1.6 points higher for the HD-tRNS group, and 0.90 points higher for the SD-tRNS group compared to the 1.32 points Sham average score on the 48 point-scale, see Fig. S2 in supplementary material). This result indicates that some participants within these groups experienced an effect of the stimulation which was not present in the sham group. Looking specifically at the responses on each question (see Fig. S3 in supplementary material), we can identify that participants from the stimulated groups reported slightly higher psychological (fatigue, difficulties in concentrating) and physical (tingling, itching, flickering) symptoms. However, one should note that most participants in all groups reported no symptoms at all, especially in some important ones (pain, nausea, headache). These results are in agreement with a systematic review<sup>68</sup>, which showed that active stimulated groups often report slightly higher (but often non-significant) adverse effects than sham groups.

A limit lies in the moderate statistical power of the present study (ranging from 0.19 to 0.76). While being slightly higher than most of recent tRNS studies that include 10 to 18 participants per group<sup>6,14,16,17,20,21,23-25,27,69</sup>, this is still not enough to obtain a 0.80 power with a *p* value criteria of 0.05, as recommended<sup>70</sup>. This statistical power should be further improved to increase the confidence we have in the significant and non-significant results. In general, future studies should embrace the open science opportunity in order to predetermine sample size and to produce registered reports when appropriate, in order to eliminate some common statistical biases<sup>71</sup>. Future studies should also consider investigating the neural mechanisms of tRNS. Indeed, experimental works with animals are crucially lacking, and to date, only hypotheses about these mechanisms have been proposed, with few empirical data to support them<sup>1,11</sup>.

In conclusion, this study is the first to report that a 4 × 1 high-definition tRNS montage coupled with cognitive training on a complex video game task may be efficient to promote outlasting performance gain. This result is especially important because the learning period was relatively short—with only three 20-min learning sessions coupled with tRNS. This finding may have implications for students learning a profession with either complex tasks to be performed or prolonged training, such as pilots, air traffic controllers, surgeons, or software developers. Future studies may also explore the possibility of such montage to treat patients with cognitive disorders.

## Methods

**Participants.** Based on an a priori power analysis (available in supplementary material), 66 participants were initially planned. In total, 70 participants were recruited in the ISAE-SUPAERO (Toulouse, France) and Montpellier University campuses through flyers and mailing lists. Participants had to meet the following eligibility criteria and safety guidelines for tES use: to be free of a history of neurological, psychiatric or sensory disorders, cardiac or cardiovascular pathology, and no recent use of medications or substances that might interfere with brain activity.

From these 70 participants, 61 constituted the final sample. Four participants withdrew from the study (two because they were not available for the long-term session, and two because they were not psychologically comfortable with brain stimulation after the first session), and five were excluded because of impedance unstability detected by the stimulation software. All participants were given detailed information about the experiment and gave informed consent prior to start. The demographics of the sample are described in Table 1. The study
was approved by the local ethic committee of EuroMov-Montpellier (IRB1805A) and performed following the Declaration of Helsinki.

**Design and procedure.** A double-blind, randomized, placebo-controlled design was used for this study. Each participant was randomly assigned (by computer generation) to one of the three stimulation groups (Sham vs. SD-tRNS vs. HD-tRNS) using a priori randomization schedule. To ensure the double-blind, and as sham was an SD-like montage, the experimenter was led to believe that the design also included an HD-Sham condition (fictive fourth group). The experimenter was not aware of our hypothesis of a superior effect of the HD over the SD montage. The principal investigator did not take part in the experiments. He created a list which randomly assigned each of the 66 participants to one of the three groups (HD-tRNS, SD-tRNS, Sham). To ensure the double-blind for the HD-tRNS group, the 22 participants in this group were then divided into two fictive groups in the list, HD-tRNS and HD-sham (also generated randomly). For each new participant entering the study, the experimenter was given the corresponding group, and only at this moment. In other words, he did not have access to the entire list prior to the study and was led to believe to the existence of 4 groups of 22 participants each. The double-blind option resulting in 4 fictive groups (group 1, group 2, group 3 and group 4) for the experimenter was activated at the beginning of the study in the NIC software (Neuroelectrics, Barcelona, Spain). In addition, the principal investigator was the only one to know the password to unblind the protocol. The experiment was then stopped when we reached our goal of 66 participants. At this time, we unblinded the protocol, and we checked that the experimenter was indeed not able to identify the groups and was convinced by the fictive 4-group protocol.

For each participant, the experimentation lasted for five consecutive days at a fixed schedule with one additional session ten days later. At the beginning of the first day, participants completed questionnaires to gather demographics data (age, education level, handedness and game-level). Then, they read the rules of the SF task and practiced the video game for approximately 10 min in order to get used to the game rules (see below for the detailed practice description). Finally, they performed the first game session (baseline, day 1), playing SF for 10 min. From day 2 to day 4 (training period), participants played two game sessions, each lasting 10 min. During these experimental sessions, stimulation was performed according to the group assignment (sham vs. SD-tRNS vs. HD-tRNS). The last two experimental sessions (short and long-term evaluation) were similar, but with no stimulation (see Fig. 1).

All experimental sessions were performed in an experimental room with no window and with a stable temperature. The SF task was displayed on a 17" laptop. The stimulation device was systematically installed, and participants had to answer a questionnaire about potential sensations that could be related to electric stimulation (see below for details).

**Transcranial random noise stimulation.** The tRNS was applied only during the training sessions (days 2, 3 and 4, see Fig. 1b). The other days, such as baseline (day 1), short-term (day 5) or long-term (day 15), the participants did not receive stimulation.

During all sessions, the participants were mounted with a neoprene head cap (see Fig. 5). The SD-tRNS and sham groups had the same montage, with two circular sponstim (25 cm²) positioned on F3 and F4 (left and right DLPFCs) according to the 10–20 system of electrode placements. The HD-tRNS group had a 4 × 1 montage, with 5 circular (π cm²) NG Pistim gel electrodes placed on the C4, Fp2, Fz, F8 locations for the peripheral electrodes and F4, over the right DLPFC, as the central electrode.

The tRNS waves for the stimulated groups (SD-tRNS and HD-tRNS) were generated using a Starstim device (Neuroelectrics®). The random level of current was generated for every sample (sampling rate 1280 samples/s). The current intensity was randomized, between –.5 and .5 mA (mean = 0; SD = 0.333) following a normal distribution, which was in line with previous studies²¹,²⁵. As the current distribution in large band of high frequencies has been recommended for tRNS⁷², we chose the largest possible range of current frequency allowed by our device, which was distributed between 100 and 500 Hz. In the case of SD-tRNS montage, the two electrodes always send or received 100% of the delivered current. In the HD-tRNS montage (4 × 1), the “1” central electrode (F4) always send or received 100% of the delivered current, whereas the “4” peripheral electrodes send or receive about 25% (23% for F8 and Fz and 27% for C4 and Fp2) of the current, individually. Note that the percentages values have been calculated depending on the distance from F4. Both stimulated groups received a gradual rise (current ramp up) of 15 s at the beginning of stimulation and a gradual decrease (current ramp down) during the last 15 s. Between ramps, 20 min of stimulation matched the 20 min of Space Fortress training. Both stimulated groups received a total of 394.6 mC during the whole stimulation. Resulting influence maps are presented in Fig. 5. Note, however, that this total of electric charge was differently distributed in these groups due to the montages. In the HD-tRNS montage, this amount of electric charge was distributed mainly on one brain area (the right DLPFC), while in the SD-tRNS montage, it was equally distributed across both hemispheres (mainly on right and left DLPFCs). The sham group received the same current ramp up and down than the stimulated groups at the beginning and end, respectively, but no stimulation during the 20 min of SF playing. Consequently, the sham group received a total of 7 mC counting both ramp up and ramp down.

**The space fortress game.** For the purpose of this study, a Python-based (ver. 2.7) version of SF was chosen from the github.com/CogWorks website (see Fig. 1). A clear overview of the game is described by Boot et al.⁴⁵. The authors' description highlights the complexity of this complex task and the variety of cognitive resources needed to play the game. In brief, the main goal of the game is to control a spaceship in space with no gravity (first task). The second goal is to destroy the fortress (second task). In addition, the player has to memorize three specific letters that will help him to identify and destroy different types of mines (type-1 and type-2) that regu-
larly appear in the game (third task). Simultaneously, she/he must keep focusing on sequences of symbols (e.g. # and $) that appear continuously throughout the game in order to capture bonuses (fourth task). To obtain the higher possible score, the participant has to perform the four sub-tasks in parallel. To this goal, participants were given the following instruction: "You will maximize your score by performing all sub-tasks to the best of your abilities". A more precise description of rules and points distribution are available in Table S1.

During the first day, participants practiced the game. This practice started with reading a document that described the task (environment and rules). After reading, participants were asked to play the SF game for approximately 10 min. They practiced one task at a time following the variable-training methods used in previous studies⁴⁵. More precisely, participants were asked: to control the ship and only focus on destroying the fortress (step 1); to capture bonuses only (step 2); to destroy mines only (step 3); and finally to perform all tasks together (step 4). Note that participants were informed about the points distribution prior to play. The experimenter made sure that participants understood and had experienced all the rules of the SF game before starting the baseline evaluation.

A total of 11 SF game sessions (GS) of 10 min each were performed by the participants during six experimen-
tal sessions (day 1 to day 5 plus day 15, see Fig. 1). Two game sessions per experimental session were completed
except for day 1 that consisted of one game session only (baseline evaluation). Participants saw their score dis-
played at the end of each session. One SF total score was calculated per game session and corresponded to the
sum of the points that included all sub-tasks.

**Video game experience questionnaire.** Participants were asked three questions to assess their video game habits: (1) "In the last 12 months, how many times have you launched a video game (PC, home or portable consoles, smartphone)" (five-point Likert scale), (2) "Currently, on average, how much time per day do you spend playing video games?" (four-point Likert scale) and (3) "Have you had a period, or periods, in your life when you played video games intensively (more than 2 h a day on average for at least 3 months)?" (yes/no). The total score of this questionnaire was computed as the sum of all responses and ranged from 0 to 10.

**tRNS Adverse Effects Questionnaire.** After each session, a translated version (french) of the tDCS Adverse Effects Questionnaire⁶⁸ was administered to the participants (renamed tRNS Adverse Effects Questionnaire in this article). This questionnaire included 12 items on how they currently evaluate different sensations (headache, difficulty to concentrate, change of mood, change in visual perception, tingling, itching, burning, pain, fatigue, nervousness, unpleasant sensations and nausea, see Fig. S3) in which the participants rated on a five-point Likert scale (from 0 to 4). The total score of this questionnaire was computed as the sum of all responses and ranged from 0 to 48.

**Statistical analyses.** Statistical analyses were performed using the R studio software (R v.4.0.5)⁷³,⁷⁴. The assumptions of normality and homoscedasticity were checked with visual inspection of residuals (normal probability plot and residuals vs. predicted values respectively). Statistical tests (Shapiro-Wilk and Levene tests respectively) were additionally run when necessary.

We computed the learning rate using a log adjustment model ($y = a + b \times \ln(x)$), where y is SF performance, x is the game sessions over time, a is the intercept and b the regression coefficient)⁵⁵. This learning rate was computed for each participant and comprised all experimental sessions (two games per session) including baseline training and short and long-term evaluations. The regression coefficient from the model is the individual learning rate.

A one-way ANCOVA was conducted to assess significant difference between groups (sham vs. SD-tRNS vs. HD-tRNS) on the learning rate (b) controlling for video game experience (hypothesis 1). As gaming experience is likely to influence SF performance⁴⁵,⁶⁷, we decided to add it as a co-variate to ensure that the results showed the effect of the stimulation, independently of gaming experience.

We then measured the delta of performance between the baseline and the short-term session (GS9-GS1), between the baseline and the long-term session (GS11-GS1) and between the short-term and the long-term sessions (GS11-GS9) corresponding to the retention period. ANCOVAs were conducted to assess differences between groups on the delta of performance of either short-term (hypothesis 2.a), long-term (hypothesis 2.a) or retention (hypothesis 3), each time controlling for video game experience.

Whenever significant effect of the group was observed on previous ANCOVAs, we performed post-hoc comparisons to assess the differences between each groups (hypothesis 4). P-values have been adjusted using the Holm correction⁷⁵. To measure the effect size, we computed the $\eta_p^2$. We specified the assessed Beta value (referred to as b in the text) for the co-variate results to indicate the direction (positive or negative) of the effect.

**Data availability**

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Received: 1 October 2021; Accepted: 28 February 2022

Published online: 17 March 2022

**References**

1. Fertonani, A. & Miniussi, C. Transcranial electrical stimulation: What we know and do not know about mechanisms. *Neuroscientist* **23**, 109–123 (2017).

2. Kuo, M.-F. & Nitsche, M. A. Effects of transcranial electrical stimulation on cognition. *Clin. EEG Neurosci.* **43**, 192–199 (2012).
3. Dedoncker, J., Brunoni, A. R., Baeken, C. & Vanderhasselt, M.-A. The effect of the interval-between-sessions on prefrontal transcranial direct current stimulation (TDCS) on cognitive outcomes: A systematic review and meta-analysis. *J. Neural Transm.* **123**, 1159–1172 (2016).

4. Feltman, K. A., Hayes, A. M., Bernhardt, K. A., Nwala, E. & Kelley, A. M. Viability of TDCS in military environments for performance enhancement: A systematic review. *Mil. Med.* **185**, e53–e60 (2020).

5. Ho, K.-A., Taylor, J. L. & Loo, C. K. Comparison of the effects of transcranial random noise stimulation and transcranial direct current stimulation on motor cortical excitability. *J. ECT* **31**, 67–72 (2015).

6. Murphy, O. et al. Transcranial random noise stimulation is more effective than transcranial direct current stimulation for enhancing working memory in healthy individuals: Behavioural and electrophysiological evidence. *Brain Stimul.* **13**, 1370–1380 (2020).

7. Terney, D., Chaieb, L., Moliadze, V., Antal, A. & Paulus, W. Increasing human brain excitability by transcranial high-frequency random noise stimulation. *J. Neurosci.* **28**, 14147–14155 (2008).

8. Moliadze, V., Antal, A. & Paulus, W. Electrode-distance dependent after-effects of transcranial direct and random noise stimulation with extracephalic reference electrodes. *Clin. Neurophysiol.* **121**, 2165–2171 (2010).

9. Fertonani, A., Pirulli, C. & Miniussi, C. Random noise stimulation improves neuroplasticity in perceptual learning. *J. Neurosci.* **31**, 15416–15423 (2011).

10. Stacey, W. C. & Durand, D. M. Stochastic resonance improves signal detection in hippocampal CA1 neurons. *J. Neurophysiol.* **83**, 1394–1402 (2000).

11. Antal, A. & Herrmann, C. S. Transcranial alternating current and random noise stimulation: Possible mechanisms. *Neural Plast.* **2016**, 1-12 (2016).

12. Miniussi, C., Harris, J. A. & Ruzzoli, M. Modelling non-invasive brain stimulation in cognitive neuroscience. *Neurosci. Biobehav. Rev.* **37**, 1702–1712 (2013).

13. Lema, A., Carvalho, S., Fregni, F., Gonçalves, Ó. F. & Leite, J. The effects of direct current stimulation and random noise stimulation on attention networks. *Sci. Rep.* **11**, 1–15 (2021).

14. Mulquiney, P. G., Hoy, K. E., Daskalakis, Z. J. & Fitzgerald, P. B. Improving working memory: Exploring the effect of transcranial random noise stimulation and transcranial direct current stimulation on the dorsolateral prefrontal cortex. *Clin. Neurophysiol.* **122**, 2384–2389 (2011).

15. Brauer, H., Kadish, N. E., Pedersen, A., Siniatchkin, M. & Moliadze, V. No modulatory effects when stimulating the right inferior frontal gyrus with continuous 6 Hz tacs and trns on response inhibition: A behavioral study. *Neural Plast.* **2018**, 1–11 (2018).

16. Brevet-Aeby, C., Mondino, M., Poulet, E. & Brunelin, J. Three repeated sessions of transcranial random noise stimulation (TRNS) leads to long-term effects on reaction time in the go/no go task. *Neurophysiol. Clin.* **49**, 27–32 (2019).

17. Dondé, C., Brevet-Aeby, C., Poulet, E., Mondino, M. & Brunelin, J. Potential impact of bifrontal transcranial random noise stimulation (TRNS) on the semantic stroop effect and its resting-state EEG correlates. *Neurophysiol. Clin.* **49**, 243–248 (2019).

18. Bieck, S. M., Artemenko, C., Moeller, K. & Klein, E. Low to no effect: Application of TRNS during two-digit addition. *Front. Neurosci.* **12**, 176 (2018).

19. Looi, C. Y. et al. Transcranial random noise stimulation and cognitive training to improve learning and cognition of the atypically developing brain: A pilot study. *Sci. Rep.* **7**, 1–10 (2017).

20. Popescu, T. et al. Transcranial random noise stimulation mitigates increased difficulty in an arithmetic learning task. *Neuropsychologia* **81**, 255–264 (2016).

21. Snowball, A. et al. Long-term enhancement of brain function and cognition using cognitive training and brain stimulation. *Curr. Biol.* **23**, 987–992 (2013).

22. Harty, S. & Cohen Kadosh, R. Suboptimal engagement of high-level cortical regions predicts random-noise-related gains in sustained attention. *Psychol. Sci.* **30**, 1318–1332 (2019).

23. Holmes, J., Byrne, E. M., Gathercole, S. E. & Ewbank, M. P. Transcranial random noise stimulation does not enhance the effects of working memory training. *J. Cogn. Neurosci.* **28**, 1471–1483 (2016).

24. Parkin, B. L., Bhandari, M., Glen, J. C. & Walsh, V. The physiological effects of transcranial electrical stimulation do not apply to parameters commonly used in studies of cognitive neuromodulation. *Neuropsychologia* **128**, 332–339 (2019).

25. Brem, A.-K. et al. Modulating fluid intelligence performance through combined cognitive training and brain stimulation. *Neuropsychologia* **118**, 107–114 (2018).

26. Almquist, J.N.-F. et al. Fast: A novel, executive function-based approach to cognitive enhancement. *Front. Hum. Neurosci.* **13**, 235 (2019).

27. Pasqualotto, A. Transcranial random noise stimulation benefits arithmetic skills. *Neurobiol. Learn. Mem.* **133**, 7–12 (2016).

28. Villamar, M. F. et al. Technique and considerations in the use of 4x1 ring high-definition transcranial direct current stimulation (HD-TDCS). *J. Vis. Exp. JoVE* **77**, e50309 (2013).

29. Heise, K.-F. et al. Evaluation of a modified high-definition electrode montage for transcranial alternating current stimulation (TACS) of pre-central areas. *Brain Stimul.* **9**, 700–704 (2016).

30. Reinhart, R. M. Disruption and rescue of interareal theta phase coupling and adaptive behavior. *Proc. Natl. Acad. Sci.* **114**, 11542–11547 (2017).

31. Klírová, M. et al. Modulating inhibitory control processes using individualized high definition theta transcranial alternating current stimulation (hd θ-tacs) of the anterior cingulate and medial prefrontal cortex. *Front. Syst. Neurosci.* **15**, 25 (2021).

32. Lang, S., Gan, L. S., Alrazi, T. & Monchi, O. Theta band high definition transcranial alternating current stimulation, but not transcranial direct current stimulation, improves associative memory performance. *Sci. Rep.* **9**, 1–11 (2019).

33. Wu, L., Liu, T. & Wang, J. Improving the effect of transcranial alternating current stimulation (TACS): A systematic review. *Front. Hum. Neurosci.* **15**, 255 (2021).

34. Wager, T. D., Jonides, J. & Reading, S. Neuroimaging studies of shifting attention: A meta-analysis. *Neuroimage* **22**, 1679–1693 (2004).

35. Rottschy, C. et al. Modelling neural correlates of working memory: A coordinate-based meta-analysis. *Neuroimage* **60**, 830–846 (2012).

36. Zhang, R., Geng, X. & Lee, T. M. Large-scale functional neural network correlates of response inhibition: An fMRI meta-analysis. *Brain Struct. Funct.* **222**, 3973–3990 (2017).

37. Garavan, H., Ross, T. J., Murphy, K., Roche, R. A. & Stein, E. A. Dissociable executive functions in the dynamic control of behavior: Inhibition, error detection, and correction. *Neuroimage* **17**, 1820–1829 (2002).

38. Panikratova, Y. R. et al. Functional connectivity of the dorsolateral prefrontal cortex contributes to different components of executive functions. *Int. J. Psychophysiol.* **151**, 70–79 (2020).

39. Erickson, K. I. et al. Training-induced functional activation changes in dual-task processing: An fMRI study. *Cereb. Cortex* **17**, 192–204 (2007).

40. Anguera, J. A. et al. Video game training enhances cognitive control in older adults. *Nature* **501**, 97–101 (2013).

41. Kaller, C. P., Rahm, B., Spreer, J., Weiller, C. & Unterrainer, J. M. Dissociable contributions of left and right dorsolateral prefrontal cortex in planning. *Cereb. Cortex* **21**, 307–317 (2011).

42. Diamond, A. Executive functions. *Annu. Rev. Psychol.* **64**, 135–168 (2013).

43. Mané, A. & Donchin, E. The space fortress game. *Acta Physiol. (Oxf)* **71**, 17–22 (1989).

44. Donchin, E. Video games as research tools: The space fortress game. *Behav. Res. Methods Instrum. Comput.* **27**, 217–223 (1995).
45. Boot, W. R. *et al*. Transfer of skill engendered by complex task training under conditions of variable priority. *Acta Physiol. (Oxf)* **135**, 349–357 (2010).

46. Voss, M. W. *et al*. Effects of training strategies implemented in a complex videogame on functional connectivity of attentional networks. *Neuroimage* **59**, 138–148 (2012).

47. Niendam, T. A. *et al*. Meta-analytic evidence for a superordinate cognitive control network subserving diverse executive functions. *Cogn. Affect. Behav. Neurosci.* **12**, 241–268 (2012).

48. Martínez, K. *et al*. Changes in resting-state functionally connected parietofrontal networks after videogame practice. *Hum. Brain Mapp.* **34**, 3143–3157 (2013).

49. Chenot, Q., Lepron, E., De Boissezon, X. & Scannella, S. Functional connectivity within the fronto-parietal network predicts complex task performance: A FNIRS study. *Front. Neuroergonomics.* **2**, 1–14 (2021).

50. Fuster, J. *The Prefrontal Cortex* (Academic Press, 2015).

51. Datta, A., Elwassif, M., Battaglia, F. & Bikson, M. Transcranial current stimulation focality using disc and ring electrode configurations: Fem analysis. *J. Neural Eng.* **5**, 163 (2008).

52. Kuo, H.-I. *et al*. Comparing cortical plasticity induced by conventional and high-definition 4× 1 ring TDCS: A neurophysiological study. *Brain Stimul.* **6**, 644–648 (2013).

53. Muthalib, M., Besson, P., Rothwell, J. & Perrey, S. Focal hemodynamic responses in the stimulated hemisphere during high-definition transcranial direct current stimulation. *Neuromodul. Technol. Neural Interface* **21**, 348–354 (2018).

54. Lu, H. *et al*. Effect of repeated anodal HD-TDCS on executive functions: Evidence from a pilot and single-blinded FNIRS study. *Front. Hum. Neurosci.* **14**, 609 (2021).

55. Mathewson, K. E. *et al*. Different slopes for different folks: Alpha and delta EEG power predict subsequent video game learning rate and improvements in cognitive control tasks. *Psychophysiology* **49**, 1558–1570 (2012).

56. Bolognini, N., Pascual-Leone, A. & Fregni, F. Using non-invasive brain stimulation to augment motor training-induced plasticity. *J. Neuroeng. Rehabil.* **6**, 1–13 (2009).

57. Lee, H. *et al*. Videogame training strategy-induced change in brain function during a complex visuomotor task. *Behav. Brain Res.* **232**, 348–357 (2012).

58. Hebb, D. O. *The Organisation of Behaviour: A Neuropsychological Theory* (Science Editions New York, 1949).

59. Konorski, J. *Conditioned Reflexes and Neuron Organization* (Cambridge University Press, 1948).

60. Bliss, T. V. & Collingridge, G. L. A synaptic model of memory: Long-term potentiation in the hippocampus. *Nature* **361**, 31–39 (1993).

61. Stuchlik, A. Dynamic learning and memory, synaptic plasticity and neurogenesis: An update. *Front. Behav. Neurosci.* **8**, 106 (2014).

62. Cappelletti, M. *et al*. Transfer of cognitive training across magnitude dimensions achieved with concurrent brain stimulation of the parietal lobe. *J. Neurosci.* **33**, 14899–14907 (2013).

63. Daskalakis, Z. J., Christensen, B. K., Fitzgerald, P. B., Roshan, L. & Chen, R. The mechanisms of interhemispheric inhibition in the human motor cortex. *J. Physiol.* **543**, 317–326 (2002).

64. Ferbert, A. *et al*. Interhemispheric inhibition of the human motor cortex. *J. Physiol.* **453**, 525–546 (1992).

65. Soleimani, G. *et al*. Group and individual level variations between symmetric and asymmetric DLPFC montages for TDCS over large scale brain network nodes. *Sci. Rep.* **11**, 1–13 (2021).

66. Joober, R., Schmitz, N., Annable, L. & Boksa, P. Publication bias: What are the challenges and can they be overcome? *J. Psychiatry Neurosci. JPN* **37**, 149 (2012).

67. Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M. & Gratton, G. The effects of video game playing on attention, memory, and executive control. *Acta Physiol. (Oxf)* **129**, 387–398 (2008).

68. Brunoni, A. R. *et al*. A systematic review on reporting and assessment of adverse effects associated with transcranial direct current stimulation. *Int. J. Neuropsychopharmacol.* **14**, 1133–1145 (2011).

69. Van der Groen, O., Tang, M. F., Wenderoth, N. & Mattingley, J. B. Stochastic resonance enhances the rate of evidence accumulation during combined brain stimulation and perceptual decision-making. *PLoS Comput. Biol.* **14**, e1006301 (2018).

70. Button, K. S. *et al*. Power failure: Why small sample size undermines the reliability of neuroscience. *Nat. Rev. Neurosci.* **14**, 365–376 (2013).

71. Collaboration, O. S. *et al*. Estimating the reproducibility of psychological science. *Science* **349**, aac4716 (2015).

72. Moret, B., Donato, R., Nucci, M., Cona, G. & Campana, G. Transcranial random noise stimulation (TRNS): A wide range of frequencies is needed for increasing cortical excitability. *Sci. Rep.* **9**, 1–8 (2019).

73. Allaire, J. Rstudio: Integrated development environment for r. *Boston MA* **770**, 394 (2012).

74. R Core Team. *R: A Language and Environment for Statistical Computing*. (R Foundation for Statistical Computing, 2017).

75. Holm, S. A simple sequentially rejective multiple test procedure. *Scand. J. Stat.* **6**, 65–70 (1979).

## Acknowledgements

The authors would like to thank Pierre Esteves for his help during data acquisition.

## Author contributions

Q.C.: conceptualization, methodology, software, formal analysis, investigation, writing—original draft, writing—review and editing. C.H.: software, formal analysis, investigation, data curation, writing—review and editing, visualization. E.L.: formal analysis, writing—original draft, writing—review and editing. P.B.: methodology, investigation, writing—review and editing. X.D.B.: writing—review and editing, funding acquisition. S.P.: resources, writing—review and editing, supervision. S.S.: conceptualization, methodology, formal analysis, resources, writing—review and editing, supervision, project administration, funding acquisition.

## Funding

This work was supported by the French National Research Agency (ANR) and the Defence Procurement Agency (DGA), ASTRID program [Grant number ANR-17-ASTR-0005].

## Competing interests

The authors declare no competing interests.

## Additional information

**Supplementary Information** The online version contains supplementary material available at https://doi.org/10.1038/s41598-022-08545-x.

Correspondence and requests for materials should be addressed to Q.C. or S.S.
**Reprints and permissions information** is available at www.nature.com/reprints.

**Publisher's note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_88.png" /></figure>

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International

License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

© The Author(s) 2022
<figure><img src="image_89.png" /></figure>

Check for
updates

# Functional Connectivity Within the Fronto-Parietal Network Predicts Complex Task Performance: A fNIRS Study

Quentin Chenot¹*, Evelyne Lepron¹, Xavier De Boissezon² and Sébastien Scannella¹

¹ISAE-SUPAERO, Université de Toulouse, Toulouse, France, ²Toulouse Neurolmaging Center (ToNIC), Université de Toulouse, INSERM, Toulouse, France

Performance in complex tasks is essential for many high risk operators. The achievement of such tasks is supported by high-level cognitive functions arguably involving functional activity and connectivity in a large ensemble of brain areas that form the fronto-parietal network. Here we aimed at determining whether the functional connectivity at rest within this network could predict performance in a complex task: the Space Fortress video game. Functional Near Infrared Spectroscopy (fNIRS) data from 32 participants were recorded during a Resting-State period, the completion of a simple version of Space Fortress (monotask) and the original version (multitask). The intrinsic functional connectivity within the fronto-parietal network (i.e., during the Resting-State) was a significant predictor of performance at Space Fortress multitask but not at its monotask version. The same pattern was observed for the functional connectivity during the task. Our overall results suggest that Resting-State functional connectivity within the fronto-parietal network could be used as an intrinsic brain marker for performance prediction of a complex task achievement, but not for simple task performance.

**Keywords:** functional connectivity, fronto-parietal network, resting-state, video game, Space Fortress, complex task, performance, functional near infrared spectroscopy

## 1. INTRODUCTION

The ability to predict performance based on psychophysiological markers is one of the major interests of neuroergonomics (Ayaz et al., 2019). It would be profitable for many fields such as sports performance (Kipp et al., 2019), driving security (Asimakopulos et al., 2012), military selection (Barron and Rose, 2017), successful surgery (Carthey et al., 2003), or piloting a plane (Scannella et al., 2018). High-risk operators, in particular, are faced with multiple, complex tasks, and their ability to perform them successfully is important as it can have a major economic or human impact. The prediction of this capacity could also be used as part of a selection process and/or as an indicator to target individualized trainings. The main purpose of this study is to explore the brain markers of complex task performance. We focused specifically on the relationship between functional connectivity within the fronto-parietal network (FPN) at rest, and performance in Space Fortress (Mané and Donchin, 1989), a complex and semi-ecological task.

Complex task performance is likely to rely on the functional activity of brain areas and networks involved in high-level cognitive functions. Amongst all brain networks, the FPN has a central role in a variety of cognitive functions such as working memory, attention, shifting, and reasoning

OPEN ACCESS

**Edited by:**
Davide Valeriani,
Neurable Inc., United States

**Reviewed by:**
Zhen Yuan,
University of Macau, China
Saugat Bhattacharyya,
Ulster University, United Kingdom

*Correspondence:
Quentin Chenot
quentinchenot@gmail.com

**Specialty section:**
This article was submitted to Augmented and Synthetic Neuroergonomics, a section of the journal Frontiers in Neuroergonomics

**Received:** 31 May 2021
**Accepted:** 14 July 2021
**Published:** 10 August 2021

**Citation:**
Chenot Q, Lepron E, De Boissezon X and Scannella S (2021) Functional Connectivity Within the Fronto-Parietal Network Predicts Complex Task Performance: A fNIRS Study. Front. Neuroergon. 2:718176. doi: 10.3389/fnrgo.2021.718176
(Niendam et al., 2012; Martínez et al., 2013). The global functional connectivity in this network is high (Cole et al., 2012) and increases when dealing with novel complex tasks and high cognitive control (Cole and Schneider, 2007; Duncan, 2010; Cole et al., 2012). The FPN connectivity might therefore represent a good indicator of complex task performance. However, while measuring the FC during tasks provides crucial information about the biological role of this functional network, Resting-State FC (rsFC) would be an additional candidate to predict performance. Indeed, it would be interesting not to have to perform a task in order to obtain a neural marker for the very same task and easier to proceed to a 5-min recording at rest to obtain such information. The intrinsic functional connectivity of the FPN has already been correlated with high-level cognitive function capacities such as fluid intelligence (Cole et al., 2012; Hearne et al., 2016), executive functions in general (Xu et al., 2015), attention (Markett et al., 2014; Fellrath et al., 2016), and working memory in particular (van Dam et al., 2015; Liu et al., 2017). These skills have been measured with specific laboratory cognitive tasks such as the N-back task for working memory (Liu et al., 2017) or the Raven's matrices regarding fluid intelligence (Hearne et al., 2016). Here, in the context of neuroergonomics research, we aim at determining whether the rsFC within the FPN also predicts performance in a more complex and semi-ecological task than conventional executive tasks.

Among complex tasks that are plausible as real life situations and can still be easily studied in the laboratory, video games appear to be of particular interest (Boot et al., 2017). Video game practicing effects on cognitive processes, brain functions, and structural changes, have been extensively investigated during the last two decades (Boot et al., 2008; Bavelier et al., 2011; Kühn et al., 2019). In the present study we focused on one of them: the Space Fortress (SF) video game (Boot, 2015).

We selected this game because it has been specifically designed by psychologists to study complex skills acquisition (Mané and Donchin, 1989). According to the authors, the goal behind this task was to create a complex task with multiple demanding and overlapping component tasks that simulate the complexity of real-world tasks such as piloting, air traffic control or radar monitoring (Maclin et al., 2011). The rationale behind the choice of this task was—besides the fact that it was created to engage cognitive functions similar to real-life tasks—first the important history of scientific publication both in psychology and neuroscience fields (Boot et al., 2008, 2011; Maclin et al., 2011; Lee et al., 2012a,b; Mathewson et al., 2012) which facilitates the comparison; second its parameters (e.g., number of events, time duration, and so on) that can be easily controlled unlike a mainstream video game (Boot et al., 2008; Palaus et al., 2017; Dale et al., 2020).

Specifically, the SF video game can be decomposed into four sub-tasks: controlling a ship in the space, destroying a fortress, destroying moving mines, and capturing bonuses. Each of these sub-tasks involves different cognitive functions. The mine-task necessitates to memorize and pay attention to letters in order to correctly identify and destroy them; the bonus task necessitates to maintain and update symbols in short-term memory (much like a n-back task, Jaeggi et al., 2010) in order

to get bonuses; the fortress task necessitates to inhibit rapid fire (as it would not destroy the fortress) and to maintain attention to check the vulnerability of the fortress; controlling the ship necessitates visuo-spatial and motor skill abilities. Finally, when performed together, all of these sub-tasks necessarily involve several cognitive abilities such as selective and divided attention, planning, executive functions (updating, inhibition, switching, Miyake et al., 2000; Friedman and Miyake, 2017), episodic memory, cognitive control, distributing and allocating cognitive resources, decision-making and motor skill (Mané and Donchin, 1989; Donchin, 1995; Boot et al., 2010). While there is a need for more fundamental research to better understand the exact cognitive correlates of SF performance (Boot et al., 2017), it has been positively correlated with intelligence (Rabbitt et al., 1989). Moreover, SF training has been shown to improve performance on the N-back task (working memory) and the Raven's matrices task (fluid intelligence, Lee et al., 2012a). Therefore, the literature shows that this video game may be sufficiently complex in terms of cognitive functions recruitment to meet our needs of a complex and engaging task (Boot et al., 2011) in the context of neuroergonomics research.

As expected by the multitasking nature of SF, increased functional connectivity of the FPN was observed in participants who have been trained with SF (Voss et al., 2012). RsFC however, has not been studied with SF specifically so far, but using a puzzle video game (*Professor Layton and the Pandora's Box*), rsFC of the FPN has been shown to increase with training and was positively correlated with improvements on video game performance (Martínez et al., 2013).

The rsFC of the FPN can be assessed with different brain imaging technics such as fMRI. We aimed, however, to record it with a light, portable and relatively easy to use one: the functional near-infrared spectroscopy (fNIRS) that is more suited for the *brain at work* assessment. The question was then to examine if the rsFC of the FPN assessed with fNIRS could be a good indicator of later performance in SF. Due to its high portability and reasonable cost, fNIRS has proven to be a valuable tool to study neurophysiological correlates of complex virtual and ecological tasks (Ayaz et al., 2012; Gateau et al., 2015; Foy et al., 2016; Verdière et al., 2018; Deligianni et al., 2020; Fan et al., 2021). As an example, a widely reported fNIRS result is the increased level of oxygenated hemoglobin (HbO) in frontal and parietal areas during higher-level cognitive tasks (Causse et al., 2017; Fairclough et al., 2018). However, only few studies about FC with fNIRS have been run, partly because only cortical activity under the optodes can be measured, contrary to fMRI where the whole brain activity is measured. Recent advances in the neuroergonomics field showed, however, that the measure of functional connectivity in cortical networks with fNIRS can be of particular interest. For example, Verdière et al. (2018) performed correlations analyses and other connectivity metrics on fronto-occipital fNIRS signal. Their results show that functional connectivity analyses can better classify two flight simulator scenarios (automated vs. manual landing) than activation data. Similarly, Deligianni et al. (2020) were able to identify functional connectivity differences within frontal areas during a surgical task between junior and senior surgeons.
<table><caption>**TABLE 1 | Sample demographics.**</caption><tr><td>n</td><td>32</td></tr><tr><td>Gender</td><td>24 men / 8 women</td></tr><tr><td>Handedness</td><td>30 right-handed / 2 left-handed</td></tr><tr><td>Age</td><td>23.93 (sd* = 4.05)</td></tr><tr><td>Education</td><td>16.47 (sd = 2.01)</td></tr><tr><td>VGexp</td><td>4.01 (sd = 2.86)</td></tr></table>

Taken together, these results highlight the fact that the fNIRS has been successfully used to measure functional connectivity in cortical networks, and that it can predict behavioral performance. However, to our knowledge, no fNIRS study has attempted to investigate the functional connectivity of the FPN during both Resting-State and task, and its relationship with performance in a complex task. Our main hypothesis postulates that functional connectivity within this network, both at rest and during the task, is positively associated with complex task performance but not with simple task performance.

To test this hypothesis, we developed a fNIRS protocol with optodes positioned in frontal and parietal areas over the FPN. During the acquisition, participants performed first a Resting-State and then played two versions of SF. A monotask version where the participant's goal was only to destroy the fortress, which has been developed to less rely on higher-level cognitive functions; and a multitask version where the participant's aim was to capture bonuses and destroy mines in addition to the fortress, which should rely on higher-level functions (Boot, 2015) such as executive functions (i.e., shifting, updating, and inhibition; Miyake et al., 2000; Friedman and Miyake, 2017). A global value of the functional connectivity within the FPN has been calculated by averaging the correlation coefficients between all fNIRS channels. We expected significant positive relationship between rsFC of the FPN and multitask SF score only. We also expected the same results with on-task functional connectivity. To achieve a high-standard, this article followed recommendations and best practices recently proposed (Yücel et al., 2021) for fNIRS publications.

# 2. MATERIALS AND METHODS

## 2.1. Participants

Forty one healthy adults were recruited in the ISAE-SUPAERO (Toulouse, France) campus through flyers and mailing lists. Participants were welcomed at the laboratory and first filled questionnaires collecting basic demographics data, eligibility to the research and video-game expertise. Recruited participants had to meet eligibility criteria and to be free of a history of neurological or psychiatric disorders, cardiac or cardiovascular pathology, and no recent use of medications or substances that might interfere with brain activity. Participants also read a text explaining the experimental protocol and the goal and rules of the SF video game. All participants gave informed consent prior to the experiment and the local ethic committee of Toulouse

University approved the study (Ref 2019-151). The final sample that included fNIRS analyses was composed of 32 participants and their demographics data are described in the **Table 1**.

## 2.2. The Video Game Expertise Questionnaire

Participants were asked to answer three questions based on a previous questionnaire<fnref n="1" />. The first one was “In the last 12 months, how many times have you launched a video game (PC, home, or portable consoles, smartphone)” (five points Likert-scale), the second “Currently, on average, how much time per day do you spend playing video games?” (four points Likert-scale) and the third “Have you had a period or periods in your life when you played video games intensively (more than 2 h per day on average for at least 3 months)?” (dichotomic). The total score of this questionnaire was computed as the sum of all responses and ranged from 0 to 10.

## 2.3. The Space Fortress Game

### 2.3.1. General Principle

For the purpose of this study, a Python-based (ver. 2.7) version of SF was chosen<fnref n="2" /> (see **Figure 1**). A clear overview of the game is described by Boot et al. (2010). The authors' description point out the complexity of the intricate tasks and the variety of resources needed to play the game. In brief, the main goal of the game is to control a spaceship in the space with no gravity (first task). The second goal is to destroy the fortress (second task). In addition, the player has to memorize specific letters that will help him to identify and destroy two types of mines (type-1 or type-2) that regularly appear in the game (third task). Simultaneously, she/he must keep focusing on sequences of symbols that appear continuously throughout the game in order to capture bonuses (fourth task). More specifically, to obtain the higher possible score, the participant has to perform the four sub-tasks in parallel. A more precise description of rules and points distribution of this task are available in the **Table 2**.

### 2.3.2. Monotask and Multitask Versions

The game parameters were modified to create two versions. A monotask version was created as a “control” task that limited the involved cognitive functions. This SF monotask version consisted of only the two first sub-tasks, controlling the ship and destroying the fortresses. These sub-tasks mostly involved visuo-motor abilities. Regarding the monotask version specifically, participants were given the following instruction: “to maximize their score by destroying the fortress as many times as possible.”

The SF Multitask version consisted of the complete video game, with all four sub-tasks: controlling the ship, destroying the fortresses, destroying mines, and capturing bonuses. These four tasks performed concomitantly involved higher-level cognitive functions as the participant has to perform multitasking in order to obtain a higher score. Most of the high cognitive abilities may therefore be involved in this version, with (non-exhaustive) selective and divided attention, planning, updating, inhibition,

<footnote>*sd, standard deviation; VG, video game expertise questionnaire.</footnote>

<footnote n="1">¹https://spacefortress.blogspot.com/</footnote>

<footnote n="2">²https://github.com/CogWorks/SpaceFortress</footnote>
A Space Fortress monotask

<figure><img src="image_90.png" /></figure>

B Space Fortress multitask

<figure><img src="image_91.png" /><figcaption>**FIGURE 1** | In-game screenshot of both Space Fortress versions: (A) the monotask Space Fortress version with only the ship to control and the fortress to destroy. (B) A multitask Space Fortress version with the mines and bonus in addition.</figcaption></figure>

<table><caption>**TABLE 2** | Score distribution of Space Fortress sub-tasks.</caption><thead><tr><th>Sub-task</th><th>Keyboard*</th><th>Rules</th><th>Points</th></tr></thead><tbody><tr><td>1. Controlling the ship</td><td>z, q, d</td><td>- Avoid being hit<br>- Avoir being destroyed (after 3 hits)<br>- Avoid crossing game borders<br>- Avoid colliding with the Space Fortress<br>- Manage the missile stock</td><td>-50<br>-100<br>-35<br>-35<br>None</td></tr><tr><td>2. Destroy the fortress</td><td>Spacebar</td><td>- Hit the fortress 10 times with at least 250 ms between each shot<br>- Destroy the fortress with a double shot (after 10 hits)</td><td>None<br>250</td></tr><tr><td>3. Destroy the mines</td><td>j</td><td>- Memorize three letters at the beginning<br>- Identify Type-1 or Type-2 mines according to the memorized letters<br>- Destroy a mine<br>- Fail to destroy a mine before it disappears</td><td>None<br>None<br>50 or 60<br>-50</td></tr><tr><td>4. Capture bonuses</td><td>k, l</td><td>- Random pairs of symbols will appear, one symbol at a time (# $ &amp; or @)<br>- If the pair is # then $, capture the bonus<br>- Fail to capture a bonus</td><td>None<br>50 or 100<br>-50</td></tr></tbody></table>

switching, working memory, episodic memory, cognitive control, distributing and allocating cognitive resources, decision making, and visuo-motor abilities (Mané and Donchin, 1989; Donchin, 1995; Boot et al., 2010). Regarding the multitask version specifically, participants were given the following instruction: “to maximize your score by performing all sub-tasks to the best of your ability.”

### 2.3.3. Training

Training included reading a document that described the task (environment and rules) and practicing the SF task for 12 min in four 3 min steps following variable-training methods used in previous studies (Boot et al., 2010). Step 1: participants were

asked to control the ship and only focus on destroying the fortress. Step 2: to capture bonuses only. Step 3: to destroy mines only. Step 4: all tasks performed together. The experimenter made sure that participants understood and had experimented all the rules of the SF game before starting.

### 2.3.4. Scores Computation

One SF total score was calculated for each version (monotask and multitask) and corresponded to the sum of the points that included all sub-tasks. Note that the scores for the two versions are computed differently (the monotask doesn't have the mine and bonuses sub-tasks), so any direct comparison between the two scores cannot be interpreted as they do not reflect

<footnote>*Note that the experiment was conducted on an AZERTY computer keyboard.</footnote>
the difficulty of the task. The difficulty is inherent to the on-
going cognitive processes involved in the two versions, with a
very strong demand in multi-tasking in the multitask version
compared to the monotask version. For the multitask version, we
also extracted four sub-scores, which were calculated as the sum
of the points per sub-task (flight, fortress, bonus, mine). Details
of the points distribution is described in the **Table 2**. Note that
the participants were informed about the point distribution prior
to play, and saw their score displayed at the end of each run.

As the score for the two tasks are computed differently (see **Table 2**), the scores cannot be compared directly and they do not reflect the difficulty of the task. The difficulty is inherent to the on-going cognitive processes involved in the two versions, with a very strong demand in multi-tasking in the multitask version compared to the monotask version The number of pressed keyboard per minute (Action Per Minute, APM) was also calculated for the multitask version.

**2.4. Experimental Procedure**

The experiment took place in an experimental room with
no window and with a stable temperature. The participants
performed the demographic and video game expertise
questionnaires, followed by the SF video game on a 17"
laptop with the laptop keyboard.

The experiment consisted of three conditions, each lasting 10 min: Resting-State (RS), SF Monotask, and SF Multitask. During the RS period, participants were asked to let their mind wander for 10 min with eyes closed and without falling asleep. The SF monotask version consisted of a simplified version of the game with all sub-tasks removed except the flying one. The SF multitask version consisted of the full version of the game with all four sub-tasks. During all conditions, the participants were asked to “refrain from moving as best as they could to reduce potential neuroimaging artifacts.”

The experiment started with the RS period and was followed
by a training to the SF game. After training, participants played 10
min of the SF monotask and 10 min of SF multitask. The order
of both mono- and multi-task conditions was counterbalanced
across participants.

**2.5. fNIRS Acquisition and Analyses**

**2.5.1. fNIRS Acquisition and Montage**

Brain activity was measured in all three experimental conditions (Resting-State, SF monotask, SF multitask) with a functional near-infrared spectroscopy (fNIRS) device allowing the measurement of the hemodynamic response at the level of brain cortical areas. The system used was a continuous-wave NIRS instrument (NIRScout system, NIRx medical technologies, LLC) with 28 optodes (14 sources, 14 detectors, 32 channels, see **Figure 2**). The sample rate was 12.5 Hz and the measured wavelengths were 760 and 850 nm. A calibration was performed before and after the experiment in order to check each optode's signal quality.

A NIRScap (NIRx medical technologies, LLC) was used to
hold the optodes on the participant's head. The custom montage
consisted of 32 channel measurements with a 30 mm source-
detector separation that was assured by plastic spacers. Optodes

were positioned on the scalp to cover the brain areas from the
fronto-parietal network based on previous literature (Niendam
et al., 2012). These areas mainly included the middle and
superior frontal gyri in the frontal lobe, and the inferior and
superior parietal lobules in the parietal lobe (for more details,
see **Table 3**). The montage sensitivity through the targeted brain
areas was estimated with a Monte-Carlo simulation in the fNIRS
Homer 2 AtlasViewer<fnref n="3" /> toolbox (v2.8, p2.1; see **Figure 2** and
**Supplementary Table 1**).

The MNI coordinates of the centroid for each channel
measurement were obtained with AtlasViewer. We then localized
the regions targeted by our montage with Talairach software
(Lancaster et al., 1997, 2000). We first converted MNI
coordinates into Talairach coordinates with the icbm2tal Matlab
program (Lancaster transform). In the Talairach Client, we used
the ±4 mm option to get a description of the brain regions within
a 9 mm<fnref n="3" /> volume around each channel coordinate. The sub-
regions have been concatenated across the channels (Table 3).

**2.5.2. Data Quality Check**

To assess the quality of the fNIRS signal, data were visually inspected for each participant (n = 41) and each channel (n = 32). To complete visual inspection, the Scalp Coupling Index (SCI) was computed (Pollonini et al., 2016). This index is based on the presence of the cardiac signal, which should be observed in raw data for each channel. Values that approach 1 indicate that the data are clean. We rejected all channels with a value < 0.6. Because the fNIRS montage has been designed to specifically assess the average functional connectivity within the FPN, we set the acceptable number of rejected channels to 6 (~20% of the channels). Participants with more than six rejected channels would have potentially presented missing data in a whole brain area that would not have been acceptable for connectivity analysis purposes. Using this criterion, we found that nine participants had data that were not sufficient in quality and were therefore excluded for further analyses. Of the remaining 32 participants, 20 had 0 rejected channels, 5 had one rejected channel, and the last 7 participants had between 2 and 5 rejected channels.

**2.5.3. Preprocessing**

The preprocessing was done with MATLAB (2020B) using a
pipeline based on the scripts from the NIRS brain AnalyzIR
toolbox (Santosa et al., 2018). The preprocessing was performed
on the whole signal for each participant (see **Figure 3**). First,
fNIRS raw data were transformed in optical density. Second, a
motion correction algorithm (Temporal Derivative Distribution
Repair, TDDR; Fishburn et al., 2019) was applied. This motion
correction procedure uses an iterative-reweighting approach
based on the temporal derivative of the signal, and removes
effectively baseline shifts and motion spikes. Third, data were
converted in concentrations (HbO and HbR) using the beer-
Lambert Law (Jacques, 2013) with a partial pathlength factor of
5/50 (Scholkmann and Wolf, 2013). Data were then resampled
at 10 Hz and a fourth-order bandpass Butterworth filter with
a high-pass frequency of 0.01 Hz and a low-pass frequency

<footnote n="3">³https://www.nitrc.org/frs/shownotes.php?release_id=3956</footnote>
<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /><figcaption>**FIGURE 2** | Results of the Monte-Carlo simulation (Homer 2 AtlasViewer; v2.8, p2.1) over the frontal cortex (A), the parietal cortex (B), and both cortices from a lateral view (C). Red dots represent the LED emitters, blue dots the photoreceptors and yellow lines the channels. The color bar represents the spatial sensitivity of the fNIRS measurements. Its unit is expressed in mm⁻¹ and values range from 0.01 to 1 in log10 units: -2 to 0. The 2D fNIRS montage using the 10-20 system (EEG) as a reference is presented in (D), bold numbers stand for the measurement channel index.</figcaption></figure>

<table><caption>**TABLE 3** | Brain regions targeted by the fNIRS montage and corresponding channels.</caption><thead><tr><th rowspan="2">Lobe</th><th rowspan="2">Brain region</th><th colspan="3">Left hemisphere</th><th colspan="3">Right hemisphere</th></tr><tr><th>Channels</th><th>Voxels</th><th>BA</th><th>Channels</th><th>Voxels</th><th>BA</th></tr></thead><tbody><tr><td rowspan="6">Frontal</td><td>Inferior frontal gyrus</td><td>24</td><td>48</td><td>46</td><td>8</td><td>196</td><td>46</td></tr><tr><td>Medial frontal gyrus</td><td>21,22</td><td>113</td><td>6,9</td><td>5</td><td>42</td><td>9</td></tr><tr><td>Middle frontal gyrus</td><td>18,19,22,24,26</td><td>927</td><td>8,10,46</td><td>2,3,6,9,10</td><td>971</td><td>8,9,10,46</td></tr><tr><td>Precentral gyrus</td><td>-</td><td>-</td><td>-</td><td>10</td><td>74</td><td>9</td></tr><tr><td>Sub-gyral</td><td>22</td><td>50</td><td>8</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Superior frontal gyrus</td><td>18:23</td><td>744</td><td>8,9,10</td><td>1:7</td><td>949</td><td>8,9,10</td></tr><tr><td rowspan="5">Parietal</td><td>Inferior parietal lobule</td><td>28,30,31</td><td>517</td><td>40</td><td>12:15</td><td>784</td><td>40</td></tr><tr><td>Postcentral gyrus</td><td>28</td><td>99</td><td>2,40</td><td>12</td><td>185</td><td>2,40</td></tr><tr><td>Precuneus</td><td>32</td><td>68</td><td>19</td><td>14</td><td>45</td><td>7</td></tr><tr><td>Superior parietal lobule</td><td>32</td><td>246</td><td>7</td><td>14,15</td><td>80</td><td>7</td></tr><tr><td>Supramarginal gyrus</td><td>29</td><td>105</td><td>40</td><td>13</td><td>45</td><td>40</td></tr></tbody></table>

BA, broadman areas.

of 0.09 Hz was applied in order to remove artifacts (cardiac oscillations, respiration, Pinti et al., 2019). Finally, a PCA analysis was carried out in order to reject the potential remaining systemic

component, specifically the vasomotion (Mayer's waves) around 0.1 Hz. Doing so, no component in this frequency range has been identified, probably due to the narrowness of our band-pass filter.
<figure><img src="image_97.png" /><figcaption>**FIGURE 3** | Pipeline of fNIRS data analysis. The final results are the average of the matrix containing all correlation coefficients between channels per condition and per participant. The 32 × 32 matrices presented at the end are an example of result for one participant. Avg, average.</figcaption></figure>

### 2.5.4. Functional Connectivity Analyses

The functional connectivity analyses were also performed with MATLAB. An illustration of the analyses can be found in the **Figure 3**. First, the preprocessed signal was segmented for each condition (Resting-State, SF monotask, and SF multitask) and for each participant, using the entire 600 s of duration. Therefore, a total of 6,000 samples (10 Hz × 600 s) was obtained for each participant and each condition. Functional connectivity (FC) was calculated on HbO only, which has been shown to have better signal-to-noise ratio (Pinti et al., 2019). By computing Pearson's correlations between all channels, we obtained a coefficient for

each pair in a C × C matrices (32 × 32 channels) for each condition and each participant. We then computed the average r coefficient of each resulting global matrix. This latest measure corresponds to the average functional connectivity within the FPN and is obtained for each participant and each condition (Resting-State, SF monotask, SF multitask). This dependent variable has been used for hypothesis testing.

## 2.6. Statistical Analyses

Statistical analyses were performed using the R studio software (R v. 4.0.5; Allaire, 2012; R Core Team, 2017). Two types of statistical analysis were used to determine the relationship between variables: Pearson's correlations and robust regressions. Significant level has been set to *p* < 0.05 and *post-hoc p*-values were corrected for multiple comparisons when necessary using a Holm-Bonferroni procedure (Abdi, 2010).

# 3. RESULTS

## 3.1. Functional Connectivity

The average functional connectivity within the FPN was computed for each condition at the group level (Figure 4). During the RS, the average *r* of the FPN within our sample was 0.553 (*sd* = 0.105, min = 0.378, max = 0.769). During the SF multitask version, the average *r* was .496 (*sd* = 0.140, min = 0.220, max = 0.751). During the SF multitask version, the average *r* was .506 (*sd* = 0.114, min = 0.240, max = 0.694). The distribution of the average functional connectivity within the FPN were not different from a normal distribution according to Shapiro-Wilk's tests for all conditions (all *p* > 0.05).

## 3.2. Space Fortress Performance and Video Game Expertise

### 3.2.1. Space Fortress Performance

#### 3.2.1.1. Space Fortress Total Scores

The distribution of SF scores were not different from a normal distribution according to Shapiro-Wilk's tests, both in SF monotask (*p* = 0.50) and SF multitask (*p* = 0.91) versions. In the monotask version, the mean score was 3,697 (*sd* = 2,649); and 3,373 (*sd* = 2,961) in the multitask version. A Pearson's correlation between both version revealed a significant and positive relationship (*r* = 66, *p* < 0.001). However, this significant correlation between both versions should be interpreted with caution. Indeed, one should note that the way to obtain points is very different in the two versions. In the monotask version, the most efficient way to obtain points is to destroy the fortress (mean destroyed fortress in monotask = 16.6, *sd* = 9.7). In the multitask version, the participants need to perform additional sub-tasks (mines and bonuses), and therefore get less points from the fortress task (mean destroyed fortress in multitask = 7.3, *sd* = 5.3), but get more points from the additional sub-tasks.

#### 3.2.1.2. Space Fortress Multitask Sub-scores

The four sub-scores in the multitask version were calculated according to the **Table 2**, and for only 29 participants (three participants were discarded from these analyses due to missing
A

Avg r = .533

<figure><img src="image_98.png" /></figure>

B

Avg r = .496

<figure><img src="image_99.png" /></figure>

C

Avg r = .506

<figure><img src="image_100.png" /></figure>

**FIGURE 4** | Mean functional connectivity matrices (32 × 32) during the three experimental conditions. (A) Resting-state, (B) Space Fortress monotask, (C) Space Fortress multitask. Avg, average; RH, right hemisphere; LH, left hemisphere; Fr, frontal; Par, parietal.

files). The mean sub-score 1 (Flight score) was −1,570 (sd = 1,431). The mean sub-score 2 (Fortress score) was 1,827 (sd = 1,382). The mean sub-score 3 (Mine score) was 1,709 (sd = 1003). The mean sub-score 4 (Bonus score) was 840 (sd = 573).

### 3.2.1.3. Space Fortress Multitask Action Per Minute

The mean action per minute (APM) was computed as the average number of keystrokes pressed per minute during the 10 min of SF multitask. The mean APM was 114.4 (sd = 27.3). A Pearson's correlation between APM and SF multitask score revealed a strong and significant positive relationship (r = 0.864, *p* < 0.001).

### 3.2.2. Video Game Expertise

To test a possible relationship between video game expertise and SF performance, a Pearson's correlation was computed between the score in the Video Game Questionnaire and SF multitask score. It revealed a significant and positive relationship (r = 0.43, *p* = 0.013). Because this correlation showed significance, this variable has been added as a co-variable in further analyses.

## 3.3. Relationship Between Functional Connectivity and Complex Task Performance

### 3.3.1. Hypothesis 1. Resting-State Functional Connectivity Predicts Space Fortress Performance

#### 3.3.1.1. Hypothesis 1.a

A multiple robust linear regression was computed to predict SF multitask performance based on average Resting-State functional connectivity (rsFC) and video game expertise questionnaire score (VGexp). A significant regression equation was found [F<sub>(2,29)</sub> = 10.65, *p* = 0.005] with *r*<sup>2</sup> = 0.26. Participant's predicted SF multitask score is equal to -2855.6 + 8330.5 (rsFC) + 393.3 (VGexp), where rsFC is the mean r of the connectivity matrix during Resting-State (from -1 to 1) and video game expertise is the score at the video game expertise questionnaire (from 0 to

10). Therefore, participant's score increased of 833.05 for every additional 0.1 of mean r of the connectivity matrix during RS and 393.3 for each point in the video game expertise questionnaire. Both rsFC (*p* = 0.047) and VGexp (*p* = 0.007) were significant predictors of SF multitask score (Figure 5A and Table 4).

#### 3.3.1.2. Hypothesis 1.b

A multiple robust linear regression was computed to predict SF monotask performance based on average rsFC and VGexp. The regression equation was not significant [F<sub>(2,29)</sub> = 3.35, *p* = 0.187, Table 4].

### 3.3.2. Hypothesis 2. On-Task Functional Connectivity Explains Space Fortress Performance

#### 3.3.2.1. Hypothesis 2.a

A multiple robust linear regression was computed to predict SF multitask performance based on average on-task functional connectivity (otFC) during SF multitask and video game expertise (VGExp). A significant regression equation was found [F<sub>(2,29)</sub> = 14.07, *p* < 0.001] with an *R*<sup>2</sup> of .32. Participant's predicted SF multitask score is equal to -3374.4 + 10050.2 (otFC) + 397.1 (VGexp). Therefore, participant's score increased of 1005.02 for every additional 0.1 of mean r of the connectivity matrix during the task and 397.1 for each point in the video game expertise questionnaire. Both otFC (*p* = 0.005) and VGexp on the questionnaire (*p* = 0.009) were significant predictors of SF multitask score (Figure 5B and Table 4).

#### 3.3.2.2. Hypothesis 2.b

A multiple robust linear regression was computed to predict SF monotask performance based on average otFC during SF monotask and VGexp. The regression equation was not significant [F<sub>(2,29)</sub> = 2.37, *p* = 0.305, Table 4].
<figure><img src="image_101.png" /><figcaption>**FIGURE 5** | Hypothesis 1.a and 2.a results. Linear regressions between the fronto-parietal network (FPN) functional connectivity during Resting-State (A) and during Space Fortress multitask (B) as a function of multitask SF score. Note that while the *r* and *R*² are reported for these regressions only, the *p*-value is based on the robust multiple linear regression that took into account the video-game expertise co-variate. SF, space fortress; RS, resting-state.</figcaption></figure>

<table><caption>**TABLE 4** | Hypotheses and regression results.</caption><thead><tr><th>Hypothesis</th><th>Regression model</th><th>F</th><th>R<sup>2</sup></th><th>p</th><th>p (FC)</th><th>p (VGexp)</th></tr></thead><tbody><tr><td>H1.a</td><td>SF multitask score ~ rsFC + VGexp</td><td>10.65</td><td>0.26</td><td><strong>0.005**</strong></td><td><strong>0.047*</strong></td><td><strong>0.007**</strong></td></tr><tr><td>H1.b</td><td>SF monotask score ~ rsFC + VGexp</td><td>3.35</td><td>0.09</td><td>0.187</td><td>0.265</td><td>0.125</td></tr><tr><td>H2.a</td><td>SF multitask score ~ otFC + VGexp</td><td>14.07</td><td>0.32</td><td><strong>&lt;0.001***</strong></td><td><strong>0.005**</strong></td><td><strong>0.009**</strong></td></tr><tr><td>H2.b</td><td>SF monotask score ~ otFC + VGexp</td><td>2.37</td><td>0.08</td><td>0.305</td><td>0.565</td><td>0.179</td></tr></tbody></table>

SF, Space Fortress; rsFC, resting-state functional connectivity; otFC, on-task functional connectivity; VGexp, video game expertise. Bold values represent significant results. *p < 0.05; **p < 0.01; ***p < 0.001.

### 3.3.3. Space Fortress Sub-scores Contribution to Resting-State Functional Connectivity Prediction

Because a positive relationship was found between the FPN functional connectivity during RS and SF multitask performance, we investigated from *post-hoc* correlations<fnref n="4" /> to what extent the sub-scores contributed to this relationship. Note that these analyses were done on only 29 participants. The results (see **Figure 6**) gave a significant and positive correlation between the FPN mean functional connectivity and the Flight sub-score (*r* = 0.43, *p* = 0.036); a positive but non-significant correlation with the Fortress sub-score (*r* = 0.23, *p* = 0.210); and no significance with the mine and bonus sub-scores (*r* = -0.08, *p* = 1, *r* = 0.06, *p* = 1, respectively, **Figure 6**).

## 4. DISCUSSION

The main aim of this study was to investigate the neural markers of performance in a complex and semi-ecological task (the SF video game) with fNIRS functional connectivity

analyses performed during both Resting-State and on-task. We hypothesized that intrinsic functional connectivity within the FPN can predict the performance on the complex version of SF, but not on the simple one. Our results are in favor of this hypothesis as we observed a significant relationship between the intrinsic functional connectivity of the FPN and SF multitask performance. These associations were positive and significant for the complex SF version, but not significant for the simple SF version. In addition, we observed the same pattern for on-task functional connectivity and on-task performance, and this relationship was stronger using on-task functional connectivity compared to intrinsic functional connectivity.

For the purpose of this article, we developed a functional connectivity methods adapted to fNIRS data. Indeed, unlike fMRI, fNIRS does not measure whole brain activity due to the limited number of sensors and its depth resolution (i.e., 1 or 2 cm). Instead, fNIRS users need to select relevant cortical areas to be measured. According to our hypotheses, we wanted to focus on the executive functions network. We therefore targeted functional connected brain regions defined in the literature (Niendam et al., 2012): the middle and superior frontal gyri in the frontal lobe; and the inferior and superior parietal lobules in

<footnote n="4">⁴Note that the *p*-values reported in this section are corrected with the Holm-method for multiple comparisons.</footnote>
<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /><figcaption>**FIGURE 6** | Correlations between Space Fortress sub-scores and average functional connectivity of the FPN during the RS (A) mine score, (B) bonus score; (C) flight Score, (D) fortress score. SF, space fortress; RS, resting-state.</figcaption></figure>

the parietal lobe. Despite a superficial cortical regions coverage, we were able to perform a relevant measure of average functional connectivity within these fronto-parietal brain areas which could be easily transferred to future studies using similar material.

The analyses revealed a highly positively correlated network in our sample, with an average correlation (at rest or during task) of 0.5 in the whole network, ranging between 0.3 and 0.8 for all participants (no participants had a negative value). This high degree of functional connectivity within the FPN highlights that the choice of optodes positioning was coherent. It also provides additional evidence that functional activity in areas that constitute this network are correlated. This result seems robust as it has been shown with various metrics (Cole et al., 2010). Additionally, a recent fNIRS study (Baker et al., 2018) showed a high degree of coherence within this network, which have been observed during both RS and during a working-memory task.

These results underscore the fact that whole-brain measurements (i.e., measured by fMRI) are not mandatory to perform relevant functional connectivity for neuroergonomics purpose. Moreover, the methods developed here (average functional connectivity) is interesting because it uses basic

pre-processing and simple functional connectivity analyses, and can therefore be easily reproduced and adapted to other networks. Coupled with the numerous advantages of fNIRS (high portability, low cost, less susceptible to artifacts, Nam, 2020), this methodology may have widespread applicability in the fNIRS and neuroergonomics field (Gramann et al., 2017).

The main finding of this study is the significant relationship observed between intrinsic average functional connectivity of the FPN and SF multitask performance. Indeed, this neural marker at rest explained a significant part of the performance variance. This result is in accordance with previous work that highlighted the role of the FPN in high-level cognitive functions implementation. Meta-analytic evidence is in favor of a distributed large network constituted of frontal and parietal areas where functional activity supports a broad range of executive functions (Niendam et al., 2012), which are typically recruited when dealing with complex and novel tasks (Miyake et al., 2000; Friedman et al., 2008; Friedman and Miyake, 2017). Moreover, intrinsic functional connectivity of this network has already been shown to predict performance in cognitive tasks involving working memory and intelligence in an fMRI study (Cole et al., 2012).
This result is also coherent with a study that showed functional
modifications of the FPN while participants were trained with
SF (Lee et al., 2012a). As this network arguably plays a key role
in complex task performance, tasks or training methods that
modify this network may be able to also modify the efficiency
of high-level cognitive functions underpinned by this network.
To support this direction, some studies showed that SF training
benefits may be transferred to specific tasks such as the N-back
task or the Raven's matrices (Boot et al., 2010).

From the conclusions drawn by previous studies, we argue
now that the intrinsic average functional connectivity of the FPN
may reflect its efficiency in the implementation of high-level
executive functions, with better connection efficiency between
nodes (Cole and Schneider, 2007; Duncan, 2010; Cole et al., 2012;
Gong et al., 2016). This intrinsic characteristic within the network
would allow individuals to more efficiently recruit higher-level
cognitive functions while performing complex tasks such as the
SF multitask version.

Yet, our regression *post-hoc* results between the intrinsic
average functional connectivity of the FPN and SF sub-tasks
performance showed significant results solely for the flight score.
A plausible explanation could be that the FC of the FPN is
not directly related to the involvement of different executive
functions but rather to the management of them when there is a
need to dynamically adapt in order to achieve a multidimensional
task (i.e., doing the four subtasks at the same time). The
handling of the ship corresponding to the flight sub-task, should,
therefore, induce some level of multitasking itself as the player
has to control it, avoid being hit and collisions and manage
missile stock.

Another finding of this study is the positive relationship
observed between on-task average functional connectivity of
the FPN and SF multitask performance. This result was
expected and is similar to the one obtained with the intrinsic
functional connectivity. It can be explained by taking the same
rationale as above. However, it should be highlighted that
this relationship is slightly increased compared to the rest (an
explained variance of 0.19 vs. 0.14), suggesting that the FPN
functional connectivity is dynamic and may be different during
rest and during a task. Considering that the fronto-parietal
brain areas are more activated and may be more functionally
connected in participants that perform better during a cognitive
task (Niendam et al., 2012), this result is not surprising. Such
result may also be in line with advances in dynamic network
neuroscience that highlighted the complex relationship between
cognitive functions and brain networks dynamics (Preti et al.,
2017; Hartwigsen, 2018; Michel and Koenig, 2018). Recent
studies showed that the variability of functional connectivity
in frontal and parietal brain areas were related to working
memory and executive function tasks (Douw et al., 2016; Nomi
et al., 2017). Another study found that the FPN was able to
reconfigure itself (i.e., the interactions between the modules
forming this network were flexible) while performing a working-
memory task compared to rest (Braun et al., 2015). Further
studies may be able to explore and investigate this question
using both fNIRS and dynamic functional connectivity methods
(Preti et al., 2017).

Contrary to multitask, monotask performance were not
predicted by either intrinsic or on-task average functional
connectivity of the FPN. Our interpretation is that this easier
version of the game—that was created as a control task—does
not require the involvement of high-level cognitive functions
like in the more difficult multitask version. Performance at the
monostask would be therefore independent of the strength of the
connectivity within the FPN.

Our study also reveals the importance of considering more
ecological tasks as a useful bridge between fundamental and
applied research. Indeed, tasks such as SF are more complex
and engaging than conventional laboratory tasks such as the n-
back (Jaeggi et al., 2010), the Stroop (Vendrell et al., 1995), or
the Wisconsin Card Sorting Test (Anderson et al., 1991), with
the additional benefit of being more convenient to study than
ecological or real life tasks such as flying a plane or performing a
surgery. Furthermore, the precise cognitive mechanisms involved
in SF have not been precisely evidenced and need to be assessed
in future studies, although part of them are inherent to the nature
of the task (such as working memory or divided attention).

On the practical level, the assessment of a neural marker of
complex task performance is of particular interest in the context
of neuroergonomics. Even when considering the shortcomings
of the fNIRS measures and our methodology (a lower and less
precise spatial resolution than fMRI and a single averaged value
of the functional connectivity of the FPN), we were still able
to extract a relevant predictor of SF multitask performance.
Moreover, after several hours of training, SF performance is
known to reach a plateau phase (Mathewson et al., 2012). Future
longitudinal studies might consider investigating if there are
neural markers of the rapidity to attain this plateau phase, or
if there are neural markers of long-term performance in this
task, after several months without practice. This kind of research
is invaluable for the neuroergonomics field as it opens up the
possibility to extract similar markers for more complex tasks
such as piloting a plane or performing a surgery. Along with
other physiological, behavioral, and socio-cultural markers (such
as expertise and knowledge), such marker could also potentially
be used either as a predictive tool for selection purposes or as an
opportunity to elaborate individualized cognitive training.

Besides the already discussed limits, another possible
improvement would be in the sample size. The number of
remaining participants was smaller than expected (n = 32 instead
of the 41 first included). Although larger than most fNIRS
studies (Herold et al., 2018), this resulted in a moderate-powered
study, which should be a concern, especially considering the
recent reproducibility crisis in neurosciences and psychology
(Collaboration et al., 2015). Indeed, small samples tend to
produce false positives (Button et al., 2013). Having greater
statistical power in studies is a challenge that all science fields,
including neuroergonomics, need to better tackle in the future.

As a conclusion, we provided evidence that fNIRS—a portable
and accessible brain imaging tool—can be used to assess large
scale cortical networks. Intrinsic average functional connectivity
within the fronto-parietal network was found to predict complex
task performance. Moreover, both on-task and Resting-State
functional connectivity metrics seem to be sensitive enough to
predict a part of behavioral performance. This result is worth considering for the neuroergonomics field as future studies should try to use a similar method to extract the neural markers of more complex and ecological tasks.

## DATA AVAILABILITY STATEMENT

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

## ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Comité d'Éthique de la Recherche—Université de Toulouse. The patients/participants provided their written informed consent to participate in this study.

## AUTHOR CONTRIBUTIONS

QC and SS contributed to conception and design of the study. QC performed the data acquisition, statistical analyses, and wrote the

first draft of the manuscript. QC, SS, EL, and XD wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.

## FUNDING

This work was supported by the French National Research Agency (ANR) and the Defence Procurement Agency (DGA), ASTRID program [grant number ANR-17-ASTR-0005].

## ACKNOWLEDGMENTS

The authors would like to thank Kevin J. Verdière and Caroline Hamery for their help in writing and running the MATLAB and R code.

## SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnrgo.2021.718176/full#supplementary-material

## REFERENCES

* Abdi, H. (2010). Holm's sequential bonferroni procedure. *Encyclop. Res. Des.* 1, 1–8.

* Allaire, J. (2012). *Rstudio: Integrated Development Environment for R*. Boston, MA.

* Anderson, S. W., Damasio, H., Jones, R. D., and Tranel, D. (1991). Wisconsin card sorting test performance as a measure of frontal lobe damage. *J. Clin. Exp. Neuropsychol.* 13, 909–922. doi: 10.1080/01688639108405107

* Asimakopulos, J., Boychuck, Z., Sondergaard, D., Poulin, V., Ménard, I., and Korner-Bitensky, N. (2012). Assessing executive function in relation to fitness to drive: a review of tools and their ability to predict safe driving. *Austral. Occup. Ther. J.* 59, 402–427. doi: 10.1111/j.1440-1630.2011.00963.x

* Ayaz, H., Curtin, A., Mark, J., Kraft, A., and Ziegler, M. (2019). “Predicting future performance based on current brain activity: an fNIRS and EEG study,” in *2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) (Bari)*, 3925–3930. doi: 10.1109/SMC.2019.8914412

* Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., and Onaral, B. (2012). Optical brain monitoring for operator training and mental workload assessment. *Neuroimage* 59, 36–47. doi: 10.1016/j.neuroimage.2011.06.023

* Baker, J. M., Bruno, J. L., Gundran, A., Hosseini, S. H., and Reiss, A. L. (2018). fNIRS measurement of cortical activation and functional connectivity during a visuospatial working memory task. *PLoS ONE* 13:e0201486. doi: 10.1371/journal.pone.0201486

* Barron, L. G., and Rose, M. R. (2017). Multitasking as a predictor of pilot performance: validity beyond serial single-task assessments. *Milit. Psychol.* 29, 316–326. doi: 10.1037/mil0000168

* Bavelier, D., Green, C. S., Han, D. H., Renshaw, P. F., Merzenich, M. M., and Gentile, D. A. (2011). Brains on video games. *Nat. Rev. Neurosci.* 12, 763–768. doi: 10.1038/nrn3135

* Boot, W. R. (2015). Video games as tools to achieve insight into cognitive processes. *Front. Psychol.* 6:3. doi: 10.3389/978-2-88919-553-4

* Boot, W. R., Basak, C., Erickson, K. I., Neider, M., Simons, D. J., Fabiani, M., et al. (2010). Transfer of skill engendered by complex task training under conditions of variable priority. *Acta Psychol.* 135, 349–357. doi: 10.1016/j.actpsy.2010.09.005

* Boot, W. R., Blakely, D. P., and Simons, D. J. (2011). Do action video games improve perception and cognition? *Front. Psychol.* 2:226. doi: 10.3389/fpsyg.2011.00226

* Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M., and Gratton, G. (2008). The effects of video game playing on attention, memory, and executive control. *Acta Psychol.* 129, 387–398. doi: 10.1016/j.actpsy.2008.09.005

* Boot, W. R., Sumner, A., Towne, T. J., Rodriguez, P., and Anders Ericsson, K. (2017). Applying aspects of the expert performance approach to better understand the structure of skill and mechanisms of skill acquisition in video games. *Top. Cogn. Sci.* 9, 413–436. doi: 10.1111/tops.12230

* Braun, U., Schäfer, A., Walter, H., Erk, S., Romanczuk-Seiferth, N., Haddad, L., et al. (2015). Dynamic reconfiguration of frontal brain networks during executive cognition in humans. *Proc. Natl. Acad. Sci. U.S.A.* 112, 11678–11683. doi: 10.1073/pnas.1422487112

* Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. *Nat. Rev. Neurosci.* 14, 365–376. doi: 10.1038/nrn3475

* Carthey, J., de Leval, M. R., Wright, D. J., Farewell, V. T., Reason, J. T., et al. (2003). Behavioural markers of surgical excellence. *Saf. Sci.* 41, 409–425. doi: 10.1016/S0925-7535(01)00076-5

* Causse, M., Chua, Z., Peysakhovich, V., Del Campo, N., and Matton, N. (2017). Mental workload and neural efficiency quantified in the prefrontal cortex using fNIRS. *Sci. Rep.* 7, 1–15. doi: 10.1038/s41598-017-05378-x

* Cole, M. W., Pathak, S., and Schneider, W. (2010). Identifying the brain's most globally connected regions. *Neuroimage* 49, 3132–3148. doi: 10.1016/j.neuroimage.2009.11.001

* Cole, M. W., and Schneider, W. (2007). The cognitive control network: Integrated cortical regions with dissociable functions. *Neuroimage* 37, 343–360. doi: 10.1016/j.neuroimage.2007.03.071

* Cole, M. W., Yarkoni, T., Repovš, G., Anticevic, A., and Braver, T. S. (2012). Global connectivity of prefrontal cortex predicts cognitive control and intelligence. *J. Neurosci.* 32, 8988–8999. doi: 10.1523/JNEUROSCI.0536-12.2012

* Dale, G., Joessel, A., Bavelier, D., and Green, C. S. (2020). A new look at the cognitive neuroscience of video game play. *Ann. N.Y. Acad. Sci.* 1464, 192–203. doi: 10.1111/nyas.14295

* Deligianni, F., Singh, H., Modi, H., Jahani, S., Yucel, M., Darzi, A., et al. (2020). Expertise and task pressure in fNIRS-based brain connectomes. *arXiv preprint arXiv:2001.00114*

* Donchin, E. (1995). Video games as research tools: the space fortress game. *Behav. Res. Methods Instrum. Comput.* 27, 217–223. doi: 10.3758/BF03204735

* Douw, L., Wakeman, D. G., Tanaka, N., Liu, H., and Stufflebeam, S. M. (2016). State-dependent variability of dynamic functional connectivity between
frontoparietal and default networks relates to cognitive flexibility. Neuroscience 339, 12-21. doi: 10.1016/j.neuroscience.2016.09.034

Duncan, J. (2010). The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour. *Trends Cogn. Sci.* 14, 172–179. doi: 10.1016/j.tics.2010.01.004

Fairclough, S. H., Burns, C., and Kreplin, U. (2018). fNIRS activity in the prefrontal cortex and motivational intensity: impact of working memory load, financial reward, and correlation-based signal improvement. *Neurophotonics* 5:035001. doi: 10.1117/1.NPh.5.3.035001

Fan, S., Blanco-Davis, E., Zhang, J., Bury, A., Warren, J., Yang, Z., et al. (2021). The role of the prefrontal cortex and functional connectivity during maritime operations: an fNIRS study. *Brain Behav.* 11:e01910. doi: 10.1002/brb3.1910

Fellrath, J., Mottaz, A., Schnider, A., Guggisberg, A. G., and Ptak, R. (2016). Theta-band functional connectivity in the dorsal fronto-parietal network predicts goal-directed attention. *Neuropsychologia* 92, 20–30. doi: 10.1016/j.neuropsychologia.2016.07.012

Fishburn, F. A., Ludlum, R. S., Vaidya, C. J., and Medvedev, A. V. (2019). Temporal derivative distribution repair (TDDR): a motion correction method for fNIRS. *Neuroimage* 184, 171–179. doi: 10.1016/j.neuroimage.2018.09.025

Foy, H. J., Runham, P., and Chapman, P. (2016). Prefrontal cortex activation and young driver behaviour: a fNIRS study. *PLoS ONE* 11:e0156512. doi: 10.1371/journal.pone.0156512

Friedman, N. P., and Miyake, A. (2017). Unity and diversity of executive functions: Individual differences as a window on cognitive structure. *Cortex* 86, 186–204. doi: 10.1016/j.cortex.2016.04.023

Friedman, N. P., Miyake, A., Young, S. E., DeFries, J. C., Corley, R. P., and Hewitt, J. K. (2008). Individual differences in executive functions are almost entirely genetic in origin. *J. Exp. Psychol.* 137:201. doi: 10.1037/0096-3445.137.2.201

Gateau, T., Durantin, G., Lancelot, F., Scannella, S., and Dehais, F. (2015). Real-time state estimation in a flight simulator using fNIRS. *PLoS ONE* 10:e0121279. doi: 10.1371/journal.pone.0121279

Gong, D., He, H., Ma, W., Liu, D., Huang, M., Dong, L., et al. (2016). Functional integration between salience and central executive networks: a role for action video game experience. *Neural Plast.* 2016:9803165. doi: 10.1155/2016/9803165

Gramann, K., Fairclough, S. H., Zander, T. O., and Ayaz, H. (2017). Trends in neuroergonomics. *Front. Hum. Neurosci.* 11:165. doi: 10.3389/978-2-88945-203-3

Hartwigsen, G. (2018). Flexible redistribution in cognitive networks. *Trends Cogn. Sci.* 22, 687–698. doi: 10.1016/j.tics.2018.05.008

Hearne, L. J., Mattingley, J. B., and Cocchi, L. (2016). Functional brain networks related to individual differences in human intelligence at rest. *Sci. Rep.* 6, 1–8. doi: 10.1038/srep32328

Herold, F., Wiegel, P., Scholkmann, F., and Müller, N. G. (2018). Applications of functional near-infrared spectroscopy (fNIRS) neuroimaging in exercise-cognition science: a systematic, methodology-focused review. *J. Clin. Med.* 7:466. doi: 10.3390/jcm7120466

Jacques, S. L. (2013). Optical properties of biological tissues: a review. *Phys. Med. Biol.* 58:R37. doi: 10.1088/0031-9155/58/14/5007

Jaeggi, S. M., Buschkuehl, M., Perrig, W. J., and Meier, B. (2010). The concurrent validity of the n-back task as a working memory measure. *Memory* 18, 394–412. doi: 10.1080/09658211003702171

Kipp, S., Kram, R., and Hoogkamer, W. (2019). Extrapolating metabolic savings in running: implications for performance predictions. *Front. Physiol.* 10:79. doi: 10.3389/fphys.2019.00079

Kühn, S., Gallinat, J., and Mascherek, A. (2019). Effects of computer gaming on cognition, brain structure, and function: a critical reflection on existing literature. *Dialog. Clin. Neurosci.* 21:319. doi: 10.31887/DCNS.2019.21.3/skuehn

Lancaster, J., Rainey, L., Summerlin, J., Freitas, C., Fox, P., Evans, A., et al. (1997). Automated labeling of the human brain: a preliminary report on the development and evaluation of a forward-transform method. *Hum. Brain Mapp.* 5, 238–242. doi: 10.1002/(SICI)1097-0193(1997)5:4<238::AID-HBM6>3.0.CO;2-4

Lancaster, J. L., Woldorff, M. G., Parsons, L. M., Liotti, M., Freitas, C. S., Rainey, L., et al. (2000). Automated talairach atlas labels for functional brain mapping. *Hum. Brain Mapp.* 10, 120–131. doi: 10.1002/1097-0193(200007)10:3<120::AID-HBM30>3.0.CO;2-8

Lee, H., Boot, W. R., Basak, C., Voss, M. W., Prakash, R. S., Neider, M., et al. (2012a). Performance gains from directed training do not transfer to untrained tasks. *Acta Psychol.* 139, 146–158. doi: 10.1016/j.actpsy.2011.11.003

Lee, H., Voss, M. W., Prakash, R. S., Boot, W. R., Vo, L. T., Basak, C., et al. (2012b). Videogame training strategy-induced change in brain function during a complex visuomotor task. *Behav. Brain Res.* 232, 348–357. doi: 10.1016/j.bbr.2012.03.043

Liu, H., Yu, H., Li, Y., Qin, W., Xu, L., Yu, C., et al. (2017). An energy-efficient intrinsic functional organization of human working memory: a resting-state functional connectivity study. *Behav. Brain Res.* 316, 66–73. doi: 10.1016/j.bbr.2016.08.046

Maclin, E. L., Mathewson, K. E., Low, K. A., Boot, W. R., Kramer, A. F., Fabiani, M., et al. (2011). Learning to multitask: effects of video game practice on electrophysiological indices of attention and resource allocation. *Psychophysiology* 48, 1173–1183. doi: 10.1111/j.1469-8986.2011.01189.x

Mané, A., and Donchin, E. (1989). The space fortress game. *Acta Psychol.* 71, 17–22. doi: 10.1016/0001-6918(89)90003-6

Markett, S., Reuter, M., Montag, C., Voigt, G., Lachmann, B., Rudorf, S., et al. (2014). Assessing the function of the fronto-parietal attention network: insights from resting-state fMRI and the attentional network test. *Hum. Brain Mapp.* 35, 1700–1709. doi: 10.1002/hbm.22285

Martínez, K., Solana, A. B., Burgaleta, M., Hernández-Tamames, J. A., Álvarez-Linera, J., Román, F. J., et al. (2013). Changes in resting-state functionally connected parietofrontal networks after videogame practice. *Hum. Brain Mapp.* 34, 3143–3157. doi: 10.1002/hbm.22192

Mathewson, K. E., Basak, C., Maclin, E. L., Low, K. A., Boot, W. R., Kramer, A. F., et al. (2012). Different slopes for different folks: alpha and delta EEG power predict subsequent video game learning rate and improvements in cognitive control tasks. *Psychophysiology* 49, 1558–1570. doi: 10.1111/j.1469-8986.2012.01474.x

Michel, C. M., and Koenig, T. (2018). EEG microstates as a tool for studying the temporal dynamics of whole-brain neuronal networks: a review. *Neuroimage* 180, 577–593. doi: 10.1016/j.neuroimage.2017.11.062

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., and Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: a latent variable analysis. *Cogn. Psychol.* 41, 49–100. doi: 10.1006/cogp.1999.0734

Nam, C. S. (ed.). (2020). *Neuroergonomics: Principles and Practice*. Springer Nature.

Niendam, T. A., Laird, A. R., Ray, K. L., Dean, Y. M., Glahn, D. C., and Carter, C. S. (2012). Meta-analytic evidence for a superordinate cognitive control network subserving diverse executive functions. *Cogn. Affect. Behav. Neurosci.* 12, 241–268. doi: 10.3758/s13415-011-0083-5

Nomi, J. S., Vij, S. G., Dajani, D. R., Steimke, R., Damaraju, E., Rachakonda, S., et al. (2017). Chronnectomic patterns and neural flexibility underlie executive function. *NeuroImage* 147, 861–871. doi: 10.1016/j.neuroimage.2016.10.026

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. *Science* 349:aac4716. doi: 10.1126/science.aac4716

Palaus, M., Marron, E. M., Viejo-Sobera, R., and Redolar-Ripoll, D. (2017). Neural basis of video gaming: a systematic review. *Front. Hum. Neurosci.* 11:248. doi: 10.3389/finhum.2017.00248

Pinti, P., Scholkmann, F., Hamilton, A., Burgess, P., and Tachtsidis, I. (2019). Current status and issues regarding pre-processing of fNIRS neuroimaging data: an investigation of diverse signal filtering methods within a general linear model framework. *Front. Hum. Neurosci.* 12:505. doi: 10.3389/finhum.2018.00505

Pollonini, L., Bortfeld, H., and Oghalai, J. S. (2016). Phoebe: a method for real time mapping of optodes-scalp coupling in functional near-infrared spectroscopy. *Biomed. Opt. Exp.* 7, 5104–5119. doi: 10.1364/BOE.7.005104

Preti, M. G., Bolton, T. A., and Van De Ville, D. (2017). The dynamic functional connectome: state-of-the-art and perspectives. *Neuroimage* 160, 41–54. doi: 10.1016/j.neuroimage.2016.12.061

R Core Team (2017). *R: A Language and Environment for Statistical Computing*. Vienna: R Foundation for Statistical Computing.

Rabbitt, P., Banerji, N., and Szymanski, A. (1989). Space fortress as an IQ test? Predictions of learning and of practised performance in a complex interactive video-game. *Acta Psychol.* 71, 243–257. doi: 10.1016/0001-6918(89)90011-5

Santosa, H., Zhai, X., Fishburn, F., and Huppert, T. (2018). The NIRS brain analyzir toolbox. *Algorithms* 11:73. doi: 10.3390/a11050073
Scannella, S., Peysakhovich, V., Ehrig, F., Lepron, E., and Dehais, F. (2018). Assessment of ocular and physiological metrics to discriminate flight phases in real light aircraft. *Hum. Fact.* 60, 922–935. doi: 10.1177/0018720818787135

Scholkmann, F., and Wolf, M. (2013). General equation for the differential pathlength factor of the frontal human head depending on wavelength and age. *J. Biomed. Opt.* 18:105004. doi: 10.1117/1.JBO.18.10.105004

van Dam, W. O., Decker, S. L., Durbin, J. S., Vendemia, J. M., and Desai, R. H. (2015). Resting state signatures of domain and demand-specific working memory performance. *Neuroimage* 118, 174–182. doi: 10.1016/j.neuroimage.2015.05.017

Vendrell, P., Junqué, C., Pujol, J., Jurado, M. A., Molet, J., and Grafman, J. (1995). The role of prefrontal regions in the stroop task. *Neuropsychologia* 33, 341–352. doi: 10.1016/0028-3932(94)00116-7

Verdière, K. J., Roy, R. N., and Dehais, F. (2018). Detecting pilot's engagement using fNIRS connectivity features in an automated vs. manual landing scenario. *Front. Hum. Neurosci.* 12:6. doi: 10.3389/fnhum.2018.00006

Voss, M. W., Prakash, R. S., Erickson, K. I., Boot, W. R., Basak, C., Neider, M. B., et al. (2012). Effects of training strategies implemented in a complex videogame on functional connectivity of attentional networks. *Neuroimage* 59, 138–148. doi: 10.1016/j.neuroimage.2011.03.052

Xu, J., Yin, X., Ge, H., Han, Y., Pang, Z., Tang, Y., et al. (2015). Attentional performance is correlated with the local regional efficiency of intrinsic

brain networks. *Front. Behav. Neurosci.* 9:200. doi: 10.3389/fnbeh.2015.00200

Yücel, M. A., Lühmann, A. v., Scholkmann, F., Gervain, J., Dan, I., Ayaz, H., et al. (2021). Best practices for fNIRS publications. *Neurophotonics* 8:012101. doi: 10.1117/1.NPh.8.1.019802

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

**Publisher's Note:** All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Copyright © 2021 Chenot, Lepron, De Boissezon and Scannella. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
<figure><img src="image_106.png" /></figure>

ELSEVIER

Contents lists available at ScienceDirect

Behavioural Brain Research

journal homepage: www.elsevier.com/locate/bbr

<figure><img src="image_107.png" /></figure>

Behavioural Brain Research

Research report

# Inattentional deafness to auditory alarms: Inter-individual differences, electrophysiological signature and single trial classification

<figure><img src="image_108.png" /></figure>

Check for
updates

F. Dehais*, R.N. Roy, S. Scannella

ISAE-SUPAERO, Université de Toulouse, France

## ARTICLE INFO

**Keywords:**

- Inattentional deafness
- Electroencephalography
- Event related potential
- Visual dominance over hearing
- Single trial classification
- Auditory alarm
- Flight simulator

## ABSTRACT

Inattentional deafness can have deleterious consequences in complex real-life situations (e.g. healthcare, aviation) leading to miss critical auditory signals. Such failure of auditory attention is thought to rely on top-down biasing mechanisms at the central executive level. A complementary approach to account for this phenomenon is to consider the existence of visual dominance over hearing that could be implemented via direct visual-to-auditory pathways. To investigate this phenomenon, thirteen aircraft pilots, equipped with a 32-channel EEG system, faced a low and high workload scenarii along with an auditory oddball task in a motion flight simulator. Prior to the flying task, the pilots were screened to assess their working memory span and visual dominance susceptibility. The behavioral results disclosed that the volunteers missed 57.7% of the auditory alarms in the difficult condition. Among all evaluated capabilities, only the visual dominance index was predictive of the miss rate in the difficult scenario. These findings provide behavioral evidences that other early cross-modal competitive process than top down modulation process could account for inattentional deafness. The electrophysiological analyses showed that the miss over the hit alarms led to a significant amplitude reduction of early perceptual (N100) and late attentional (P3a and P3b) event-related potentials components. Eventually, we implemented an EEG-based processing pipeline to perform single-trial classification of inattentional deafness. The results indicate that this processing chain could be used in an ecological setting as it led to 72.2% mean accuracy to discriminate missed from hit auditory alarms.

## 1. Introduction

Since the pioneering work of Cherry [1] on dichotic listening, several studies have confirmed and expanded the findings that fully perceptible auditory stimuli can remain undetected under perceptual and attentional demanding settings. Echoing elegantly with the famous inattentional blindness paradigm of Simons and Levins [2], Dalton and Fraenkel [3] have shown that participants might experience inattentional deafness by failing to notice the sentence “I am a Gorilla” while attending to an auditory conversation. The occurrence of this phenomenon has also been demonstrated in music listening, whereby some listeners were unable to report a salient electric guitar solo embedded in the XIXth century “Thus Spoke Zarathustra” lyrics poem [4]. Cross-modality interactions are also known to induce inattentional deafness and drive some relevant research on this topic. Accordingly, there is now a solid corpus of evidence that the detection of auditory cues may be impaired when engaged under visually demanding settings [5–7,67].

These studies provide valuable explanations to conceptualize attentional failures to auditory alarms that have been reported in complex

real-life situations [8]. This is particularly true in aviation whereby safety analyses have reported absence of response to auditory alarms as a causing factor to several accidents [9,10]. For instance, the co-pilot of the ill-fated Air France flight 447 from Rio de Janeiro continued to put the aircraft into a steep climb instead of descending, despite more than 70 audible stall warnings [11]. Some experiments conducted in realistic flight simulators [12–14] and in actual flight conditions [15] confirmed that inattentional deafness could indeed occur in the cockpit at an early perceptual [16,17] or at a later attentional stage [18].

It is generally admitted that the existence of limited cognitive resources at a central level may account for transient attentional impairments [19–22] such as inattentional deafness [6]. Accordingly, individuals with a higher pool of central resources – as measured by working memory span – tend to exhibit better divided and sustained attentional abilities [23,24] and should be more likely to detect unexpected events during highly demanding tasks. However, the authors reported an absence of correlation between individual working memory capacity and inattentional deafness [25]. The latter concluded that their results appealed in favor of the attentional set theory [26] which

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: frederic.dehais@isae.fr (F. Dehais), sebastien.scannella@isae.fr (S. Scannella).</footnote>
stipulates that only task relevant stimuli are attended to and consequently processed. One could envisage the implementation of shielding mechanisms controlled at a central level to save the future efforts required to perform the task at hand and hence avoid resource depletion [22,27]. These mechanisms described by Hancock and Warm [28] and Hockey [29], may lead one to think that the brain enters a “fail-safe mode”, limiting the access to the pool of resources to process unexpected signals such as auditory ones.

A complementary hypothesis that has not yet been considered to account for inattentional deafness could be linked to the existence of visual dominance over hearing mechanisms [30–32]. Accordingly, a recent experiment reported a superior ability to inhibit irrelevant spatial auditory distractors when processing visual targets than the opposite [33]. The authors argued that the existence of direct visuo-auditory pathways [34–36] could underpin the modulation of auditory processing. Since flying mainly involves the processing of visual cues (e.g. gauges, out of the window environment), it is more likely that pilots are biased to rely on these latter than on auditory ones when facing critical situations [37]. To the authors’ best knowledge, no study has yet attempted to validate the hypothesis of such interactions between visual and auditory modalities, especially in ecological settings.

Beyond the understanding of the mechanisms underpinning inattentional deafness, there is a need to implement online mental state monitoring based on neurophysiological measures to detect the occurrence of this phenomenon. Tremendous progress has been achieved using cerebral measures to infer cognitive state using processing pipelines called passive brain-computer interfaces (pBCIs; [74]). In laboratory settings, EEG-based passive BCIs have enabled researchers to accurately estimate various mental states of interest for transportation applications, such as mental fatigue and cognitive workload [69]. Previous research indicated in particular that this approach successfully led to classify auditory processing at the single trial level in an oddball paradigm (e.g. frequent versus rare sounds; [38]) and in an absent versus present auditory sound paradigm under flight simulator and real flight settings [39].

In the present paper, we report the results from a study dedicated to 1) assess the visual dominance over hearing hypothesis as a predictor of failure of auditory attention, 2) to identify electrophysiological correlates of inattentional deafness and 3) to implement a passive BCI to detect alarm misperception. To meet these goals, the volunteers were asked for their flight experience (number of flying hours) and screened with two cognitive tests dedicated to assess their working memory span and their visual dominance over hearing index respectively. Pilots were then placed in a motion flight simulator and faced low and a high workload flying scenarios while responding to rare auditory targets (Oddball paradigm). In the low workload scenario, the participants had to supervise the flight trajectory controlled by the auto-flight system. In the high workload scenario, they had to perform a critical landing with no visibility and smoke in the cabin to simulate a fire. An expert pilot, silently observing the participants, was also left seated in the simulator as an additional stressor. We hypothesized that the difficult scenario, combining high visual task demand and psychological stress, would affect both early and late auditory processing to an extent that it would yield to a high auditory alarm miss rate. This latter point was of importance so as to conduct electrophysiological-based sound detection analyses and to train a classifier to detect inattentional deafness.

# 2. Material and method

## 2.1. Participants

Thirteen healthy male pilots (mean age = 26.3 years, SD = 5.2; flight experience = 81.1 h, SD = 43.8), all French defense staff from Institut Supérieur de l'Aéronautique et de l'Espace (ISAE-SUPAERO) campus, were recruited by local advertisement and did not receive any payment for their participation. They all reported normal or corrected-

<figure><img src="image_109.png" /><figcaption>Fig. 1. The N-back task – stimuli were either letters displayed in the center of the 3\*3 grid or a moving square. Here the condition is “N-1” back.</figcaption></figure>

to-normal vision and normal audition. Typical total duration of a subject's session (informed consent approval, practice task, and real task) was about two hours and a half. This work was approved by the Inserm Committee of Ethics Evaluation (Comité d'Evaluation Ethique de l'Inserm—IRB00003888 - 18-460).

## 2.2. Working memory and visual dominance assessment

The subjects were asked to perform two neuropsychological tests. These tests consisted of a working memory test (N-back task), and a spatial audiovisual conflict test to derive a visual over hearing dominance index.

### 2.2.1. N-back task

This test has been applied using the neuropsychological testing battery PEBL® [75]. The stimuli were presented on a computer screen, and consisted in the appearance of successive letters every 3 s in the center of the screen along with the same letter pronounced in a headphone (see Fig. 1). The participants had to press the left "Shift" key when the current letter was identical to the N-2 one (2-back). The subject then had to repeat the experiment with no more letters, but a square moving in a 3 \* 3 square grid. When the square appeared twice in the same box, the subject had to press the right "Shift" key.

### 2.2.2. Spatial audiovisual conflict task

The task design was adapted from Scannella et al. (2015). Stimuli were delivered with Presentation software (Neurobehavioral system). Auditory stimuli (i.e. 1000 Hz normalized pure tones at 78 dB SPL) were presented using binaural headphones and visual stimuli (i.e. filled white circles of 2-degree diameter), were presented at a constant angle of 15° on the left or the right of a white central fixation cross on a MSI 17" monitor placed one meter in front of the participant. Auditory and visual stimuli were presented simultaneously during 200 ms and were either on the same side (i.e. congruent trials), or on opposite sides (i.e. incongruent trials; Fig. 2). The inter-trial interval was set to 2100 ms with a 500-ms jitter, while the white fixation cross remained always visible. Behavioral responses (accuracy and reaction times) were recorded with a 2-button mouse (left button for left target; right button for right target) across two blocks. In one block, participants had to detect the presentation side of the visual stimuli while ignoring auditory distractors; in the other block, they had to detect the auditory stimuli while ignoring visual distractors. In both blocks, they had to focus on the central fixation cross. The presentation order of the two blocks was counterbalanced across participants. For each block, 60 congruent and 60 incongruent trials were presented resulting in two 5-minute blocks.
<figure><img src="image_110.png" /><figcaption>**Fig. 2.** Spatial audiovisual conflict task. Stimuli (sounds and white circles) were presented either on the same side (congruent trials) or on opposite sides (incongruent trials).</figcaption></figure>

## 2.3. Flight simulator

The ISAE three-axis motion (roll, pitch, and height) flight simulator designed by the French Flight Test Center was used to conduct the experiment (Fig. 3). It simulates a twin-engine aircraft flight model and the user interface is composed of a Primary Flight Display, a Navigation Display, and an upper Electronic Central Aircraft Monitoring Display page. The flight simulator is equipped with classical actuators such as the sidestick, the rudder, the throttle, the flaps levers and a complete autopilot to control the flight. Two stereophonic speakers, located under the displays on each side of the cabin, were used to broadcast continuous radio communications, engine sounds (77 dB SPL), and to trigger the oddball sounds (90 dB SPL).

## 2.4. Flying scenarii

The participants performed one low and one high workload scenarii along with a classical oddball paradigm with a total of 400 auditory

stimuli: 20% were targets (i.e. 80 sounds) and 80% were non-targets (i.e. 320 sounds). Two types of sounds were used: one pure tone at 1000 Hz and one pure tone at 1100 Hz, both normalized at 90 dB SPL. Half of the volunteers had the 1000 Hz tone as a target and the 1100 Hz tone as a distractor whereas the other half had the opposite. Stimuli were presented through the simulator sound system at a random time interval of 2–5 s (mean = 3.5 s) resulting in a total of two 24-minute sessions. The volunteers had to ignore the frequent non-targets and used the trigger of the side-stick to respond to the auditory targets only. The order of the two scenarii was counterbalanced across participants. The participants were expressly asked not to prioritize on a specific task (flying or responding to alarms) and that their performance level was calculated based on both their flying accuracy and hit rate. A former air force pilot was right seated during the two flying conditions. He did not perform any actions and was silently observing the participants. These latter were told that this expert pilot's judgement was used to evaluate their behavior and performance. The presence of the expert pilot was used as an additional stressor to induce social pressure [40]. Indeed, the

<figure><img src="image_111.png" /><figcaption>**Fig. 3.** ISAE-SUPAERO three axis motion flight simulator. The participants were left seated and equipped with a Biosemi 32-electrode EEG system.</figcaption></figure>
feeling of being observed and evaluated by one's peers is known to increase anxiety [41,42] that in return negatively affect attentional control [43]. It was expected that this deleterious effect on attentional abilities would be magnified in the high workload scenario and contribute to induce inattentional deafness.

### 2.4.1. Low workload scenario

This scenario was the reference flight. In this experimental condition, the autopilot was engaged to level off the plane at a constant speed. The only task for the pilot was to respond to the auditory target stimuli while supervising the flight trajectory.

### 2.4.2. High workload scenario

The pilot had to perform a night approach and landing on the runway 14R at Toulouse Blagnac airport (France) while facing a cabin fire that was simulated using a Power Lighting Fogburst 600 W generator and a flashing red light. The aircraft position was initialized 20 nautical miles (nm) from the airport at an altitude of 4000 feet, a heading of 310° and a speed of 170 knots (kts). The volunteers had to steer 200° while descending to an altitude of 3000 feet. When reaching a 12-nm distance from the airport, participants had to turn to the west (heading 270°) until they intercepted the runway axis using the instrument landing system. Once intercepted, the pilots had to take a heading of 144° in order to line up with the center line of the runway. At 5 nm from the landing ground, they had to reduce speed to 130 knots to initiate the final descent. The presence of an airplane on the runway required the pilots to perform a go around and to circle around at an altitude of 2500 feet to realign on the runway axis and try to land again. The visibility was bad, causing the airport to appear and disappear several times between the cloud layers.

## 2.5. Protocol

Once the participants were told about the purpose of the experiment and signed the informed consent, they first started to complete the two cognitive tasks (N-back task and Spatial audiovisual conflict task). The order of these two cognitive tasks was counterbalanced across participants. The volunteers were then introduced to the flight instructor who trained them for a 30-minute session to handle the simulator and to perform several manual approaches and landings. The participants were also trained to perform the oddball task for 5 min. After the training was completed, the EEG and the ECG electrodes were respectively placed on the volunteers' head and torso. The experiment was eventually started: the simulator motion was engaged to reproduce realistic flight sensations, and a continuous radio communication was also broadcasted to reproduce more ecological flight conditions. The participants had to fly the two scenarios in a random order under the silent supervision of the flight instructor.

## 2.6. Data acquisition and processing

EEG data were recorded continuously with the BioSemi ActiveTwo EEG system (BioSemi, Amsterdam) from 32 active Ag-AgCl scalp electrodes positioned according to the International 10/20 system, at a 512 Hz sampling rate and with a 0–104 Hz band-pass filter. During the experiment, electrode offsets were kept under 20 mV as recommended by the manufacturer. The data were then re-referenced offline to the algebraic average of the left and right mastoids, down-sampled to 500 Hz, and filtered with a band-pass FIR filter of 0.1–40 Hz. An independent component analysis was performed using the RUNICA function with EEGLab (13.4.4b version) to isolate and reject eye blinks and movements. Data were later segmented into 1200 ms epochs starting 200 ms before the onset of each sound. Then, ERPs were computed with a baseline correction using the first 200 ms (i.e. pre-stimulus activity).

Two external EEG channels were additionally used to measure the

heart rate (EX7 placed on the left clavicle and EX8 placed on the left ribs of the participants). The R-R intervals of the raw ECG signal were then detected using the build-in QRS detection algorithm of Kubios HRV software [70]. All the recordings were manually revised for missed or false positive R peak detections. We eventually computed the average Heart Rate (HR, in beat per minute) for each scenario and participant.

## 2.7. Analyses

Electrophysiological statistical analyses were carried out using the built-in EEGLab permutation test. P-values were adjusted using the false discovery rate (FDR) corrections. All other statistical analyses were carried out using Statistica (V10, StatSoft). The p-value threshold for significance has been set to 0.05 if not otherwise mentioned. When appropriate, post-hoc comparisons have been carried out using the Tukey's Honestly Significant Difference test. Correlational analyses were conducted using Pearson correlation test.

### 2.7.1. Flight performance

Participants' ability to succeed in landing the airplane was used as a binary performance index. Indeed, the scenario was designed in such a way that any deviation from the flightpath would lead the pilots not to reach the runway threshold on time.

### 2.7.2. Visual dominance index (Vdi)

This index was meant to represent the propensity to be less distracted by auditory information when paying attention to visual ones than the opposite [33]. It has been calculated as the reaction time cost difference between the time to detect an auditory target ($A_{inc}$) and the time to detect a visual target ($V_{inc}$) in audiovisual incongruent trials: $V_{di} = A_{inc} - V_{inc}$.

### 2.7.3. N-back accuracy

N-Back accuracy scores were averaged for each subject within each difficulty level.

### 2.7.4. Oddball accuracy

A two-tailed t-test was carried out on the number of missed auditory targets between the low and high workload scenarios to evaluate the impact of scenario difficulty on the detection rate of auditory alarms.

### 2.7.5. Auditory miss rate correlations

A linear multiple regression analysis with the flight experience, the visual dominance index and the working memory score within the high load scenario was conducted to find out which of the flying experience, the visual dominance susceptibility or the working memory ability was the most predictive of the auditory alarm miss rate.

### 2.7.6. ECG

To evaluate the impact of scenario difficulty over the heart rate as computed from the ECG signal, a two-tailed t-test was carried out on the whole flying scenario between the low and high workload scenarios.

### 2.7.7. Group ERP analyses

Point-by-point permutation tests from EEGLAB (v13.4.4b) were used to analyze Hit (i.e. correctly detected target) versus Miss (i.e. undetected target) ERP component amplitudes for all 32 electrodes.

### 2.7.8. ERP classification analyses

A classification analysis was performed in order to determine whether alarm misperception could be detected in a reliable fashion. The main idea is to train a learning algorithm on a portion of the data and then test it on the remaining data. Here we focused our work on a single-trial classification, i.e. estimating from one single ERP whether the alarm is missed or detected by the pilot. The processing chain – that
is to say the various algorithm parts- used to perform hit versus miss estimation was based on the ERPs of the target sounds and is described hereafter. Initially, the first 500 ms of the auditory ERPs were corrected for ocular artifacts in an automated fashion using the SOBI algorithm (Second Order Blind Identification) and the vertical EOG (Electrooculographic) signal. The two sources that were most correlated with the EOG activity were cancelled out. Next, the cleaned data were decimated to 100 Hz and centered on zero. Then, they were spatially filtered using a Canonical Correlation Analysis (CCA) filtering process that is shown to increase discriminability for ERP-based BCIs [69,71]. Two CCA filters were used. Hence the features consisted of a vector of 100 points (2 filters*50 ERP time points). Lastly, these features were classified using a Fisher Linear Discriminant Analysis with a shrinkage estimation of the covariance matrices [72]. This was performed using a 10-fold cross-validation procedure in which an equal number of hits and misses were systematically drawn to create the training (9 out of 10 subsets) and the testing sets (10th subset).

# 3. Results

## 3.1. Behavioral and physiological results

Scenario effect: We found that the target sound detection accuracy was significantly affected by the scenario load with only 0.33% (± 0.47) of missed targets in the low load scenario compared to 57.73% (± 12.63) in the high load one (t = 15.73, p < 0.001, Cohen's d = -6.44) (Fig. 4).

### 3.1.1. Flying performance

All the pilots managed to land the aircraft on the runway prior to the end of the experiment.

### 3.1.2. Linear regression analysis

Among the three auditory miss predictors that were tested (Fig. 5), the visual dominance index was the only one that significantly correlated with the percentage of missed auditory targets during the high load scenario (semi-partial r = 0.57; t = 2.60; p < 0.05). This correlation showed that the more the pilots can be distracted by the visual distractor while responding to the auditory target in the audiovisual conflict task, the higher the number of missed auditory alarms in the high load flying scenario. The working memory ability and the flying

experience correlated only poorly and non-significantly with the number of missed auditory targets (semi-partial r = -0.16; t = 1.48; p = 0.47 and semi-partial r = -0.32; t = 0.75; p = 0.17 respectively).

### 3.1.3. ECG results

The statistical analyses disclosed a significant effect of the scenario difficulty over the cardiac activity t = 2.68; p < 0.05. Cohen's d = 1.06) with a higher average HR in the high workload scenario (mean = 89.5, SD = 17.2) than in the low workload one (mean = 72.8, SD = 14.3).

## 3.2. Electrophysiological results

In the high load flying scenario, both auditory targets and distributors have elicited event related potentials with different components. Among them we found characteristic auditory-related exogenous (N100/P200) and endogenous (P3a and P3b) components (see Fig. 6). The N100 reached its maximum mean amplitude around 116 ms with a fronto-central scalp distribution. The mean hit-related N100 amplitude was significantly larger than the miss-related one (hit: -7.92 µV, miss: -5.49 µV; p < 0.05, Cohen's d = 0.65). Similarly, the P3a component, with a centro-parietal distribution was maximum at 370 ms for the hit auditory targets and larger than the miss-related ones (hit: 2.48 µV, miss: 0.65 µV; p < 0.05, Cohen's d = 0.62). Finally, the P3b component amplitude was also affected by the detection type and led to a maximum amplitude 450 ms after the stimulus onset in the parieto-occipital region. Its amplitude was significantly larger for the detected sounds (2.99 µV) than for the missed ones (-0.25 µV; p < 0.001, Cohen's d = 1.28).

#### 3.2.1. Single trial inattentional deafness classification

The classification pipeline that was used allowed us to obtain 72.2% of correct classification of the hit and missed targets in average across participants. This is significantly higher than the adjusted chance level threshold of 59%, as computed to take into account the number of available trials following Combrisson and Jerbi's recommendations [44]. Fig. 7 displays the spatial patterns of the filters used to enhance the discriminability between the two classes (i.e. hit and miss). As mainly illustrated by the first filter's patterns, the electrode sites that enable such a high classification accuracy are located at fronto-central sites.

<figure><img src="image_112.png" /><figcaption>Fig. 4. Scenario effect over the oddball auditory target detection. The dots represent the mean accuracy over the group, the grey box and the whiskers respectively represent the standard deviation and the 95% confidence interval.</figcaption></figure>
<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /><figcaption>Fig. 5. Oddball accuracy during the high load scenario as a function of a. the visual dominance index in the spatial audiovisual conflict task, b. the working memory ability and c. the flight experience. Semi-partial correlations are reported.</figcaption></figure>

<figure><img src="image_116.png" /><figcaption>Fig. 6. Group ERP results. a. Averaged ERPs for hit and missed auditory targets in the difficult flying scenario at Cz (left) and Pz (right) electrodes. Shapes represent the group standard deviations. Black lines at the x axis represent the significant differences between hit and miss (permutation test; p < 0.05; FDR corrected). b. 2-D topographical views for hit and missed auditory targets at 116 ms (up, N100), 370 ms (left, P3a) and 450 ms post-stimulus (right, P3b).</figcaption></figure>
<figure><img src="image_117.png" /><figcaption>Fig. 7. Spatial patterns of the two CCA filters used to enhance discriminability between classes.</figcaption></figure>

## 4. Discussions

The objective of this paper was to study the inattentional deafness phenomenon under ecological settings in the context of flying. A first research question was to identify individual specificities that could reveal evidences of visual to auditory dominance as a possible complementary mechanism to account for inattentional deafness. A second research question was to identify the electrophysiological correlates of inattentional deafness to auditory alarm in the cockpit. Eventually, a last question was to assess the reliability of an off-line processing pipeline dedicated to detect alarm misperception using electrophysiological responses.

To meet this goal, the participants had to face two contrasted flying scenarios in terms of difficulty while responding to auditory alarms. The high workload scenario combined several stressors such as high cognitive demand (landing with no visibility and windshear), aversive stimuli (smoke in the cabin and flashing red light) and social pressure with the presence of an expert pilot. Accordingly, psychophysiological results disclosed that the HR was higher during the high workload scenario compared to the low workload one hence reflecting increased mental demand and psychological stress [12,45]. The behavioral results showed the efficiency of the high workload scenario to promote high rates of auditory misses (i.e. 57%) in comparison to the low workload one. These findings confirmed that primary task difficulty [5, 17, 18, 46] as well as unexpected stressful situations [14] can elicit inattentional deafness in the cockpit. Our experimental protocol can't allow us to conclude which of the stressors was the most efficient to distract the pilots from processing the auditory alarms. Indeed, as we did not manipulate them separately, it is more likely that the combination of all these stressors had the intended deleterious effects on auditory attentional abilities [43].

This high rate of inattentional deafness allowed however to conduct correlational analyses. The objective was to determine whether individual working memory, flying experience or visual dominance index would predict occurrence of inattentional deafness. As hypothesized, the working memory score did not predict the propensity to remain aware of the auditory alarms. This result is in line with previous studies that report an absence of relation between the working memory span and the occurrence of inattentional blindness [25,47–49] or inattentional deafness [25]. Though working memory capacity—as a measure of cognitive resources at a central level—has been shown to be related to individuals’ sustained and divided attentional performance [23,24,50], this construct seems not appropriated to account for inattentional deafness. In line with Kreitz et al. [25], our results rather advocate in favor of the attentional set theory [26]: cognition is goal-directed and promotes the selection and the processing of task at hand relevant stimuli. In our experiment, the participants were told that the two tasks (i.e. flying and responding to auditory alarms) were of equal importance but the behavioral results suggested that the volunteers

probably prioritized the flying task as they all managed to land the aircraft while missing at least 20% of auditory alerts in the best case. Pilots are highly trained individuals who are taught to prioritize tasks according to the “first aviate, then navigate and eventually communicate” rule. The application of this rule can explain why our volunteers may have naturally put more mental efforts on the flying task. The flying task was naturally more challenging and rewarding from a pilot’s perspective (i.e. night landing with smoke in the cabin with an experienced flight instructor on the right seat) and had immediate consequences (i.e. missing the approach and the landing) contrary to the achievement of the auditory alarm task itself. Consequently, and in accordance with the attentional set theory, the participants were more likely to process visual information in the cockpit and were less inclined to respond to auditory targets. Such behavior well as the absence of correlation between inattentional deafness rate and working memory abilities could be predicted by the Compensatory Control Model [29]. This model postulates the existence of a motivational control mechanism that dynamically modulates mental effort to shield against performance decline and resources depletion. It includes three decisions units that are dedicated to select and hierarchize goals according to their utility value (the goal regulation unit), to monitor the efficiency of the on-going strategy (performance evaluation unit) and to increase or maintain mental effort (effort regulation unit). The effort regulation unit involves a compensatory allocation of resources, eventually leading to increased level of effort budget to maintain high utility goals that are compromised to the detriment of low priority ones. Indeed, goals with high utility, such as operating the aircraft in our task, would remain in place but with a higher level of mental effort to process visual flight parameters. Thus, one has to consider that such resources allocation is a dynamic process that could not depend on a structural characteristic such as working memory.

Eventually, and in line with our hypothesis, visual dominance over hearing mechanisms could provide complementary explanations to account for this phenomenon. Indeed, the ability to respond to auditory targets was biased by inter-individual cross-modality susceptibility: the “more visual dominant” volunteers exhibited a higher miss rate than the “less visual dominant” ones in the difficult scenario. These early mechanisms could possibly be implemented via direct visual-to-auditory connections [34–36] to modulate the auditory response at the brainstem [50], the auditory cortex [33,37] and/or the auditory integrative levels [46]. The strength and the efficiency of these visuo-auditory interactions may vary among individuals according to their modality preferences that are known to impact their abilities to process auditory or visual materials [51–55].

A second motivation of this study was to identify the electrophysiological correlates of auditory alarm misperception. The high miss rate in the difficult scenario allowed performing a miss versus hit contrast. In line with Callan and collaborators [16], our findings disclose that the N100 amplitude was significantly reduced for misses
compared to hits. This result shows that inattentional deafness to au-
ditory alarms can take place at an early perceptual stage or processing
as this component is the electrophysiological signature of stimulus de-
tection and processing in the primary auditory cortices [73]. This is also
consistent with a previous inattentional deafness fMRI study, using a
repetitive single tone auditory stimulus, that disclosed lower auditory
cortex activation during auditory misses in comparison to auditory hits
[17]. We found, in addition, that he processing of the auditory alarms
was also consequently affected at later attentional stages during in-
attentional deafness events. Indeed, the amplitude of two sub-
components of the P300, namely the P3a and the P3b, were sig-
nificantly diminished for misses relative to hits. On the one hand, the
P3a, also known as early novelty P3, is thought to reflect automatic
orientation and engagement of attention towards unexpected stimuli
[56]. On the other hand, the P3b, also known as late novelty P3, has
been proposed to account for stimulus recognition involving working
memory mechanisms (pattern matching and updating) [56,57]. This is
akin to previous inattentional deafness findings reporting lower audi-
tory-related P300 [18], P3a [46] and P3b [7] amplitudes associated
with the impaired processing of deviant sounds while performing a
demanding visual task. However, these authors did not perform hit
versus miss contrast as the miss rate was too low in their studies and
thus solely reported the deleterious effect of visual load on the global
(i.e. hits and misses averaged) auditory-related P300, P3a and P3b. We
hence confirm previous results and reconcile the literature by showing
that both early and late auditory processing stages are affected during
inattentional deafness events. In accordance with other authors
[16,17], our assumption is that inattentional deafness is related to the
lack of detection of the warning taking place at the early perceptual
level (as attested by the N100 effects). Hence, subsequent processing at
later attentional stages could be affected by this early effect (i.e. P3a
and P3b attenuation). The attenuation of the P3a and P3b amplitudes
also reveal that inattentional deafness could result from an inability to
automatically shift attention to the alarm that has been correctly de-
tected or from an inability to process and recognize the warning.

Eventually, our last motivation was to perform classification of EEG
features that reflect the auditory alarm misperception. To that end, we
implemented a signal processing chain that had proven its efficiency for
auditory evoked-potentials classification [69,71]. Thanks to this chain,
we achieved a satisfying accuracy of 72.2% of correctly classified hit
versus missed targets, a significantly higher accuracy than the adjusted
chance level (i.e., 59%). These results indicate that this chain could be
used in a quite ecological setting (i.e. a full motion flight simulator) to
detect the inattentional deafness phenomenon.

Mental state estimation is a growing research interest and although
high performance is reached nowadays in the laboratories, studies re-
main to be carried out to determine solutions for ecological settings. To
our knowledge, this study is the first demonstration that inattentional
deafness could be estimated in such settings. The natural next step is to
achieve an online estimation, i.e. to estimate the operator's mental state
and potential misperception of alarms during the realization of the task.
Moreover, efforts should be spent to try and perform this estimation as
early as possible, that is to say using data that precedes the occurrence
of a stimulation (e.g. one minute or one second before the alarm is
triggered), as currently studied by Senoussi and collaborators [68]. In
this case we would predict rather than detect the occurrence of in-
attentional deafness. As a consequence, one could imagine the design of
an adaptive cockpit that would take the information of stress level and
inattention to alarms into account to implicitly adapt itself with a set of
counter-measures [58].

**5. Conclusion and limitation of this study**

This study demonstrates the importance of conducting neuroergo-
nomics experiments in ecological settings rather than in simplified la-
boratory settings. This approach contributes to the understanding of the

brain when facing critical real-life situations [15,16,59-61] and al-
lowed us to obtain high miss rates that could not be induced in la-
boratory conditions. In addition, our study is the first to report a re-
lationship between individuals' visual dominance over hearing
susceptibility and failure of attention. This result opens promising
perspectives for human operator selection and the development of
cognitive training solutions to improve auditory abilities. Taken to-
gether, the behavioral and electrophysiological results of this experi-
ment bring insights on subtle competitive mechanisms taking place
from the perceptual to the later attentional stages. This study thus
brings clues that conciliate diverging studies attributing the inatten-
tional deafness phenomenon to pre-attentive [16,37] or attentional
processes [7,18,62]. Eventually, the finding of ERPs as a neural sig-
nature of inattentional deafness shows that they are good candidates as
features to detect the occurrence of missed alarms using passive brain-
computer interfaces and could be used to design adaptive “alarms” (e.g.
dynamic modification of the alarm modality) and feedback to mitigate
transient attentional impairments.

This study was a first step toward the identification of potential
behavioral and neural correlates of inattentional deafness and the im-
plementation of a pBCI. To meet this goal, the design of our protocol
relied on a compromised between ecological and controlled laboratory
settings. This approach has several limitations though. For instance, the
participants faced a high number of auditory targets, contrarily to real
operational situations. The high rate of these auditory alarms was
needed for our ERP analyses and machine learning purposes. The alarm
sound was also not a real cockpit alarm but a brief and pure tone to be
consistent with classical auditory oddball paradigm so as to elicit ty-
pical ERPs in a timely manner. Moreover, these alarms were not related
to the flying task per se: missing the alarms had no consequences on the
flight performance although it was mentioned that the global flight
appreciation would take the alarm hit rate into account. As a matter of
face, these frequent and “non-relevant” alarms were more likely to be
ignored by the participants. Eventually, our flight simulator did not
allow collecting flight parameters. This prevented us from analyzing
possible shared attentional strategies between the flying and the audi-
tory alarm detection tasks throughout the flight. Further experiments
should integrate more realistic alarms relevant to the flying task and the
analysis of the flight performance that could also be used to improve
the classification algorithm to predict inattentional deafness. Finally,
other experiments should investigate the effect of time-on-task on in-
attentional deafness as previous studies have shown that it can nega-
tively impair auditory processing [63].

**Acknowledgements**

The research conducted by the authors and reported in this review is
supported by the AXA Research Fund through the Neuroergonomics For
Flight Safety Chair.

**References**

[1] E.C. Cherry, Some experiments on the recognition of speech, with one and with two ears, J. Acoust. Soc. Am. 25 (1953) 975.

[2] D.J. Simons, D.T. Levin, Change blindness, Trends Cogn. Sci. 1 (7) (1997) 261–267.

[3] P. Dalton, N. Fraenkel, Gorillas we have missed: sustained inattentional deafness for dynamic events, Cognition 124 (3) (2012) 367–372.

[4] S. Koreimann, B. Gula, O. Vitouch, Inattentional deafness in music, Psychol. Res. 78 (3) (2014) 304–312.

[5] J.S. Macdonald, N. Lavie, Visual perceptual load induces inattentional deafness, Atten. Percept. Psychophys. 73 (6) (2011) 1780–1789.

[6] D. Raveh, N. Lavie, Load-induced inattentional deafness, Atten. Percept. Psychophys. 77 (2) (2015) 483–492.

[7] M. Scheer, H.H. Bülthoff, L.L. Chuang, Auditory task irrelevance: a basis for inattentional deafness, Hum. Factors (2018) 0018720818760919.

[8] B. Hasanain, A.D. Boyd, J. Edworthy, M.L. Bolton, A formal approach to discovering simultaneous additive masking between auditory medical alarms, Appl. Ergon. 58 (2017) 500–514.

[9] J.P. Bliss, Investigation of alarm-related accidents and incidents in aviation, Int. J. Aviat. Psychol. 13 (3) (2003) 249–268.
[10] R.J. Mumaw, Analysis of alerting system failures in commercial aviation accidents, Proceedings of the Human Factors and Ergonomics Society Annual Meeting 61 (2017) 110–114.

[11] Bureau d'Enquêtes et d'Analyses, Final Report on the Accident on 1st June 2009 to the Airbus A330-203 Registered F-GZCP Operated by Air France Flight AF 447 Rio de Janeiro-Paris, BEA, Paris, 2012.

[12] F. Dehais, M. Causse, N. Régis, E. Menant, P. Labedan, F. Vachon, S. Tremblay, Missing critical auditory alarms in aeronautics: evidence for inattentional deafness? vol. 56, pp. 1639–1643, Presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting (2012).

[13] F. Dehais, C. Tessier, L. Christophe, F. Reuzeau, The perseveration syndrome in the pilots' activity: guidelines and cognitive countermeasures, Human Error, Safety and Systems Development, (2010), pp. 68–80.

[14] Dehais, Frédéric, M. Causse, F. Vachon, N. Régis, E. Menant, S. Tremblay, Failure to detect critical auditory alerts in the cockpit evidence for inattentional deafness, Hum. Factors J. Hum. Factors Ergon. Soc. 56 (4) (2014) 631–644.

[15] Frederic Dehais, R.N. Roy, G. Durantin, T. Gateau, D. Callan, EEG-engagement index and auditory alarm misperception: an inattentional deafness study in actual flight condition, Proceedings of the AHFE'17 (2017).

[16] D.E. Callan, T. Gateau, G. Durantin, N. Gonthier, F. Dehais, Disruption in neural phase synchrony is related to identification of inattentional deafness in real-world setting, Hum. Brain Mapp. 39 (6) (2018) 2596–2608.

[17] Durantin, Gautier, F. Dehais, N. Gonthier, C. Terzibas, D.E. Callan, Neural signature of inattentional deafness, Hum. Brain Mapp. 38 (11) (2017) 5440–5455.

[18] L. Giraudet, M.-E. St-Louis, S. Scannella, M. Causse, P300 event-related potential as an indicator of inattentional deafness? PLoS One 10 (2015) e0118556–e0118556.

[19] M. Brand-D'Abrescia, N. Lavie, Task coordination between and within sensory modalities: effects on distraction, Atten. Percept. Psychophys. 70 (3) (2008) 508–515.

[20] V. Santangelo, M. Olivetti Belardinelli, C. Spence, The suppression of reflexive visual and auditory orienting when attention is otherwise engaged, J. Exp. Psychol. Hum. Percept. Perform. 33 (1) (2007) 137.

[21] E.M. Hannon, A. Richards, Is inattentional blindness related to individual differences in visual working memory capacity or executive control functioning? Perception 39 (3) (2010) 309–319.

[22] M.N. Tombu, C.L. Asplund, P.E. Dux, D. Godwin, J.W. Martin, R. Marois, A unified attentional bottleneck in the human brain, Proc. Natl. Acad. Sci. 108 (33) (2011) 13426–13431.

[23] G.J. Colflesh, A.R. Conway, Individual differences in working memory capacity and divided attention in dichotic listening, Psychon. Bull. Rev. 14 (4) (2007) 699–703.

[24] N. Unsworth, R.W. Engle, The nature of individual differences in working memory capacity: active maintenance in primary memory and controlled search from secondary memory, Psychol. Rev. 114 (1) (2007) 104.

[25] C. Kreitz, P. Furley, D.J. Simons, D. Memmert, Does working memory capacity predict cross-modally induced failures of awareness? Conscious. Cogn. 39 (2016) 18–27.

[26] S.B. Most, B.J. Scholl, E.R. Clifford, D.J. Simons, What you see is what you set: sustained attentional blindness and the capture of awareness, Psychol. Rev. 112 (1) (2005) 217.

[27] J.B. Fritz, M. Elhilali, S.D. David, S.A. Shamma, Attentional attention — focusing the searchlight on sound, Curr. Opin. Neurobiol. 17 (2007) 1–19.

[28] P.A. Hancock, J.S. Warm, A dynamic model of stress and sustained attention, Hum. Factors 31 (1989) 519–537.

[29] G.R.J. Hockey, Compensatory control in the regulation of human performance under stress and high workload: a cognitive-energetical framework, Biol. Psychol. 45 (1) (1997) 73–93.

[30] F.B. Colavita, D. Weisberg, A further investigation of visual dominance, Atten. Percept. Psychophys. 25 (4) (1979) 345–347.

[31] S. Sinnett, C. Spence, S. Soto-Faraco, Visual dominance and attention: the Colavita effect revisited, Percept. Psychophys. 69 (5) (2007) 673–686.

[32] S. Yuval-Greenberg, L.Y. Deouell, The dog's meow: asymmetrical interaction in cross-modal object recognition, Exp. Brain Res. 193 (4) (2009) 603–614.

[33] S. Sébastien Scannella, J. Pariente, X. De Boissezon, E. Castel-Lacanal, N. Chauveau, M. Causse, et al., N270 sensitivity to conflict strength and working memory: a combined ERP and sLORETA study, Behav. Brain Res. 297 (2005) 231–240.

[34] C. Cappe, P. Barone, Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey, Eur. J. Neurosci. 22 (11) (2005) 2886–2902.

[35] E. Macaluso, J. Driver, Multisensory spatial interactions: a window onto functional integration in the human brain, Trends Neurosci. 28 (5) (2005) 264–271.

[36] Y. Wang, S. Celebrini, Y. Trotter, P. Barone, Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence, BMC Neurosci. 9 (1) (2008) 79.

[37] Scannella, Sébastien, M. Causse, N. Chauveau, J. Pastor, F. Dehais, Effects of the audiovisual conflict on auditory early processes, Int. J. Psychophysiol. 89 (1) (2013) 115–122.

[38] S. Debener, R. Emkes, M. De Vos, M. Bleichner, Unobtrusive ambulatory EEG using a smartphone and flexible printed electrodes around the ear, Sci. Rep. 5 (2015).

[39] D. Callan, G. Durantin, C. Terzibas, Classification of single-trial auditory events using dry-wireless EEG during real and motion simulated flight, Front. Syst. Neurosci. 9 (2015) 11.

[40] J.S. Shrauger, Self-esteem and reactions to being observed by others, J. Pers. Soc. Psychol. 23 (2) (1972) 192.

[41] J.U. Frisch, J.A. Häusser, A. Mojzisch, The Trier Social Stress Test as a paradigm to study how people respond to threat in social interactions, Front. Psychol. 6

(2015) 14.

[42] A.P. Allen, P.J. Kennedy, J.F. Cryan, T.G. Dinan, G. Clarke, Biological and psychological markers of stress in humans: focus on the Trier Social Stress Test, Neurosci. Biobehav. Rev. 38 (2014) 94–124.

[43] J. Allsop, R. Gray, Flying under pressure: effects of anxiety on attention and gaze behavior in aviation, J. Appl. Res. Mem. Cogn. 3 (2) (2014) 63–71.

[44] E. Combrisson, K. Jerbi, Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy, J. Neurosci. Methods 250 (2015) 126–136.

[45] F. Dehais, M. Causse, F. Vachon, S. Tremblay, Cognitive conflict in human-automation interactions: a psychophysiological study, Appl. Ergon. (2011).

[46] K. Molloy, T.D. Griffiths, M. Chait, N. Lavie, Inattentional deafness: visual load leads to time-specific suppression of auditory evoked responses, J. Neurosci. 35 (49) (2015) 16046–16054.

[47] V. Beanland, E.H.C. Chan, The relationship between sustained inattentional blindness and working memory capacity, Atten. Percept. Psychophys. 78 (3) (2016) 808–817.

[48] K. Bredemeier, D.J. Simons, Working memory and inattentional blindness, Psychon. Bull. Rev. 19 (2) (2012) 239–244.

[49] C. Kreitz, P. Furley, D. Memmert, D.J. Simons, The influence of attention set, working memory capacity, and expectations on inattentional blindness, Perception 45 (4) (2016) 386–399.

[50] P. Sörqvist, A. Nöstl, N. Halin, Working memory capacity modulates habituation rate: evidence from a cross-modal auditory distraction paradigm, Psychon. Bull. Rev. 19 (2) (2012) 245–250.

[51] M. Besson, F. Faïta, An event-related potential (ERP) study of musical expectancy: comparison of musicians with nonmusicians, J. Exp. Psychol.-Hum. Percept. Perform. 21 (1995) 1278–1296.

[52] Mireille Besson, F. Faïta, J. Requin, Brain waves associated with musical incongruities differ for musicians and non-musicians, Neurosci. Lett. 168 (1) (1994) 101–105.

[53] R. Cecere, G. Rees, V. Romei, Individual differences in alpha frequency drive crossmodal illusory perception, Curr. Biol. 25 (2) (2015) 231–235.

[54] D. Gurler, N. Doyle, E. Walker, J. Magnotti, M. Beauchamp, A link between individual differences in multisensory speech perception and eye movements, Atten. Percept. Psychophys. 77 (4) (2015) 1333–1341.

[55] A.R. Jensen, Individual differences in visual and auditory memory, J. Educ. Psychol. 62 (2) (1971) 123.

[56] J. Polich, Updating P300: an integrative theory of P3a and P3b, Clin. Neurophysiol. 118 (10) (2007) 2128–2148.

[57] A. Kok, On the utility of P3 amplitude as a measure of processing capacity, Psychophysiology 38 (3) (2001) 557–577.

[58] C. Glatz, S.S. Krupenia, H.H. Bülthoff, L.L. Chuang, Use the right Sound for the right job: verbal commands and auditory icons for a task-management system favor different information processes in the brain, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM, 2018, p. 472.

[59] T. Gateau, H. Ayaz, F. Dehais, In silico versus over the clouds: on-the-fly mental state estimation of aircraft pilots, using a functional near infrared spectroscopy based passive-BCI, Front. Hum. Neurosci. 12 (2018) 187.

[60] K. Gramann, S.H. Fairclough, T.O. Zander, H. Ayaz, Trends in neuroergonomics, Front. Hum. Neurosci. 11 (2017) 165.

[61] R. McKendrick, R. Parasuraman, H. Ayaz, Wearable functional near infrared spectroscopy (fNIRS) and transcranial direct current stimulation (tDCS): expanding vistas for neurocognitive augmentation, Front. Syst. Neurosci. 9 (2015) 27.

[62] M. Causse, J.-P. Imbert, L. Giraudet, C. Jouffrais, S. Tremblay, The role of cognitive and perceptual loads in inattentional deafness, Front. Hum. Neurosci. 10 (2016).

[63] F. Dehais, A. Duprès, G. Di Flumeri, K.J. Verdière, G. Borghini, F. Babiloni, R.N. Roy, Monitoring pilot's cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI, Proceedings of the IEEE SMC Conference (2018).

[64] N. Lavie, D.M. Beck, N. Konstantinou, Blinded by the load: attention, awareness and the role of perceptual load, Philos. Trans. Biol. Sci. 369 (1641) (2014) 20130205.

[65] M. Senoussi, K.J. Verdiere, A. Bovo, C. Ponzoni Carvalho Chanel, F. Dehais, R.N. Roy, Pre-stimulus antero-posterior EEG connectivity predicts performance in a UAV monitoring task, Proceedings of IEEE SMC Conference, Canada, 2017, pp. 1167–1172 October, 5–8.

[66] R.N. Roy, S. Charbonnier, A. Campagne, S. Bonnet, Efficient mental workload estimation using task-independent EEG features, J. Neural Eng. 13 (2) (2016) 026019.

[67] M.P. Tarvainen, J.P. Niskanen, J.A. Lipponen, P.O. Ranta-Aho, P.A. Karjalainen, Kubios HRV-heart rate variability analysis software, Comput. Methods Programs Biomed. 113 (1) (2014) 210–220.

[68] R.N. Roy, S. Bonnet, S. Charbonnier, P. Jallon, A. Campagne, A comparison of ERP spatial filtering methods for optimal mental workload estimation, Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, IEEE, 2015, pp. 7254–7257. August.

[69] B. Blankertz, S. Lemm, M. Treder, S. Haufe, K.R. Müller, Single-trial analysis and classification of ERP componentsüta tutorial, NeuroImage 56 (2) (2011) 814–825.

[70] R.F. Hink, S.T. Van Voorhis, S.A. Hillyard, T.S. Smith, The division of attention and the human auditory evoked potential, Neuropsychologia 15 (4–5) (1977) 597–605.

[71] T.O. Zander, C. Kothe, Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general, J. Neural Eng. 8 (2) (2011) 025005.

[72] S.T. Mueller, B.J. Piper, The psychology experiment building language (PEBL) and PEBL test battery, J. Neurosci. Methods 222 (2014) 250–259.
<figure><img src="image_118.png" /></figure>
<figure><img src="image_119.png" /></figure>

Research report

# N270 sensitivity to conflict strength and working memory: A combined ERP and sLORETA study

<figure><img src="image_120.png" /></figure>

CrossMark

Sébastien Scannella<sup>a,b,*</sup>, Jérémie Pariente<sup>b</sup>, Xavier De Boissezon<sup>b</sup>, Evelyne Castel-Lacanal<sup>b</sup>, Nicolas Chauveau<sup>b</sup>, Mickaël Causse<sup>a</sup>, Frédéric Dehais<sup>a</sup>, Josette Pastor<sup>b</sup>

## HIGHLIGHTS

* We create an audiovisual conflict task with simultaneous target and distractor.

* We assess the N270 sensitivity to the conflict strength and the working memory.

* The conflict cost is higher for the auditory target than for the visual target.

* The N270 is sensitive to the conflict strength and the load in working memory.

* Conflict processing and working memory share common neural subtract.

## ARTICLE INFO

**Article history:**

Received 29 April 2015

Received in revised form

12 September 2015

Accepted 5 October 2015

Available online 23 October 2015

**Keywords:**

N270

Supramodal conflict processing

Working memory

Executive attention

Dorsolateral prefrontal cortex

sLORETA

## ABSTRACT

The event-related potential N270 component is known to be an electrophysiological marker of the supramodal conflict processing. However little is known about the factors that may modulate its amplitude. In particular, among all studies that have investigated the N270, little or no control of the conflict strength and of the load in working memory have been done leaving a lack in the understanding of this component. We designed a spatial audiovisual conflict task with simultaneous target and cross-modal distractor to evaluate the N270 sensitivity to the conflict strength (i.e., visual target with auditory distractor or auditory target with visual distractor) and the load in working memory (goal task maintenance with frequent change in the target modality). In a first session, participants had to focus on one modality for the target position to be considered (left-hand or right-hand) while the distractor could be at the same side (compatible) or at opposite side (incompatible). In a second session, we used the same set of stimuli as in the first session with an additional distinct auditory signal that clued the participants to frequently switch between the auditory and the visual targets. We found that (1) reaction times and N270 amplitudes for conflicting situations were larger within the auditory target condition compared to the visual one, (2) the increase in target maintenance effort led to equivalent increase of both reaction times and N270 amplitudes within all conditions and (3) the right dorsolateral prefrontal cortex current density was higher for both conflicting and active maintenance of the target situations. These results provide new evidence that the N270 component is an electrophysiological marker of the supramodal conflict processing that is sensitive to the conflict strength and that conflict processing and active maintenance of the task goal are two functions of a common executive attention system.

© 2015 Elsevier B.V. All rights reserved.

## 1. Introduction

In the spatial domain, the human attentional system often has to integrate concurrent stimulation from different modalities (e.g.,

visual and auditory) at one single location, which helps spatial orienting [22] and allows a plausible interpretation of the world. Although this integration is efficient most of the time, the spatial discrepancy between two modalities may lead to perceptual illusions like those observed in the ventriloquism [3] where the vision (puppet's mouth movements) "captures" the sound [35]. Since the princeps work of Colavita [7], the visual dominance over other modalities and particularly over hearing in most of spatial tasks has

<footnote><sup>a</sup> DCAS, ISAE, Université de Toulouse, 10 Avenue E. Belin, 31055 Toulouse cedex 4, France</footnote>

<footnote><sup>b</sup> INSERM, UMRS 825, Université de Toulouse, CHU Purpan, Pavillon Baudot, 31024 Toulouse cedex 3, France</footnote>

<footnote>* Corresponding author at: DCAS, ISAE, 10 Avenue E. Belin, 31055 Toulouse cedex 4, France.</footnote>

<footnote>E-mail address: s.scannella@isae.fr (S. Scannella).</footnote>
been largely demonstrated (for a review see [14]). Consequently, inhibiting a visual distractor when focusing on an auditory target in spatial tasks often implies a higher behavioral cost than the reciprocal – at least when visual and auditory stimuli are equisalient and well localized [1,4]. Over the last decades several researchers have tackled the neural and electrophysiological correlates of the conflict processing. It has been proposed that the N200 and N270 components of the event related potentials (ERPs) correspond to the electrophysiological markers of the conflict monitoring [11,13,30] and conflict processing [33,34,39] respectively. The studies focusing on the N200 revealed that this component is generated by the dorsal anterior cingulate cortex (dACC) (for a review see [13]). In parallel, Zhang et al. [39] showed that a N200 associated with a subsequent N270 could be elicited by the incongruent situation in a visual matching task. Using the same task in a fMRI study [38], they showed that conflicting information increased activation in the ACC (BA 24/32) together with the right dorsolateral prefrontal cortex (DLPFC, BA 9/46). Along with the strong effective connectivity existing between the ACC and the DLPFC [31], these results suggest that the N270 could correspond to a downstream process – based upon afferent ACC signal – that takes place within the DLPFC to select the appropriate motor response.

Unlike the studies of the N200 component, the study of the N270 has essentially been done using sequential *Same Different Judgment* (SDJ) tasks. In these tasks, a first stimulus (S1) is presented and followed by a second stimulus (S2) some hundred milliseconds later that is either the same (match) or different (mismatch) in a given dimension. The matching comparison has been assessed over different modalities and dimensions like the crossmodal gender match [34,25], visual color or shape match [32], number magnitude [16], arithmetic conflicts [33], and spatial matching [18,36]. All these studies have shown that the mismatch condition elicited a large negative amplitude in the fronto-central region of the scalp (although some authors [18,23] have found a more parietal topography) around 270 ms after the onset of S2. This component has been since called supramodal (i.e., independent of the stimulus modality) conflict processing component. As reported by Zhang et al. [37], one problem with the sequential SDJ tasks is that comparing the S2 stimulus to the previous S1 is necessarily a sequential process. Hence, the effects over the N270 observed in the aforementioned spatial mismatch studies could result from a comparison process between a cued attribute of a stimulus and a subsequent target stimulus – referred to as the “template mismatch” by Folstein and Van Petten [13] – than about a stimulus or response conflict processing between two competitive sensory-motor plans.

Based on this work, some authors have tackled the question to
what extent the conflict-related N270 amplitude is modulated by
the experimental context. To our knowledge only selective atten-
tion and cumulative mismatch effects have been evaluated. First,
it has been shown that the N270 amplitude could be modulated
by selective attention [32,18,37]. In these studies, the mismatch
between S1 and S2 was assessed for both a relevant attribute of
the visual stimulus (e.g., the shape) and an irrelevant one (e.g.,
the color) and led to N270 effects in both cases (i.e., larger N270
amplitude within the mismatching condition) albeit weaker within
the irrelevant condition. This result indicates that mismatches in
task-relevant and task-irrelevant dimensions are processed auto-
matically and independently. In another study, Bennett et al. [2]
found that the presentation of a visual distractor enhanced the
N270 amplitude both in a perceptual match and mismatch con-
ditions compared to situations with no distractor. They concluded
that the distractors may differentially affect the N270 through the
generation of task-irrelevant mismatch responses. Finally, Wang
et al. [32] showed no effect of cumulative relevant mismatches over
the N270 amplitude when it concerned two attributes of a same
visual stimulus (i.e., shape + color) compared to the situations with

only one attribute that mismatched. Instead, they found that the
conjunction condition elicited an additional negative peak follow-
ing the N270, leading to the conclusion that the two mismatches
have been processed successively.

These findings provide crucial information about the N270 char-
acteristics and conflict processing, however, there are still several
questions that need answers regarding this component. One of
a great importance is to know whether the N270 is sensitive to
the conflict strength. Indeed, research has often provided evidence
that the N270 is an electrophysiological marker of the supramodal
conflict processing but no indication about its sensitivity to the
strength of the conflict for task-relevant stimuli has been provided
yet. In addition, in all conflict tasks, the active maintenance of the
pertinent attributes is mandatory to achieve the goal. Thus, if the
N270 is generated by the DLPFC – as already suggested [38] – it
is legitimate to consider that manipulating the working memory
in terms of active maintenance levels of the relevant attributes in
a conflict task could interfere with the conflict processing at the
electrophysiological level namely via N270 modulations.

In the present study, we considered Zhang et al. [37] argu-
ment – regarding the sequential presentation used in most of
the SDJ tasks – and proposed to investigate the electrophysiolog-
ical mechanisms underlying the spatial audiovisual conflict with
simultaneous cross-modal target and distractor presentation. More
precisely, we first evaluated the responsiveness of the N270 to
the strength of the conflict by manipulating the target and dis-
tractor modalities. According to the literature [1,4,7,14] the visual
dominance should lead to a larger behavioral conflict cost for the
auditory target with a visual distractor than for the visual target
with an auditory distractor. Thus we hypothesized that a larger
N270 amplitude for the stronger conflict would reflect a conflict
processing cost at the electrophysiological level. Additionally, we
evaluated the impact of increasing the target maintenance effort
over the N270 using a target-modality switching task to disso-
ciate working memory from conflict processing effects. It was
expected that keeping the current target-modality in working
memory within a two target-modality conflict task could interfere
with the conflict processing observed in a single target-modality
task. This would be observed as an increase in the conflict cost at the
behavioral level, associated to larger N270 conflict effects. Finally, a
source localization analysis was used to pinpoint the N270 cortical
generators.

## 2. Method

2.1. *participants*

Sixteen healthy volunteers (8 women, mean age: 50.1, SD: 5.8) received a financial compensation for participating in this study. All were right-handed, as measured by the Edinburgh Handedness Inventory [24] and native French speakers, with normal or corrected-to-normal vision and normal hearing. No participant had a history of neurological disease, psychiatric disturbance or substance abuse, or taking psychoactive medications. This research was approved by the French “Southwest and Overseas Person Protection Committee number 1” and was conducted in accordance with the Declaration of Helsinki.

## 2.2. *Spatial audiovisual conflict tasks*

Stimuli were delivered with Presentation software (Neuro-
behavioral system®). Auditory stimuli were 1000 Hz normalized
pure tones (78 dB SPL) presented via binaural earplugs (Nordic-
NeuroLab) and visual stimuli were filled white circles (2-degree
diameter), presented at a constant angle of 15 degrees from a white
central fixation cross on an ACER 17" monitor placed one meter
<figure><img src="image_121.png" /><figcaption>**Fig. 1.** Illustration of the audiovisual stimuli. The same stimuli have been used to create spatial audiovisual congruent and incongruent trials (a) for the first session ('no-switch') where participants had to respond according to the visual target (b), or the auditory target (c), and for the second session ('switch', d) where they had to alternate between the visual and the auditory targets the trial after the switch signal. Green arrows indicate the target.</figcaption></figure>

from the participant. Auditory and visual stimuli were presented simultaneously during 200 ms and were either on the same side in congruent trials or on opposite sides in incongruent trials, (Fig. 1). Inter-trial interval was set to 2300 ms during which the white cross was always present. Behavioral responses were recorded with a 2-button mouse (left button for left target; right button for right target) across two sessions.

In the first session – referenced as the ‘no-switch’ session – the task consisted in detecting the presentation side of the visual stimuli in one block (visual targets with auditory distractors) and the auditory stimuli in another block (auditory targets with visual distractors). The presentation order of the two blocks was counterbalanced across participants. For each blocks, 60 congruent and 60 incongruent trials were presented resulting in two 5-min duration blocks. In the second session (the ‘switch’ session) an additional distinct auditory signal (pure tone at 750 Hz, 78 dB SPL) was pseudorandomly presented every 4–10 trials (mean 6.5), 32 times instead of 32 regular auditory stimuli (16 times in visual target trials and 16 times in auditory target trials). This sound – referred as the “auditory switch signal” – informed the participants that they had to alternate between visual and auditory targets. The volunteers were asked to keep responding to the current modality target for the trial containing the auditory switch signal and to prepare to switch from the current target modality to the other one for the next trial. A total of 208 trials were presented. 172 trials were distributed into 44 congruent and 44 incongruent trials within each

target modality. 32 trials containing the auditory switch signal were equally distributed into congruent and incongruent trials for each target modality. The switch session duration was 8.5 min.

## 2.3. *Experimental procedure*

Before the experiment, a 64-electrode cap was placed on the participants' head. Participants then seated in a comfortable reclining armchair, placed in a dimly lit, sound-damped room. They were instructed to keep their forearms lying on the chair's arms, with the right forefinger and middle finger resting on the two-button mouse. The overall experiment was completed for each participant in less than one hour, including installation and a training session of 5 min to ensure that they all had reached the maximum level of performance.

## 2.4. *EEG recordings and pre-processing*

EEG data were recorded continuously with a BioSemi EEG system (BioSemi, Amsterdam) from 64 active (preamplified) Ag-AgCl scalp electrodes located according to the International 10/20 system, at a 512-Hz sampling rate and with a 0–104 Hz band-pass filter. Data were processed with a matlab SPM8-based script (The Wellcome Trust Centre for Neuroimaging, UCL). They were referenced offline to the algebraic average of the left and right mastoids, filtered with a band-pass of 0.1–30 Hz and downsampled
at 500 Hz. An independent component analysis using the EEGlab
runica function (www.sccn.ucsd.edu/eeglab) was performed to
isolate eye blinks and movements that have been subsequently
subtracted to the signal. A visual inspection of the data was done to
reject residual artifacts region (1.4% of the trials). Data were later
segmented in 1200 ms epochs starting 200 ms before the onset
of each stimulus (baseline) to compute individual avaraged ERPs.
Finally, the epoched data containing artifacts (>100 µV) were auto-
matically excluded (i.e. on average 4%) before proceeding to grand
averages.

2.5. Data analyses

All behavioral and EEG data were analyzed with Statistica 10 (StatSoft). Due to a lack of variance for the response rate (almost 100% correct in each condition), we did not proceed to accuracy analyses. In addition, the trials containing the auditory switch signal in the modality-switch session were to few for an ERP analysis and involved switching processes that were of no interest regarding our hypotheses, hence were excluded from analyses. Finally, all analyses have been made on averaged left and right target trials as the laterality was not studied.

A three-way repeated-measures analysis of variance (ANOVA) was carried out on mean reaction times with within-subject 'Modality-switch' (no-switch vs. switch), 'Target modality' (auditory target vs. visual target) and 'Spatial congruency' (congruent vs. incongruent) factors. The stimulus-locked ERPs elicited within each condition were computed as the averaged responses to repeated presentations of that condition. ERPs' amplitudes for the N270 analysis were averaged in a time window from 230 to 320 ms after the stimulus onset, determined both from visual inspection and from results of consecutive analyses of 25-ms latency windows. Eight regions of interest (ROIs) were defined to identify the N270 topographic distribution. This was done by first separating the 64 electrodes into two groups: midlines (10) and laterals (54), and then defining subsets of electrodes for analysis. The midlines were divided into two ROIs: fronto-central (FPz, AFz, Fz, FCz and Cz) and centro-occipital (CPz, Pz, POz, Oz and Iz). The lateral electrodes were separated into six ROIs: left (FP1, AF7, AF3, F7, F5, F3, F1, FT7 and FC5) and right (FP2, AF8, AF4, F8, F6, F4, F2, FT8 and FC6) fronto-central; left (FC1, FC3, C1, C3, C5, T7, CP1, CP3 and CP5) and right (FC2, FC4, C2, C4, C6, T8, CP2, CP4 and CP6) centro-parietal, and left (TP7, P1, P3, P5, P7, P9, PO3, PO7 and O1) and right (TP8, P2, P4, P6, P8, P10, PO4, PO8 and O2) parieto-occipital (See Fig. 2 for a graphical representation).

For the midline electrodes, a within-subjects ANOVA with fac-
tors 'Modality-switch' (no-switch vs. switch), 'Target modality'
(auditory target vs. visual target), 'Spatial congruency' (congruent
vs. incongruent), ROI (fronto-central vs. parieto-occipital) and 'Electro-
todes' (5), was computed on the mean amplitudes of the ERPs.
A similar ANOVA was computed for the lateral electrodes, with
factors 'Modality-switch' (no-switch vs. 'switch'), 'Target modality'
(auditory target vs. visual target), 'Spatial congruency' (congruent
vs. incongruent), Hemisphere (left vs. right), ROI (fronto-central
vs. centro-parietal vs. parieto-occipital) and 'Electrodes' (9). All p-
values for ERP results were adjusted with the Greenhouse-Geisser
correction for non-sphericity. Tukey's HSD (Honestly Significant
Difference) were used for all post-hoc comparisons.

2.6. Source localization

On the basis of individual mean ERPs, sLORETA software (stan-
dardized Low Resolution Electromagnetic Tomography [27,26];
free academic software available at http://www.uzh.ch/keyinst/
loreta.htm), was used to compute the three dimensional cere-
bral distribution of the current density at the time point of the

<figure><img src="image_122.png" /><figcaption>**Fig. 2.** Electrode groups for statistical analyses. The 64 electrodes are first divided in two groups: midlines and laterals. The midlines are then subdivided in two ROI groups (red circles; FC: Fronto-Central; PO: Parieto-Occipital). The laterals are divided in two hemisphere groups (left and right) and then in three ROI groups (colored rectangles; FC: Fronto-Central; CP: Centro-Parietal; PO: Parieto-Occipital).</figcaption></figure>

individual N270 peak. sLORETA is based on a three-shell spherical
head model with electrode coordinates derived from cross-
registrations between spherical and realistic head geometry
registered to standardized stereotaxic space (MNI, Montreal Neuro-
logical Institute). sLORETA solves the inverse problem by assuming
synchronous and simultaneous activation of neighboring neurons.
Solution space consists of 6239 cortical voxels (spatial resolution
5 mm). The difference in source localization for the congruency
effect (incongruent vs. congruent) and the modality-switch effect
(no-switch vs. switch) were investigated using nonparametric
statistical analyses of functional sLORETA images (Statistical non-
Parametric Mapping; SnPM) over the N270 latency, based on the
subject-wise normalized power of the estimated electric current
density. The results correspond to maps of log-F-ratio statistic for
each voxel, for corrected *p* < 0.05. This procedure gives the exact sig-
nificance thresholds regardless of non-normality. Finally, sLORETA
log-F-ratio maps were exported in nifti format and plotted as over-
lays in mricron software (https://www.nitrc.org/projects/mricron)
to locate brain areas commonly activated by conflict processing and
working memory during the N270 occurrence.

3. Results

3.1. *Behavior*

Reaction times over all conditions are presented in Fig. 3. The ANOVA revealed a significant main effect of the target modality ($F_{(1,15)} = 22.82, p < 0.001$; $\eta_p^2 = 0.60$) that corresponded to faster RTs for the visual target than for the auditory one. A main effect of the congruency ($F_{(1,15)} = 59.50, p < 0.001$, $\eta_p^2 = 0.80$) was also found and corresponded to longer RTs for the incongruent trials. In addition, we found a significant target modality × congruency interaction ($F_{(1,15)} = 10.76, p < 0.01$, $\eta_p^2 = 0.42$). As revealed by the Tukey's HSD post-hoc tests, a significant target modality effect was present for both congruent and incongruent ($p < 0.001$) conditions, as well as the significant congruency effect was present for both target modalities (auditory target: $p < 0.001$, visual target: $p < 0.01$); the interaction effect was thus due to a larger congruency effect within the auditory target condition compared to the visual one. Finally, a main modality-switch effect ($F_{(1,15)} = 10.75, p < 0.01$, $\eta_p^2 = 0.42$) showed that RTs were longer in the switching modality session than for the non switching ones. No significant interaction of modality-switch with spatial congruency ($F_{(1,15)} = 2.98, p=0.11$)
<figure><img src="image_123.png" /><figcaption>**Fig. 3.** Reaction times for the 'No modality-switch' sessions (left) and the 'modality-switch' session (right). Error bars represent standard errors. \*\*p < 0.01. \*\*\*p < 0.001.</figcaption></figure>

or target modality ($F_{(1,15)}$=2.88, p=0.58) was found meaning that the observed target modality, spatial congruency and modality × congruency effects were not affected by the modality-switch.

## 3.2. N270

Main N270 component statistical results over the midline and lateral electrodes are summarized in Table 1.

### 3.2.1. Topographic effects

The ANOVA over the midline electrodes showed a significant ROI × electrode interaction effect ($F_{(4,60)}$=7.88, p<0.001, $\eta_p^2$ = 0.34). The ANOVA over the lateral electrodes showed a significant hemisphere × ROI × electrodes interaction effect ($F_{(16,240)}$=2.69, p<0.001, $\eta_p^2$ = 0.15). Looking at post-hoc tests for both interactions, we found that the N270 amplitude was maximum over frontal regions, more precisely the left fronto-central area (mean N270 amplitude = -1.64 µV for AF3).

### 3.2.2. Modality and congruency main effects

A main effect of the target modality was found for both midline ($F_{(1,15)}$=6.17, p<0.05, $\eta_p^2$ = 0.29) and lateral electrodes ($F_{(1,15)}$=5.92, p<0.05, $\eta_p^2$ = 0.28) and corresponded to a larger mean N270 amplitude for the auditory target than for the visual target. In addition, the congruency also led to a significant main effect

largely distributed over the scalp (midline: $F_{(1,15)}$=11.36, p<0.01, $\eta_p^2$ = 0.43; lateral: $F_{(1,15)}$=11.80, p<0.01, $\eta_p^2$ = 0.44) with a larger mean N270 amplitude for the incongruent condition.

### 3.2.3. Modality and congruency interaction effects

Interestingly, a largely scalp distributed target modality × congruency effect has been found (midline: $F_{(1,15)}$=7.93, p<0.05, $\eta_p^2$ = 0.35; lateral: $F_{(1,15)}$=7.22, p<0.05, $\eta_p^2$ = 0.32). Post-hoc tests indicated a significant congruency effect for the auditory target (p<0.001 for both midline and lateral electrodes) but not for the visual target (p=0.44 in the best case). Taking into account the ROI and electrode factors we found significant interactions with the target modality and the spatial congruency for midline electrodes only. In fact, the target modality × congruency × ROI × electrodes interaction effect was significant over the midline electrodes ($F_{(4,60)}$=3.50, p<0.05, $\eta_p^2$ = 0.19). More precisely, post-hoc tests revealed that a congruency effect was present for the auditory target condition for all midline electrodes (p<0.001 in all cases) except for FP1 (p=0.07); as well as for the visual target condition but over a reduced central region of the scalp from FCz to POz (p<0.01 for these electrodes; p=0.25 at best for frontal and occipital remaining electrodes); (see Fig. 4). Finally, the same post-hoc test revealed that the congruency effect was always larger for the auditory target than for the visual one due to significant differences between auditory incongruent and visual incongruent conditions (p<0.01 from FCz to POz), and no differences between auditory congruent and visual congruent ones (p values ranging from 0.10 to 0.80).

### 3.2.4. Modality-switch effects

First, a main effect of the modality-switch was found over the midline electrodes ($F_{(1,15)}$=6.95, p<0.05, $\eta_p^2$ = 0.32) but not over the lateral electrodes ($F_{(1,15)}$=3.09, p=0.10). This effect corresponded to a larger N270 amplitude within the switching session compared to the non-switching one. A significant interaction between the modality-switch and the topographic factors was found solely for the midline electrodes (modality-switch × ROI × electrodes: $F_{(4,60)}$=16.634, p<0.001, $\eta_p^2$ = 0.53). Post-hoc tests specified that the modality-switch effect was significant over a large fronto-occipital midline going from Fz to Oz (i.e., excluding FPz, AFz and Iz). Finally, no interaction with the target modality (midlines: p=0.74; laterals: p=0.61), spatial congruency (midlines: p=0.77; laterals p=0.99) or both (midlines: p=0.66; laterals: p=0.78) has been found (Fig. 5).

### 3.2.5. N270 cortical source localization

At the time point of the individual N270 peak, the statistical analyses for the main effect of the congruency showed a greater

<table><caption>**Table 1**

Main ANOVA results on the mean N270 amplitudes.</caption><thead><tr><th>Electrodes</th><th>Effects</th><th>F(df)</th><th>Adjusted p (G-G)</th><th>Epsilon (G-G)</th><th>η<sup>2</sup><sub>p</sub></th></tr></thead><tbody><tr><td rowspan="6">Midline</td><td>Mod</td><td>(1,15)=6.17</td><td>...</td><td>1.00</td><td>0.29</td></tr><tr><td>Congr</td><td>(1,15)=11.36</td><td>...</td><td>1.00</td><td>0.43</td></tr><tr><td>M-S</td><td>(1,15)=6.95</td><td>...</td><td>1.00</td><td>0.32</td></tr><tr><td>Mod × Congr</td><td>(1,15)=7.93</td><td>...</td><td>1.00</td><td>0.35</td></tr><tr><td>ROI × Elec</td><td>(4,60)=7.88</td><td>...</td><td>0.29</td><td>0.34</td></tr><tr><td>Mod × Congr × ROI × Elec</td><td>(4,60)=3.50</td><td>...</td><td>0.30</td><td>0.19</td></tr><tr><td rowspan="6">Lateral</td><td>Mod</td><td>(1,15)=5.92</td><td>...</td><td>1.00</td><td>0.28</td></tr><tr><td>Congr</td><td>(1,15)=11.80</td><td>...</td><td>1.00</td><td>0.44</td></tr><tr><td>M-S</td><td>(1,15)=3.09</td><td>0.10</td><td>1.00</td><td>—</td></tr><tr><td>Mod × Congr</td><td>(1,15)=7.22</td><td>...</td><td>1.00</td><td>0.32</td></tr><tr><td>Hem × ROI × Elec</td><td>(16,240)=2.70</td><td>...</td><td>0.26</td><td>0.15</td></tr></tbody></table>

Mod: Target modality; Congr: Spatial congruency; M-S: Modality-switch; Hem: Hemisphere; ROI: Region of interest (Fronto-central and centro-occipital for midline electrodes, fronto-central, centro-parietal and parieto-occipital for lateral electrodes); Elec: Electrodes.

<footnote>* p < 0.05.</footnote>

<footnote>** p < 0.01.</footnote>

<footnote>*** p < 0.001.</footnote>