Analysis of selective attention processing on experienced simultaneousinterpreters using EEG phase synchronization
Haruko Yagura, Hiroki Tanaka, Taiki Kinoshita, Hiroki Watanabe, Shunnosuke Motomura,
Katsuhito Sudoh, and Satoshi Nakamura
Abstract— This study analyzed the selective attention processing related to cognitive load on simultaneous interpretation (SI).
We tested simultaneous interpreter’s brain function using EEG
signals and calculated inter-trial coherence (ITC) extracted by
the 40-Hz auditory steady-state response (ASSR). In this experiment, we set two conditions as Japanese-English translation andJapanese shadowing cognition. We also compared two subject
groups: S rank with more than 15 years of SI experience (n=7)
and C rank with less than one year experience (n=15). As aresult, the ITCs for S rank in interpreting conditions weremore significantly increased than C rank in the shadowing
conditions (ITC: p<0.001). Our results demonstrate that 40Hz ASSR might be a good indicator of selective attention andcognitive load during SI in ecologically valid environmental
conditions. It can also be used to detect attention and cognitive
control dysfunction in ADHD or schizophrenia.

I. INTRODUCTION
Simultaneous interpretation (SI) is an advanced cognitive
task that performs multitasking and multi-modal processes.
SI translates from their native language into a non-native
language, therefore SI is a great difficult task that requires
timely processing of listening, translating, and speaking. So,
SI is called as an extreme case of multitasking [1]. Recently
interests has been increased in the brain mechanisms thatshow how simultaneous interpreters overcome cognitive load[2], [3], [4].
Cohen’s working memory model, which focuses on
switching attention, is often used as a cognitive model toexplain cognitive load during SI [2]. According to the model,
each multitask is processed with a slight delay, and theinterpreter must process each task one after another. Since
the amount of information that can be focused on during SI islimited, exceeding these limits creates cognitive burden. In
addition, small experiences with SI and different language
structures make larger increase of the cognitive load [2], [3],[4].
In recent years, there has been a large number of publications directed to studying brain functions in SIs who havetrained for managing multilingual language control. Manysuch studies using functional magnetic resonance imaging(fMRI) and electroencephalogram (EEG) have investigated
the increased experience of interpretation and improved functional connectivity of the left dorsolateral prefrontal cortex*This work was supported by JSPS KAKENHI Grant No.JP17H06101
and JP18K11437.
H.Yagura, H.Tanaka, T.Kinoshita, H.Watanabe, S.Motomura, K. Sudoh, and S.Nakamura are with the information Science, Nara Institute of Science and Technology, 8916-5, Ikoma-shi, Nara, Japan

haruko.yagura.ye4@is.naist.jp

streaming network [5], [6], [7], [8]. However, most studies
control their experimental conditions by translating short
sentences and short words without reflecting on the ongoingload faced by translators during actual simultaneous interpretations. It is also unclear how experienced simultaneousinterpreter process the cognitive load and attention switchingin the brain as described in Cohen’s model.
A noteworthy study in translation [9] investigated thecognitive load associated with attention processing using
EEG. This work focused on eliminating the process by which
the human brain reduces attention to unrelated stimuli withincreasing cognitive load and raised selective attention to
the related stimuli. For unrelated stimuli, a task-independent
probe stimulus (440 Hz, 52 ms pure sine tone) was used.
By simultaneously presenting the translated speech and tone
signals to the subject’s ear, we analyzed the change in such
event-related potential (ERP) amplitudes as N1 and P1 peaks
that are induced by the tone signal to determine whether
the filtering function was indicated. The N1 amplitude decreased, P1 increased with additional cognitive load, and theword familiarity remained low. However, they controlled theexperimental conditions by editing the translated speech intoa very short segmentation, which caused a departure fromthe actual SI translation situation.
The difficulty in measuring EEG during SI is the effect ofenvironmental noise on EEG signals, such as mouth and hand
movements, blinking, and interpreter’s voice. To eliminate
these effects, the experimental environment must be controlled, but the neural activity in a natural environment close
to SI cannot be investigated. 40-Hz ASSR is a pulse responsethat has the same effect as quickly averaging many signals.
This means that the ASSR signal must be extracted thatappears at the same time as the translation without editing theoriginal audio. Therefore, 40-Hz ASSR generally provides
a robust response to environmental changes and creates an
ecological experimental environment. Furthermore, recent
studies have shown that when tasks of varying difficulty
are aurally presented as 40-Hz ASSR, people can perform
complex tasks in phase-locked responses in the 40-Hz ASSR
[10].40-Hz ASSR is said to be associated with selective
attention [10], [11] and also affects such attention and cognitive control dysfunctions as attention Deficit Hyperactivity
Disorder (ADHD) and schizophrenia [5], [12], [13].
We conducted experiments to investigate the following two
objectives:
1) To examine the relationship between cognitive load andthe attention of simultaneous interpreters using 40-Hz

Japanese sounds

Pulse tone for ASSR
Fig. 1.

EEG measurement during SI

ASSR.
2) To verify trained SI’s brain functions in multilingual
language control functions using 40-Hz ASSR.

Fig. 2. Synthetic speech sounds of 40-Hz ASSR and Japanese radio news

II. M ETHODS
A. Subjects
Professional, Japanese female interpreters (n = 22; ages
ranged from 46 to 71, mean = 53.4, sd = 6.6) participated inour study. We ranked them as either experts (7 subjects withover 15 years of SI experience) and beginners (15 subjects
with at least one year of SI experience). We called the expert
group S and the beginner’s group C. None had hearingrelated problems or any history of neurological problems;
all were right-handed. The Research Ethics Committee of
our institution reviewed and approved this experiment. Written informed consent was obtained from them before theexperiment.

(Version 18.0, Neurobehavioral Systems, Inc., Berkeley, CA,
www.neurobs.com).

B. Sound stimuli

D. Subjective evaluation

We prepared eight topics from Japanese NHK radio news
because it is Japan’s most representative news channel. Mostof the topics were political news and unlikely to elicit
emotional responses [14]. We generated a 40-Hz pulse tonethat elicits a 40-Hz ASSR response based on previous studies[15] (Fig. 2).
The synthetic speech sounds for interpretation were eight
different 60-second length of Japanese radio news. We
presented one-minute, synthetic speech, 40-Hz ASSR andJapanese radio news sound clips to their ears. The soundpressure was normalized with maximum amplitude. The
pulse tone’s sound pressure level was edited by a speechtherapist to evaluate whether it could be heard adequately
and whether it felt uncomfortable as 5% of the maximum
amplitude of the news sounds. The 40-Hz pulse tones andnews sounds were synthesized in stereo. We created audio
files with open audio editing software called Audacity.
C. Task sequences
Our task was comprised of the following two conditions:translating from Japanese to English (TR condition) andshadowing (SW condition) as the participants heard thelanguage (Fig. 3). We randomly presented eight bits ofnews under the two conditions. We presented the stimuli bypresentation software provided by Neurobehavioral Systems

Fig. 3.

Task sequences between tasks. TR: translation, SW: shadowing

After the EEG experiment, subjects were instructed toanswer the subjective evaluation sheet for the eight topics
and five questions. The following are the question contents:
Q1: achievement for interpretation, Q2: topic field, Q3: voice
speed, Q4: a comfortable voice for listening, and Q5: Overall
interpretation difficulty. The questions were evaluated on fivescales:, where scale 1 was the most easy and scale 5 was
the most difficult. After answering all five questions, ourparticipants were encouraged to add their own comments.
E. EEG data acquisition and prepossessing
The EEG signals were recorded with a Cognionics Quick30 Dry EEG Headset with 29 electrodes (excluding one forthe reference channel) [1]. The recorded signals were bandpass filtered from 1 to 50 Hz at a sampling rate of 500.
The EEG signals were referenced by subtracting the average
signal of A1 and A2. The ground electrode was placed in
the center between Fp1 and Fp2. All the EEG signals were
processed using MATLAB (Math Works, Natick, MA, USA).
1) Inter-trial coherence (ITC): To quantify the cognitive
load using EEG signals, we extracted the phases at 40-Hz andcalculated ITC based on previous research [16]. We divided
the continuous EEG data into trials of three seconds andshifted them by one second [17]. We performed a Fouriertransform at 40 Hz for each channel and calculated the ITC
based on the following equation:

Fig. 4.
Interaction plot between RANK and TOPIC for subjective
evaluation value on Question 1

Fig. 6.

Interaction plot between TOPIC and RANK on SW condition

III. R ESULTS
A. Subjective evaluation
We performed two-factor factorial ANOVAs for all fivequestions. In Q1 and Q2, the difficulty for the C rank
significantly exceeded the S rank. The only main effects thatappeared with the rank were Q3 (F1,7 =13.7, p<0.001) andQ4 (F1,27 =22.5, p<0.001). In addition, the only topic thatshowed the main effect was on Q5 (F7,23 =4.3, p<0.001).
We found a main effect in both the rank and topic for Q1(rank; F1.9 =12.7, p<0.001, topic: F7.22 =4.3, p<0.001) andQ2 (rank:F1.4 =6.4, p<0.001, topic: F7.9 =2.1, p<0.05). On
the other hand, we identified no interactions between therank and topic for all the questions (Fig. 4).

Fig. 5.

Interaction plot between TOPIC and RANK on TR conditionPK
IT C[ch] = |f
k=1 exp(jθk [ch])K|,

where f is a frequency, ch is the channel number, θkf
is the phases of frequency f and electrode ch, k is a trial
number, and K is the number of trials. Since the cognitive
control function of SI is involved in the frontal lobe, thisstudy focused on the frontal lobe channels. The ITCs were
calculated for each news bit over the three frontal electrodes(’F3’, ’Fz’, and ’F4’) and averaged over the electrodes [16].
We selected the frontal lobe electrode because many previous
studies concluded that the multilingual processing functionsof trained SIs are localized in the frontal lobe.
2) Statistical analysis: To verify the difficulty of the ranks
during SI for subjective evaluation, we performed two-factor
factorial ANOVAs with topics (8 NHK news) and experiment
ranks (C and S). Since the contents of the five questions were
not related, they were considered independent variables, and
ANOVAs were performed for each question. For the ITC
values, we performed two-factor factorial ANOVAs with rank
(C and S) and task (TR and SW) to analyze the rank andtask effects.

B. ITCWhen the two-factor ANOVAs were applied to the ITC
values on the frontal cite electrodes, we identified a statistically significant influence of the rank for the TR condition(F1,0.5 = 12.1, p<0.001)(Fig. 5), although the rank for
the SW condition showed no significance ( F1,0.00 = 0.02,
p=0.8)(Fig. 6). The main effects of the topic on TR and SW
did not show any significance (TR: F7,0.4 = 1.5, p = 0.16;
SW: F7,0.08 =0.4, p=0.8), and no interactions were detected
between the rank and topic factors for each condition (TR:F7,0.06 = 0.2, p = 0.9; SW: F7,0.08 =0.4, p=0.8). In addition,
for the task (TR and SW) and rank (S and C) factors, twofactor factorial ANOVAs showed a statistical difference inrank (F1,0.2 = 7.5, p<0.001) and task (F1,0.6 =18.1, p<0.001)
as well as a significant interaction between rank and task
(F1,0.3 = 8.6, p<0.01) (Fig. 7).
IV. D ISCUSSION
A. The subjective evaluation value for S and C rank interpreters
Regarding the difficulty of translating for the subjective
evaluations, in 4 of the 5 subjective evaluation questions,the main effect of evaluation value between S and C wassignificant, and the value of C was significantly higher thanthe S of it. In almost all the questions, interpreting for the
C seemed more difficult than the S (Fig. 4).

R EFERENCES

Fig. 7.

Interaction plot between TASK and RANK

B. Changes in selective attention processing of translationand shadowing conditions
Considering the phase-lock response on an ITC value toattention processing, the ITC of the S rank was significantly
higher than the C rank under the TR condition (Fig. 5). But
the SW between the C and S ranks showed no difference
among the conditions (Fig. 6). Regarding the task, although
the difference in the ITC values between the ranks (especiallyin the S rank) was larger in the TR condition than in the
SW condition, we observed no difference under the SW
condition (Fig. 7). These results clearly show that the Srank phase synchronization increased during the conversions.
Earlier studies showed that 40-ASSR is associated withselective attention [16], [10], [18], [1]. The TR and SW
tasks in our study reflect differences in selective attention
focused on different tasks, and when focusing on the same
voice as NHK news, TR requires increased attention to
the translation. These responses suggest that 40-ASSR is
associated with selective attention, and for TR and SW, thesubjects reflect differences in the attention targets because
they were listening to the same news topic and differences
in selective attention. This may indicate an increase in the
S-rank selective attention while interpreting. The trained
SI’s brain function might have increased the multilingual
language control in the prefrontal cortex. This result supports
many previous studies [5], [6], [7], [8].

V. C ONCLUSIONS
We extracted phase-synchronous responses during simultaneous interpretation using 40-Hz ASSR, detected the multilingual processing control network of the frontal lobe
in trained interpreters, and supported previous studies. We
showed that ITC can be a measure of cognitive load andselective attention in SI. In the future, we will verify cognitive load and attention using such a behavior as speechinterpretation and conduct research that will lead to thediagnosis of the bio-markers of such attention diseases as
ADHD and schizophrenia.

[1] E. Camayd-Freixas, “Cognitive theory of simultaneous interpretingand training,” in Proceedings of the 52nd Conference of the American
Translators Association, (New York, ATA, USA), pp. 1–29, 2011.
[2] N. Cowan, “Processing limits of selective attention and working
memory: Potential implications for interpreting,” Interpreting, vol. 5,
no. 2, pp. 117–146, 2000.
[3] D. Gile, “Testing the effort models’ tightrope hypothesis in simultaneous interpreting-a contribution,” Hermes, vol. 23, no. 1999, pp. 153–
172, 1999.
[4] A. Mizuno, “Process model for simultaneous interpreting and working
memory,” Meta: Journal des traducteurs/Meta: Translators’ Journal,
vol. 50, no. 2, pp. 739–752, 2005.
[5] J. A. Grahn, J. A. Parkinson, and A. M. Owen, “The cognitive
functions of the caudate nucleus,” Progress in Neurobiology, vol. 86,
no. 3, pp. 141 – 155, 2008.
[6] S. Elmer and J. Kühnis, “Functional connectivity in the left dorsal
stream facilitates simultaneous language translation: An EEG study,”
Frontiers in Human Neuroscience, vol. 10, no. FEB2016, pp. 1–9,
2016.
[7] A. Hervais-Adelman and L. Babcock, “The neurobiology of simultaneous interpreting: Where extreme language control and cognitive
control intersect,” Bilingualism, 2019.
[8] E. Van de Putte, W. De Baene, L. Garcı́a-Pentón, E. Woumans,
A. Dijkgraaf, and W. Duyck, “Anatomical and functional changes
in the brain after simultaneous interpreting training: A longitudinal
study,” Cortex, vol. 99, pp. 243–257, 2018.
[9] R. Koshkin, Y. Shtyrov, A. Myachykov, and A. Ossadtchi, “Testing
the efforts model of simultaneous interpreting: An erp study,” PloS
one, vol. 13, no. 10, p. e0206129, 2018.
[10] V. Parciauskaite, A. Voicikas, V. Jurkuvenas, P. Tarailis, M. Kraulaidis,
E. Pipinis, and I. Griskova-Bulanova, “40-Hz auditory steady-state
responses and the complex information processing: An exploratory
study in healthy young males,” PLoS ONE, vol. 14, no. 10, pp. 1–15,
2019.
[11] N. Müller, W. Schlee, T. Hartmann, I. Lorenz, and N. Weisz, “Topdown modulation of the auditory steady-state response in a task-switch
paradigm,” Frontiers in Human Neuroscience, vol. 3, no. FEB, pp. 1–
9, 2009.
[12] I. Griskova-Bulanova, K. Dapsys, S. Melynyte, A. Voicikas, V. Maciulis, S. Andruskevicius, and M. Korostenskaja, “40hz auditory steadystate response in schizophrenia: Sensitivity to stimulation type (clicks
versus flutter amplitude-modulated tones),” Neuroscience Letters,
vol. 662, pp. 152 – 157, 2018.
[13] A. Khaleghi, H. Zarafshan, and M. R. Mohammadi, “Visual and auditory steady-state responses in attention-deficit/hyperactivity disorder,”European archives of psychiatry and clinical neuroscience, vol. 269,
no. 6, pp. 645–655, 2019.
[14] H. Shimizu, G. Neubig, S. Sakti, T. Toda, and S. Nakamura, “Collection of a simultaneous translation corpus for comparative analysis,” inProceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), (Reykjavik, Iceland), pp. 670–
673, European Languages Resources Association (ELRA), May 2014.
[15] Y. Yokota and Y. Naruse, “Phase coherence of auditory steady-state
response reflects the amount of cognitive workload in a modified nback task,” Neuroscience research, vol. 100, pp. 39–45, 2015.
[16] I. Griskova-Bulanova, O. Ruksenas, K. Dapsys, V. Maciulis, and S. M.
Arnfred, “Distraction task rather than focal attention modulates gamma
activity associated with auditory steady-state responses (assrs),” Clinical Neurophysiology, vol. 122, no. 8, pp. 1541–1548, 2011.
[17] Y. Yokota, S. Tanaka, A. Miyamoto, and Y. Naruse, “Estimation ofhuman workload from the auditory steady-state response recorded via
a wearable electroencephalography system during walking,” Frontiersin human neuroscience, vol. 11, p. 314, 2017.
[18] M. Sabri, C. Humphries, M. Verber, E. Liebenthal, J. R. Binder,
J. Mangalathu, and A. Desai, “Neural effects of cognitive control load
on auditory selective attention,” Neuropsychologia, vol. 61, pp. 269–
279, 2014.