Analysis of selective attention processing on
experienced simultaneous interpreters using
EEG phase synchronization
Graduate School of Information Science, Nara Institute ofScience and Technology
Haruko Yagura, Hiroki Tanaka, Taiki Kinoshita, Hiroki Watanabe, Shunnosuke Motomura,
Katsuhito Sudoh, and Satoshi Nakamura

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

0

Simultaneous Interpreta0on (SI)

Simultaneous interpretation

Consecutive interpretation

Ø Interpreter according to thespeaking speed of the speakerØ Speaker and interpreter alternate

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST1

ContentsØIntroduction:-What brain function of experienced SI are excellent-The problem of measuring Brain response for SI in realenvironment-Proposed method:40-Hz ASSR
ØThe methods for EEG measurement during SI-Data acquisition and analysis ITC
ØResults and Discussions
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

2

Background•Switching A+en-on Processing for SI•

40 Hz-ASSR•

Proposed Method
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

3

Interpreter advantage
Training of SI changes the structure of brain funcEon and improves theexecuEve funcEon related to aLenEon and memory [Van de PuLe, E., +2018][Camayd-Freixas, E., + 2011]
Brain mechanism of experienced SI has been noted as expanding the possibilityof training human a>en?on and memory

The purpose of this study: Quantifying the brain functionof experienced SI using EEG
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST4

Switch attention continuously during SI

ASen9on•• Understanding
the meaning of
the nativelanguage• Interpretationin otherlanguages inmind

Attention

Anotherlanguage• utterance• Understanding
the meaning of
the na9ve
language• Interpreta9on
in otherlanguages inmind

Nativelanguage

Na9velanguage

Attention

Attention

SI requires continuous attention switching from the native language to anotherlanguage which involved selective attention processing[Cowan,N,2000 +]- Multilingual Control, [Morels, J.,+,2015] - Executive function
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

5The Diﬃculty of EEG measurement in thereal environment for SI•

Original sounds which is not edited50 20•!

 76

Shortened and edited sound stimulus for comparison between conditions50

20



!

Cond1:

Cond2

Cond3

Cond1

Cond1


Cond2





Cond2

Cond1

Cond1

Ø Repeat short stimuli for Controlling
experimental conditions- Additive Average Processing (EEG)- Block design (fMRI)

Ø The selective attention processing which occurs continuously during SI
cannot be verified.
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST64))( 4Present with target
sDmulus

Associated withselecDve aJenDon[Mu ̈ller, N+,2009][Griskova.B, I.+, 2011]

Very short pulse soundRobust against
environmentalchanges04))(40 Hz-ASSR

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

7

ITC ; Inter-trial Coherence [Yokota+, 2015, 2017]• Extrac'on of phase-locked
response and calculate ITC
value induced from 40HzASSR• ASSR presented at the same'me the target s'muli suchas n-back task during
walking with a lot of noisesuch as body movement• Succeeded in extrac'ng
suitable response during
walking using ITC

2020/7/ EMBC2020

N-back2020@Haruko Yagura AHC-Lab, IS, NAIST

8The value of ITC

The ITC value (index indicating phasesynchronization in the brain)

Phasesynchronization

PhasesynchronizationHighmax 1

Lowmin 0
The value of ITC

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

9

Proposed method: 40Hz-ASSR and ITC values are used forextrac@ng selec@ve aBen@on processing during SI

Experimental conditions close to
the real environment
The use ofASSR

The previousmethod■Less likely to be affected by vibrations at frequencies lower
than 40 Hz (body movement, blinking, etc.)■Since the ASSR signal is presented at the same time as theSI voice, there is no need to edit the voice.

Different from the realenvironment
 Control experimental conditions
 Unable to detect continuous attention switching

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

10Proposed method

Objective of the study:• Verification of selective attention for experienced SI
in real environment using EEG signals
Proposed method:• Extract ITC value induced from 40Hz-ASSR during SI(target task)
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

11

Methods:•Subjects•

Sound Stimuli and Experimental Tasks•ITC value and hypothesis
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

12Subjects:
n 22 Japanese female interpretersrange; 46-71 yearsmean= 53.4 yearsSD = 6.6
C rank (less than 1 year of simultaneousinterpretation experience): n = 15
S rank (15+ years of simultaneous interpretation
experience): n = 7
Ø Prior to the experiment, we obtained
informed consent in writing approved by
the ethics committee of NAIST

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

13

Sound Stimuli: Present 40Hz ASSR and Japanesevoice at the same timen Presents 8 kinds oftopic Japanese radio
news on both ears•

The duration of each of the eight
Japanese sounds were 60 seconds•

Sound pressure is normalized bymaximum amplitude•Sound pressure level for pulse
tones adjusted to 5% of maximum
news sound amplitude•Japanese voice is presented
without editing source sounds
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

14

EEG settingsn Using wireless EEG
••

3 C03

-E

D : D 30 0G0; :0 G
10 3 0 5 :3GF
3 :3

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

15

Stimuli Sequencesn SI condition
SimultaneousInterpretation( Japanese to English)

Instruction

SISISHSH

Presented Soundstimulin SH conditionShadowing( Japanese to Japanese)Ø Under conditions, 8 kindsof topics are presented
randomly for 60 seconds
after instructions for 2seconds.

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

16

Analysis: Comparing ITC values between
condi:ons
nCalculate ITC value foreach condition• For each subject, divide the EEG dataobtained for each experimental
condition and topic into 3-second
trials while overlapping each second.
• After performing a discrete Fouriertransform on each of the divided
trials, calculate the ITC value with theright formula• ITC calculates the average of threefrontal lobe electrodes (F3, Fz,F4)[Griskova+2011]2020/7/ EMBC2020

Ø where f is a frequency, ch is thechannel number, f
Ø k is phases of frequency f andelectrode ch, k is a trial number, and K
is the number of trials. ITCs werecalculated forØ each news over the three central
electrodes (C3, Cz, and C4). ITCs
were then averaged over theelectrodes

2020@Haruko Yagura AHC-Lab, IS, NAIST

17ITC value and selective attention• 40Hz-ASSR amplitude and phase-locked response increase as
selective attention ability increases

The value of ITC[Mu l̈ ler, N+,2009] [Griskova.B, I.+, 2011] [Parciauskaite V.,2019+]• Selectiveattention:low(near 1)(near 0)2020@Haruko Yagura AHC-Lab, IS, NAIST

The value of value

2020/7/ EMBC2020• Selectiveattention:
High18

Hypothesis; SI-S rank> C rank

ITC

SH

SI
Japanese toEnglish

Japanese toJapanese
RecogniHon
language

2020/7/ EMBC2020

SwitchingaGenHon

Selective attention
S -rankC-rankRecognitionlanguage

2020@Haruko Yagura AHC-Lab, IS, NAIST

19

Subjective Evaluation• Performed after the EEG experiment• 5 questions are set for each of 8 topics• level 5- most difficult, level 3 -normal, and level 1most easy
Q1: achievement for interpretation
Q2: topic fieldQ3: voice speed
Q4: a comfortable voice for listening
Q5: Overall interpretation difficulty

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

20

Results and discussion•

Subjec've Evalua'on•

ITC value

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

21Subjective Evaluation Value

Most Difficult•••

Most Easy

2 way - factorial ANOVA - 8topic ✖ 2 ranks (C and S)
Main eﬀect(rank; F1,160=12.7p< 0.001; topic:F7,160 =4.3 p< 0.001)
Subsequent test; C > S (p. <0.001)

Ø C rank is rated higher than S rankØ Reflects difficulty of interpreting
due to lack of years ofexperience

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

22

ITC value••

2 way - factorial ANOVAs : 2task (SI and SH)✖ 2 ranks (Cand S)
Main eﬀect; rank (F1,348 = 7.5,p<0.001) and task (F1,348 =18.1,p<0.001)

••
SHSI

InteracQon (F1,348 = 8.6p<0.01)
Subsequent test;
SI:S > C( p<0.001)
SH: n.sØ Verified the hypothesis

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

23

Conclusions• We used 40 Hz-ASSR-induced ITC values to verify selective attention processing
with experienced SI.

• ITC value under SI condition was significantly higher in S rank than in C rank but
not observed in the SH condition. It verified the our hypothesis• We proposed that 40Hz-ASSR phase-locked response is eﬀecLve as a method toquanLfy the selecLve aRenLon processing of experienced SI in the realenvironment.
2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

24

Future worksØThe following points could not be verified in this experiment,
so it is a future works.
• Temporal change of selective attention• Relationship between difficulty level of each topic and years of SI
experience.

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

25

Thanks for your attention

2020/7/ EMBC2020

2020@Haruko Yagura AHC-Lab, IS, NAIST

26