Sentence classification based on phase patterns in EEG
neural oscillation during imagined speech
調音動作想像時の脳波位相パターンを用いた文識別
Hiroki WATANABE, Hiroki TANAKA, Sakriani SAKTI, Satoshi NAKAMURA
Graduate School of Information Science, Nara Institute of Science and Technology, Nara, Japan
Email: watanabe.hiroki.vx6@is.naist.jp

Introduction
A big picture of our research
Featureextraction

Measuring
EEG

Model
training

Previous research & research focus
classification of heard speechclassification of imagined speech• vowel classification (DaSalla et al., 2009)• syllable classification (D’zmura et al., 2009)• word classification (Martin et al., 2015)• sentence classification ➡ no research• EEG-based sentence classification(Watanabe et al, 2017)

Estimation ofspeech

How are you?

→speech imaginationin the head

• Utilizing a phase synchronization
to sentence rhythms

Applying an EEG-based sentenceclassification method to imagined
speech recognition

How are you?

Research purposes

Investigating…
(1) a phase synchronization during imaging articulatory movements of sentences(2) accuracies of EEG-based imagined sentence classification using the phase synchronization as features

Research questions

Analysis pipeline

Speech comprehensionー speech rhythm (4-8Hz)• perception data: 35 trials (subj 01), 38 trials (subj 02)• imagined data: 39 trials (subj 01), 42 trials (subj 02)

preprocess

ー neural oscillation in STG (4-8Hz)

Synchronization analysis
phase synchronization

speech rhythms ➡ replicable oscillation patterns

phase-locking value(PLV)

perception: model
imagined: produced
speech data

PLV(ch) =1
෍ exp(1𝑖 ∗ 𝐸𝐸𝐺𝜑 − 𝑆𝑃𝐸𝐸𝐶𝐻𝜑 )𝑇Classification analysis: 3 sentence classification task

phase synchronization b/w
produced speech & neuraloscillation in the motor area?
If there is a synchronization,phase information is useful features
for classification

featureextraction

perception: fronto-central• Averaged across target channels
imagined: centro-parietal• FFT decomposition• Phase extraction: [0, 2.8] sec, [2, 12] Hz

modeltraining• Logistic regression • Random forest• SVM• Template matchingevaluation• Leave-one-out cross-validation

Research questions

Participant 1 male, 1 female (L1: Japanese)
Speech material

ResultSynchronization analysis
Perception: stronger PLV infronto/ central region
centro/ parietal region

logistic regression: logregrandom forest:rf
template matching: template

LISTENSPEAK++++++IMAGINE+++

2 sec

2.5 sec

5.5 sec

2.5 sec■ imagined■ perception■ imagined■ perception

Accuracy (%)(What the girl said to me just now is true.)

Ready?

subj 02
percep: 57.9 % (logreg/template)imagine: 52.4 % (SVM)

subj 01
percep: 40.0 % (rf)imagine: 59.0 % (log reg/ SVM)

2. ついさっき女の子が私に言ったことは本当の話。

Indication of a task execution

Averaged PLV across two participants

Classification analysis(The book that you were absorbed in yesterday is interesting.)

Tasks

imagined→Imagined: stronger PLV in

1. あなたが昨日夢中で読んでいた本はおもしろかった。

3. 向こうの壁に飾っているのは彼のお兄さんが描いた絵。
(The picture on the other wall was drawn by his older brother.)perception

Accuracy (%)

EEG experiment

EEG phasespeech phase𝑡=1

Hypothesis: imagined speech

RQ. 1 Do EEG phase patterns during imagined speech synchronize withproduced speech rhythms?
RQ.2 How accurately do EEG phase patterns classify imagined sentences?

EEG

bandpass-filter (1-14Hz)

𝑇

spoken sentence classification (Watanabe et al., 2017)

perception: listening task
imagined: imagined task

listeningmodel speech is played
speaking
imitating model speech orallyimaging
imaging articulatory movement
as the same rate to the speaking
task

Selected references
H. Luo and D. Poeppel. (2007) Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex, Neuron, vol.5, pp.1001–1010.
M. D ’Zmura, S. Deng, T.L.S. Thorpe, and R. Srinivasan. (2009) Toward EEG sensing of imagined speech, In Human-Computer Interaction. New Trends, Springer, Berlin,Heidelberg, pp.40–48.
C.S. DaSalla, H. Kambara, M. Sato, and Y. Koike. (2009) Single-trial classification of vowel speech imagery using common spatial patterns, Neural Networks, vol.22, no.9,pp.1334–1339.
H. Watanabe, H. Tanaka, S. Sakti, and S. Nakamura. (2017) Subject-independent classification of Japanese spoken sentences by multiple frequency bands phase pattern ofEEG response during speech perception, In Proceedings of Interspeech, Stockholm, Sweden, pp.2431–2435.

logregrf

SVM

templatelogregrf

SVM

template

Conclusion & Future directions
RQ. 1 A tendency of phase synchronization b/w produced speech & EEG
oscillations during imagined sentences
RQ. 2 Best accuracies: [52.4 59.0 %] by SVM.
Phase patterns are reliable features for imagined sentence recognition
Future directions: (1) Increasing number of participants +(2) Source localization of this phase synchronization phenomenon