Detecting Syntactic Violations from Single-trial EEG using
Recurrent Neural Networks
Shunnosuke Motomura, Hiroki Tanaka, Satoshi Nakamura
Nara Institute of Science and Technology, Nara, Japan{motomura.shunnosuke.mj1, hiroki-tan, s-nakamura}@is.naist.jp

ICMI 2019 ACM International Conference on Multimodal Interaction - Late Breaking Results

Introduction

Features & Models

 Research goal Feature extraction [Vareka L, et.al, 2017]• Automatic evaluations of sentences for machine translation / dialog. system
Subjective evaluations are biased & ambiguous by human evaluators•

Average amplitudes b/w 100 ms and 800 ms per each 50 ms time window

Ch1

 Research purpose

t=0ms*The child throw the toy

Incorrect

t=100ms

t=800ms• Long short-term memory (LSTM)
Feature vectors were inputted in sequential manners• Stacked autoencoders (SAE)
Feature vectors of all channels were concatenatedInput
vectors

Trans. x1,14Architecture of LSTM
Hiddenlayer1 Hidden
layer2
Outputsora-gasky-NOM

ton-da (* means syntactic incorrectness)
fly-PAST

Architecture of SAE

 Data• Training :14 participants’ data (1040 sentences)• Test:4 participants’ data (314 sentences)• Correct sentence: 50% / incorrect: 50% -> chance level: 0.5
 Optimization of hyper-parameters• Grid-searching with 10-hold cross validation in the training data
• SVM
- C = {0.001, 0.01, 0.1, 1, 10, 100}

• SAE
- Number of hidden units: {10, 50, 100, 200, 300}
- Number of hidden layers: {1, 2, 3}- Activation functions: {sigmoid, rectified linear unit}• LSTM• The nominative case of second phrase as synchronous onset (t=0ms)• 40 sentences for syntactic correct and incorrect condition respectively• Speech by a professional female narrator was used for stimulus- Number of hidden units: {5, 10, 15, 20, 25, 30}, others are the same as SAE

 Multiple-trials averaged analysis• We also investigated classification performances on averaging multipletrials EEG signals

EEG Data Acquisition
 Experimental procedure

Results & Conclusions
 Sigle-trial classification results

Carried out in a soundproof room

ModelSVM
SAE
LSTM

normal or anomalouscorrectorincorrect(1) watch the ‘+’ mark1s(2) listen to the sentence4s

 EEG recording and preprocessings
EEG cap: ActiCap by Brain Products (32 channel electrodes)
Preprocessings
1. Re-referencing
2. High-pass filtering
3. Epoching at synchronous onsets
4. Reject artifacted epochs and removing muscle/eye-blink artifacts-> 1 participant was rejected (more than 30% epochs were rejected)••••

Accuracy
0.584
0.583
0.613

(3) press the button2s Participants: 19 Japanese speakers (16 males & 3 females, mean age: 24.2)
••

...

...

Trans. x31,14...

ton-dafly-PASTNOM : nominative case marker
ACC : accusative case marker
PAST : past tense morpheme•... x31,14x31,1...

x31,1 ... x31,14

Concat...

...

sora-osky-ACC(The bird flew in the sky.)*tori-gabird-NOM

x31,1x1,14

 Baseline model: linear-kernel support vector machine (SVM)• Japanese sentences manually crafted refering to [Takazawa S, et.al, 2002]• Repetition of nominative case violates Japanese grammar

b.

...

Classifications

 Syntactic violations

tori-gabird-NOM...

...

x1,1 ... x1,14

Inputlayer

x1,1Materialsa.

x1,1x31,14into one column vector• Some Neural network (NN) models well performed- Stacked autoencoders (SAE) [Vareka L, et.al, 2017]- Long short-term memory (LSTM) [Alhagry S, et.al, 2017]• In this work, neural network models (SAE and LSTM) were applied to classify
single-trial EEG signals for syntactic violations

x1,14
...

x31,1

 Models: Single-trial EEG classification• We have to evaluate each sentence -> single-trial classification• [Tanaka H, et.al, 2019] ahieved 57.7% acc. for detecting syntactic violations-> More accurate methods are necessary

Output...

Classification
model

x31,1...

...

Feature extractions

x31,1 ... x31,14

Hiddenlayersx1,1SpeechEEG

x31,14...

Ch3150ms...

 Overview

x1,14x1,1...

• Detecting syntactic violations in spoken sentences with single-trial EEG
Language-related EEG is usually studied by averaging multiple-trials due toit’s low signal-to-noise ratio

x1,1 ... x1,14

 Multiple-trials averaged accuracies•

Gradually increasing while
the number of averaging trials increase Conclusions•LSTM could achived over 60% accuracy higher than chance level (p<0.01)-> Sequential models are feasible to properly classify high-dimensionalsequential EEG signals

 In future••Raw EEG as features: NN can learn without specific feature extractions
Detection of semantic violations in sentences for evaluations of sentences

L Vařeka and Pavel Mautner. 2017. Stacked autoencoders for the P300 component detection. Frontiers in neuroscience 11, 302.
S Alhagry, and et al. 2017. Emotion recognition based on EEG using LSTM recurrent neural network. Emotion 8, 10
S Takazawa, and et al. 2002. Early components of event-related potentials related to semantic and syntactic processes in the Japanese language. Brain Topography 14, 169
H Tanaka, and et al. 2019. Electroencephalogram-Based Single-Trial Detection of Language Expectation Violations in Listening to Speech. Frontiers in Computational Neuroscience 13, 15