Sequential Attention-based Detection ofSemantic Incongruities
from EEG While Listening to Speech

Nara Institute of Science and Technology, Japan
Shunnosuke Motomura
Hiroki Tanaka
Satoshi Nakamura
1Background: Assessment of sentences

Background
Semanticincongruities[Takazawa et.al, 2002]

Taro set out on a dictionaryl Subjective evaluation has some difficulties••-Definition of clear criteria for the evaluations
Interpretation of meaning of the word-> these subjective factors can cause biases [Bakarov, 2018]

Takazawa, S et al. (2002). Early components of event-related potentials related to semantic andsyntactic processes in the Japanese language. Brain Topography, 14,169–177.
Bakarov, A. (2018). A survey of word embeddings evaluation methods. arXiv preprint

2

Goal: Automatic evaluation using EEG

Background
EEG
Taro set out on a dictionaryl Subjective evaluation has some difficulties••

Detection

Definition of clear criteria for the evaluations
Interpretation of meaning of the word-> these subjective factors can cause biases [Bakarov, 2018]l Automatic evaluation••

Unconscious & spontaneous signals exclude subjective biases
Specific to recognition process of brains [Luck, 2014]l Purpose: Automatic detection of incongruities in sentences•-As a first step, we are aiming at detecting clear incongruities

Luck, S.J. (2014). An Introduction to the Event-Related Potential Technique, MIT Press.

3Single-trial EEG classifications

Backgroundl EEG: electrical signal of neurons••

Non-invasive
High temporal-resolution (milli-second)-> Applicable for analysis of sentence processingl Single-trial EEG: assessment of single sentence•Difficult due to the low signal-to-noise ratio•Machine learning methods are feasible for EEG classification- Recurrent neural network (RNN) handles sequential signals[Sakthi et al, 2019]- Attention-based RNN extracts important time areas for classifications[Phan et al, 2018]•-Attention-based model might not be used for EEG classification related
to cognitive processing such as sentence comprehension

Sakthi, M. et al, (2019, May). Native Language and Stimuli Signal Prediction from EEG. In ICASSP
2019 (pp. 3902-3906). IEEE.
Phan, H. et al, (2018, July). Automatic sleep stage classification using single-channel eeg: Learning
sequential features with attention-based recurrent neural networks. In (EMBC) (pp. 1452-1455).

4

Semantic incongruity[Tanaka et.al, 2019][Motomura et.al, 2019]l Related works: single-trial classification of incongruities in speech
Using EEG of time region of only the target worde.g. "  "SpeechEEG•classification

Result (Sem: condition of semantics, Syn: condition of syntax)- Sem: 59.5% (MLP), Syn: 61.3% (LSTM)

l We used EEG of whole parts of sentences because ...
•••-We cannot know which words in sentences elicit the incongruities
Timing of recognition in speech may be ambiguous
Regions of listening other words may provide classification information

Tanaka, H. et al. (2019). EEG-based Single Trial Detection of Language Expectation Violations inListening to Speech. Frontiers in computational neuroscience, 13, 15.
Motomura, S. et al (2019, October). Detecting Syntactic Violations from Single-trial EEG using
Recurrent Neural Networks. In Adjunct of the 2019 ICMI (no. 4). ACM.

5

Detecting semantic incongruities in speechOverview

Taro set out on a dictionary
Listening
Classification model

Semanticincongruityl Purpose
EEG-based classifications of semantic (in)correctness in speechl Method
FeatureModel

Previous
Target word
RNN

Proposed
Whole sentence
Attention-based RNN
6

Experiment

Methodl Spoken sentences: condition of semantic incongruities•e.g. (a: semantic correct, b: semantic incorrect)

a. Taro-ga
ryoko-ni
dekake-ta(Taro set out on a journey.)b. #Taro-ga
jisho-ni
dekake-ta(Taro set out on a dictionary.)••

Last phrase clarified the semantic (in)correctness
80 sentences for semantic condition(Semantic correct: 40 sentences, semantic incorrect: 40 sentences)l Participants: 19 native Japanese speakers
l Procedure(1) Look at '+' mark(2) Listen to the sentence(3) Press the button

1 second

4 second

2 second

Speechcorrect
orincorrect7

Classification model

Methodl Attention-based RNN [sequence to label]•Assigns importance scores (= et) at each time point (= t)

e1-eT

α1w

•
•



α
αTh1

hTx1

xTv

softmax[Febro et.al, 2017]

yht : output vector of hidden layer at time point t
w : weights vector of attention layer

Felbo, B. et al, (2017). Using millions of emoji occurrences to learn any-domain representations
for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524.

8

Training and prediction

Methodl Feature

e1



eT

α1

w

α
αT

h1



hTx1



xTv

softmax

Amplitudes of EEG (low-pass filtered at 20Hz)-> 31 dimensions at each time (equivalent to number of the channels)

31 Chs•yxt: amplitudes at time ty : one-hot label(correct / incorrect)

9Classificationl Data (number of participants / sentences)
Train
Develop
11 / 856
2 /156•••

MethodTest
4 / 310

Number of correct and incorrect sentences are the same-> Chance level of classifications is 50%
Standardization of input vectors
Augmentation of training data by adding Gaussian noise-> For avoiding overfitting to the small training datal Model•

1 layered bidirectional GRU (with / without attention machanism)l Optimization of hyper-parameters•

10-fold cross validation within training and develop dataset- Dimension of hidden layer= {5, 10, 20}- Size of data augmentation (times) = {5, 10, 20}- L2 regularizer weights= {0, 0.0001, 0.001, 0.1}

10

Classification performances

Resultsl Classification accuracy, recall and precision

Accuracies of each model

0.75

GRU w/ att. (Whole-sentence)
GRU w/o att. (Whole-sentence)
GRU w/ att. (Terminal-phrase)
GRU w/o att. (Terminal-phrase)

0.70

Accuracy

0.65
0.60
0.55
0.50
0.45
0.40

123
Participant number on test set

411

Attention weights of the best model

Resultsl Successful cases of the classification•

Predicting incorrectness•Attention weights of these two patterns are different•

Red broken lines in the plot shows the onset of last phrases-> For predicting semantic incorrectness,
attention weights focused on time region of the last anomalous word

•Predicting correctness

12

Discussions and conclusions
l Our model classified semantically correct or incorrect sentencesusing EEG of whole length of sentences with attention models
...

...

••

Classifi.
ModelIncorrect
Correct

Attention mechanism worked for the sequential feature extraction
Predictions depended on the attention weights like...
: predicting incorrectness: predicting correctness

l Future works•••-

Investigation of performances on sentences including various word lengths
Comparison with other feature extractions such as time-frequency features
Predicting words' semantic expectations in sentences [Kutas et.al, 1984]

Kutas, M. et al, (1984). Brain potentials during reading reflect word expectancy and semanticassociation. Nature, 307(5947), 161.

13