Emotion Estimation from EEG Signals andExpected Subjective Evaluation
D i v i s i o n o f In forma ti on S ci ence, N ara Insti tute of S ci ence and Technol ogy, JapanK ana Mi yamoto, H i roki Tanaka, S atoshi N akamura

1

2021/2/26

TAPAS SUPPORTED BY ANR-CREST

Emotion induction using music
IndividualSituation

A

B

Emotions felt while listening to music vary depending on individual and situation
Music selection based on participants’ current emotion is required
2

TAPAS SUPPORTED BY ANR-CREST

2021/2/26

Proposed overall system [Miyamoto et al., 2020]
arousalvalenceCalculation of inputs
to music generator
arousalDesiredemotionvalence
Estimated
emotionEstimated
emotionEmotion estimation
arousalEstimated
emotion

EEG
ParticipantMusic generator
Music3TAPAS SUPPORTED BY ANR-CREST

valence

2021/2/26

Proposed overall system [Miyamoto et al., 2020]
arousalvalenceCalculation of inputs
to music generator
arousalDesiredemotionEstimated
emotionEstimated
emotionvalence

Emotion estimation
arousalEstimated
emotionvalence

EEG
ParticipantMusic generator
MusicThe purpose of this paper
Improving the performance of emotion estimation used for emotion induction
4

TAPAS SUPPORTED BY ANR-CREST

2021/2/26

Music generator for inducing emotions [Miyamoto et al., 2020]• The music generator made music that induces emotions similar to the inputs• From the evaluation of the music generator, we concluded that it effectively induced emotions

Music generator

InputsParticipant

tempo

valence(0~1)
arousal
(0~1)

5

piano

rhythmCalculation ofparameters

loudness
pitch

Generation ofMIDI signals

mode

TAPAS SUPPORTED BY ANR-CREST

violinMusicgeneration

cello2021/2/26

Music

Music generator for inducing emotions [Miyamoto et al., 2020]• The music generator made music that induces emotions similar to the inputs• From the evaluation of the music generator, we concluded that it effectively induced emotions

Music generator

InputsParticipant

tempo

valence(0.875)
arousal(0.875)

6

piano

rhythmCalculation ofparameters

loudness
pitch

Generation of
MIDI signals

mode

TAPAS SUPPORTED BY ANR-CREST

violinMusicgeneration

cello2021/2/26

MusicEmotion estimation of related studies
Emotion estimation using EEG only [Ehrlich et al., 2019] [Miyamoto et al., 2020]• Related studies used linear regression and convolutional neural network (CNN)
Emotion estimation
Estimated emotion

arousalEstimatedemotion

Music generator
valence (0~1)
arousal (0~1)

7InputsTAPAS SUPPORTED BY ANR-CREST

valence

EEG
ParticipantMusic2021/2/26

Proposed emotion estimation
Emotion estimation using EEG and expected subjective evaluations• We regarded the inputs of the music generator as expected subjective evaluations
Emotion estimation
Estimated emotion

Estimated emotion

arousalEstimatedemotion

valencevalence (0~1)
arousal (0~1)

arousalEstimatedemotion

Music generator

Inputs8Emotion estimation

InputsTAPAS SUPPORTED BY ANR-CREST

valence

EEG
ParticipantMusic2021/2/26

EEG recording
Participants
20 healthy undergraduate and graduate students

Electroencephalograph
Quick-30 manufactured by CGX

Stimuli
41 pieces of music created by the music generator
Quick-30

9TAPAS SUPPORTED BY ANR-CREST

2021/2/26

EEG recording
Procedure
Silent (5 s)
Listening to music (20 s)

Evaluation of emotions
using self-assessment mannequin[Bradley et al., 1994]

EEG

10recording

Display of SAM(25s)

TAPAS SUPPORTED BY ANR-CREST

2021/2/26

Preprocessing of EEG
1. The EEG in silence and listening to music was divided into 1 s
2. We designed second-order IIR bandpass filters
3. The features for each of the five frequency bands ! = "#$(%&' ((()*&+&))
4. We mapped the matrix reflecting the position of the EEG channels
Fp1
AF3
F7 F3Fz

FC5
T7C3CP5

Fp2
AF4F4 F8FC6Cz

C4T8CP6P3 Pz
P4P7 PO3PO4 P8PO7PO8O1O211TAPAS SUPPORTED BY ANR-CREST

AF3FP1FP2AF4AF3FP1FP2AF4AF3FP1FP2AF4AF3FP1FP2AF400AF3000AF4 0F8 0F7F3FZ
F4F7F3FZ
F4F8F7F3FZ
F4F8F7F3FZ
F4F8F70F3000F40F8FC5FC6FC5FC6FC5FC60 FC5FC500CZ 00C4 0 FC6FC60T7C3T8T7C3CZ
C4T8T7C3CZC4T8T7T70 C300 CZ00 C4CP600 T8T8CP5CP5CP6CP5CP60 CP50P3 00PZ 00P4 0 CP60P8 0P7P7P3PZ
P4P8P7P3PZ
P4P8P7P3PZP4P8P70000000P8PO7PO3PO4PO8PO7PO3PO4PO8PO7PO3PO4PO80 PO700O1PO300O2PO400 PO800O1O2O1O2000 O1O1 0 O2O2 000

2021/2/26

Comparison of two methods
1. CNN using EEG only [Miyamoto et al., 2020]

12TAPAS SUPPORTED BY ANR-CREST

2021/2/26

Regression
output layerFully connectedlayer

Dropout layer

ReLU layer

Batchnormalization layer

Convolution layer

ReLU layer

AF3FP1FP2AF4AF3FP1FP2AF4AF3FP1FP2AF4AF3FP1FP2AF400AF3000AF4 0F8 0F7F3FZ
F4F7F3FZ
F4F8F7F3FZ
F4F8F7F3FZ
F4F70 F3 000 F40 F8F8FC5FC6FC5FC6FC5FC60 FC5FC500CZ 00C4 0 FC6FC60T7C3T8T7C3CZ
C4T8T7C3CZ
C4T8T7T70 C300 CZ00 C4CP600 T8T8CP5CP5CP6CP5CP60 CP50P3 00PZ 00P4 0 CP60P8 0P7P7P3PZ
P4P8P7P3PZ
P4P8P7P3PZ
P4P8P70000000P8PO7PO3PO4PO8PO7PO3PO4PO8PO7PO3PO4
PO80 PO700O1PO300O2PO400 PO800O1O2O1O2000 O1
O1 0 O2
O2 000Batch
normalization layer

Training that takes into account the positional relationship of EEG channels

Convolution layer•Comparison of two methods
2. Neural network using EEG and inputs of the music generator

Inputs of the music generator

13TAPAS SUPPORTED BY ANR-CREST

2021/2/26

Regression
output layer

d1

Fully connectedlayer

MethoDropout layer

CNN

ReLU layer

EEG

Fully connectedlayer• Emotion estimation using emotions estimated from EEG and the inputs to the music generator

RMSE of felt and estimated emotions
Range of felt emotions
valence: 0~1, arousal: 0~1

For the Wilcoxon signed-rank test result, we found a significant difference between neural network andCNN (p<0.05)
The means of RMSE for 20 participants
2. Neural network using
EEG and inputs of the music generator

1. CNN using EEG
valencearousalvalencearousal

0.214

0.239

0.151

0.164

RMSE of felt emotions and inputs of the music generator valence: 0.232 arousal: 0.213
14

TAPAS SUPPORTED BY ANR-CREST

2021/2/26

Conclusion
Our purpose
Improving the performance of emotion estimation used for emotion induction

Prosed model
Neural network using EEG and inputs of the music generator

ResultThere was a significant difference between the proposed neural network and CNN using EEG

Future work
Construction and evaluation of the proposed emotion induction system
15

TAPAS SUPPORTED BY ANR-CREST

2021/2/26