Meta-Learning for Emotion Prediction
from EEG while Listening to Music
Nara Institute of Science and Technology, Japan
Center for Advanced Intelligence Project, RIKEN, Japan
Kana Miyamoto, Hiroki Tanaka, Satoshi Nakamura

12022/5/24

TAPAS SUPPORTED BY ANR-CREST

Emotion Induction using Music
Appropriate emotional induction is critical in mental healthMusic◦ Musical components influence emotions [Wallis+ 2011]◦ Music is used to induce emotions [Sourina+ 2012]◦ Music does not interfere with other activities

Application example
Emotional induction in presentation◦ Emotions felt while listening to music vary depending on individuals and situations [Larsen+ 1991]

Personalized music is needed for emotion induction
2TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Proposed Emotion Induction System
System generates music in real time based on participant’s emotions

Emotion prediction
using EEGMusic generation

Feedback

We demonstrated effectiveness of system [Miyamoto+ 2021 (Under review)]
3

TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Problem of Our Emotion Induction System◦ Emotion prediction models were trained for each participant to consider individuality of EEG◦ Long EEG recording was required to avoid lack of data
Process of using our system
Day 1: Data collection

Day 2: Using system
Training model

Large burden

EEGEmotionEEG

Research objective
Predicting emotions with performance accuracy using small amounts of EEG data
4

TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Training with Small Amounts of Data
Transfer learning◦ Apply pre-training model to another domain◦ Acquire highly accurate models with small amounts of data [Pan+ 2009]

Previous works using transfer learning for EEG [Lan+ 2008, Miyamoto+ 2020]◦ Pre-training model treated EEG of multiple participants as one piece of data

EEG

Training cannot take into account personal character of EEG
5TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Model-Agnostic Meta-Learning (MAML)
Acquire initial parameters 𝜃 of model from tasks and adapts them to a target task [Finn+ 2017]
MAML in EEG research
Task 1EEG

Task 2

Initial parameters of model𝜃𝜃#"(Task 2)

EEG
EEG

𝜃$"
(Task 3)

Task 3
One task is one participant’s EEG
6𝜃!"(Task 1)

𝜃 can easily be adapted to any participants

TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Related Works about Prediction using EEG with MAML•Classification of sleep stages [Banluesombatkul+ 2020]•Classification of motor imaginary tasks [Denghao+ 2021]•Predicting emotions while watching music videos or movies [Duan+ 2020]

Music of auditory stimulation is necessary for construction of emotion induction system
We investigated whether MAML is also effective for EEG while listening to music

7

TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Research Overview
Research objective
Predicting emotions with high performance using small amounts of EEG data while listening to music

Methods◦ Proposed method : Multiple participants’ EEG with MAML
◦ Baseline method 1: Multiple participants’ EEG without MAML [Lan+ 2008, Miyamoto+ 2020]◦ Baseline method 2: Single participant’s EEG [Ehrlich+ 2019, Miyamoto+ 2020]

Hypothesis
Method with MAML can reduce error of emotion prediction compared to the other two methods

8TAPAS SUPPORTED BY ANR-CREST

2022/5/24

EEG Dataset
Dataset created in our previous work [Miyamoto+ 2020]
Participants

10 males, 10 females (age: 24.3 years)

Electroencephalograph

Quick-30 manufactured by CGX

Affective stimuli41 pieces of music

Length of one piece of music

20 secEmotions

Valence and arousal (9-point scale between 0 and 1)

Quick-30
9TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Features and Model
Preprocessing
1. EEG while listening to music was divided into 1 sec
2. EEG was passed second-order IIR bandpass filters
3. Features for each of the five frequency bands 𝑓 = 𝑙𝑜𝑔(𝑣𝑎𝑟 (𝐸𝐸𝐺𝑑𝑎𝑡𝑎))
4. Mapped matrix reflecting the position of the EEG channels

AF3 AF4F7 F3FC5F4 F8FC6T7P7T8O1 O2P8Placement of channels
1000 00 AF3AF400 00
AF400 00 AF3AF3AF400 00
AF3AF400AF3AF40F700 F3F400 0F8Mapping
F7F3F4F8F700 F3F400 F8F7F3F4F70F3F400 F8F800 FC500FC600 00 FC60000 FC5FC5FC600 00 FC60000 FC5FC5FC6
T7000T8
T700 00 00 00 T8
T7
T700 00 00 00 T8
T8
T7T8
P70000P8
P70000P8
P700 00 00 00 P8P7Gamma
P70 000 00 P8P8O2High beta 000 000 O1O1O200O2 00 00
00 00 O1
O1
Low beta
O1 O2
O2 0 0AlphaTheta
TAPAS SUPPORTED BY ANR-CREST

CNN

2022/5/24

valencearousal

Proposed Method
Multiple Participants’ EEG with MAML
◦ Fine-tune model trained with MAML using a single target’s EEG= 1 task
Training data
10 participants

5-fold cross validation Test data
20 pieces of music

Validation data
9 participants

Initialization!

Test

Fine-tuning!"#Update !
Validation

11TAPAS SUPPORTED BY ANR-CREST

2022/5/24

TargetBaseline Method 1Multiple Participants’ EEG without MAML◦ Fine-tune model trained without MAML using a single target’s EEG◦ This method might not be able to grasp EEG individuality
× 10 = 1 data
Training data
10 participants

Validation data
9 participants

Initialization!

5-fold cross validation Test data
20 pieces of music
Test

Fine-tuningValidation

12TAPAS SUPPORTED BY ANR-CREST

2022/5/24

TargetBaseline Method 2Single Participant's EEG◦ Train model using only a single target’s EEG◦ This method might lack training data

No data other than target data

5-fold cross validation Test data
20 pieces of music

Initialization!

13TestTAPAS SUPPORTED BY ANR-CREST

2022/5/24

TargetResults
Range of labels of valence and arousal in dataset is between 0 and 1
There were significant differences by Wilcoxon signed-rank test in the following methods (p<.001)◦ With MAML and Without MAML
◦ With MAML and Single par.
◦ Without MAML and Single par.

The means of RMSE for 20 participants
Proposed method
Baseline method 1
Baseline method 2With MAML
Without MAML
Single par.
valencearousalvalencearousalvalencearousal

0.244

0.287

0.300

0.321

0.367

0.377

RMSE is root mean square error between predicted values and label values
14

TAPAS SUPPORTED BY ANR-CREST

2022/5/24

Conclusion
Summary of the results
Multiple participants vs. a single target participant◦ Method using EEG of a single target considers individuality of EEG, but its performance was low

MAML vs. with MAML
◦ Using MAML had high performance◦ MAML was effective in constructing our emotion induction system using music

Future work
◦ Applying models trained by MAML to our emotion induction system◦ Investigating effects of reducing preparation time for using system

15

TAPAS SUPPORTED BY ANR-CREST

2022/5/24