Paper ID: 81

Emotion Prediction Using Multi-source Biosignals During
Cognitive Behavior Therapy with Conversational Virtual Agents
Nara Institute of Science and Technology, Japan
Kana Miyamoto, Hiroki Tanaka, Kazuhiro Shidara, Satoshi Nakamura
2023/12/5

1

TAPAS SUPPORTED BY ANR-CREST

Overview
Purpose• Emotion prediction using electroencephalogram (EEG) during mental health counseling

Challenge and our approach• EEG is sensitive to speech noise• Adopt multi-source biosignals to achieve high accuracy and robustness of emotion prediction
Electroencephalogram(EEG)
Facial expressions
Electrocardiogram(ECG)
2

TAPAS SUPPORTED BY ANR-CREST

Emotion Prediction

Background

3

TAPAS SUPPORTED BY ANR-CREST

Emotion Prediction for Mental Health Counseling System• Systems have been developed to provide daily mental health care [Fitzpatric+ 2017]
Therapist

System [Shidara+2022]
Question based on
patient’s emotions

Question not based onuser’s emotions

AnswerAnswer

Application of emotion prediction would improve effectiveness of the system
4

TAPAS SUPPORTED BY ANR-CREST

Emotion Prediction Using Electroencephalogram
Electroencephalogram (EEG)• Electrical activity from brain• EEG signals reflects emotions even when we are not conscious of them
• Emotion prediction based on machine learning models has been proposed
EEG signals

Emotion prediction

Channels？
Time

5Machine learning

TAPAS SUPPORTED BY ANR-CREST？

Noise Affects EEG Signals
EEGsignals
Examples of noise sources
Mouse
EEG signals with noiseEyeHeart

6Noise masks EEG features and negatively affectaccuracy of emotion prediction [Kanoga+ 2017]➡ Difficult to use EEG during conversation

TAPAS SUPPORTED BY ANR-CREST

Towards Emotion Prediction Using EEG During ConversationEmotion prediction using EEG + other biosignals is gaining interest◦ Development of K-EmoCon dataset during discussion [Park+ 2020]◦ Emotion prediction during talking [Saffaryazdi+ 2022]◦ Applying EEG, photoplethysmography and galvanic skin response

Our study uses EEG and other biosignals during mental health counseling
7

TAPAS SUPPORTED BY ANR-CREST

Emotion Prediction During Mental Health CounselingContributions• Data collection of biosignals during mental health counseling• Applying EEG, ECG and facial expressions (FE) for training models• Comparison of accuracy of four models:① EEG model

8② ECG model② FE model

TAPAS SUPPORTED BY ANR-CREST④ EEG+ECG+FE modelData Collection

9TAPAS SUPPORTED BY ANR-CREST

Experimental SetupParticipants• 22 males and 7 females (age: 24.5±3.24 years old)

Sensors
• EEG
• ECG

: Quick-32r, manufactured by CGX: Extension of Quick-32r

EEG• Facial expression : Video camera
ECG

Cognitive behavior therapy (CBT) system [Shidara+ 2022]• CBT is a mental health care technique to resolve problems by modifying negative thought• The system provide predefined questions for changing the negative thought
10

TAPAS SUPPORTED BY ANR-CREST

Procedure＋
Baseline (5 sec)＋
ListenEmotionevaluation

Rest (5 sec)

＋Answer

Rest (5 sec)

Has anything been bothering you lately,
that’s difficult to deal with or to face?
System11

TAPAS SUPPORTED BY ANR-CREST

Emotionevaluation

Procedure＋
Baseline (5 sec)＋
ListenEmotionevaluation

Rest (5 sec)＋
Answer

Emotionevaluation

Rest (5 sec)

Self-Assessment
Manikin[Bradley+1994] using mannequins.
Valenceand arousalassessment• Valence (pleasant-unpleasant)and Arousal
Evaluate values
from 0 to(activated-deactivated)1
assessment using mannequins• Evaluate values from 0 to 1

12

TAPAS SUPPORTED BY ANR-CREST

Procedure＋Baseline (5 sec)

＋ListenEmotion
evaluation

Rest (5 sec)

＋Answer

Rest (5 sec)

I'm having problems with my friendships
Participant

13

TAPAS SUPPORTED BY ANR-CREST

Emotionevaluation

Model Training

14

TAPAS SUPPORTED BY ANR-CREST

Regression Models for Emotion Prediction
EEG signals

ECG signals

FeatureFeature

Training models for each biosignal
EEG・ECG : CNNFE: Multi-perceptron<Example>valence prediction is 0.2

extraction

extraction

ConvConv

BNBNReLU

ReLU

valencearousalvalencearousalPredicted emotions from the models wereconnected to support vector regression (SVR)

15

TAPAS SUPPORTED BY ANR-CREST

FE
Featureextraction

FC
ReLU
valencearousalSVR

SVR

valencearousalComparison of Emotion Prediction Models• Conducting 5-fold cross-validation for each participant• Evaluation using concordance correlation coefficient (CCC)• Agreement between predicted and true values• The closer to 1, the higher the degree of agreement

CCC of predicted and true values
EEG (baseline)

ECG

FE

EEG+ECG+FE

valence

0.355

0.016

0.292

0.460*

arousal

0.350

0.017

0.294

0.455*

Wilcoxon signed-rank test＊p<0.05

EEG+ECG+FE model improves accuracy of emotion prediction
during mental health counseling
16

TAPAS SUPPORTED BY ANR-CREST

Conclusion
Purpose• Emotion prediction using EEG during mental health counselingContributions• Data collection of biosignals during mental health counseling• EEG+ECG+FE model improves accuracy of emotion prediction

Future work• Match of ratio of male to female participants of dataset• Investigation of more effective fusion model structures and features such as speech
information

17

TAPAS SUPPORTED BY ANR-CREST