VBH-GNN: Variational Bayesian Heterogeneous Graph Neural Networks for Cross-subject Emotion Recognition
Authors: Chenyu Liu
XINLIANG ZHOU
Zhengri Zhu
Liming Zhai
Ziyu Jia
Yang Liu
Venue: ICLR 2024 poster (2024)
Keywords: EEG
Emotion Recognition
Multi-modal
Domain Adaptation
Cross-subject
Primary area: applications to neuroscience & cognitive science

Abstract: The research on human emotion under electroencephalogram (EEG) is an emerging field in which cross-subject emotion recognition (ER) is a promising but challenging task. Many approaches attempt to find emotionally relevant domain-invariant features using domain adaptation (DA) to improve the accuracy of cross-subject ER. However, two problems still exist with these methods. First, only single-modal data (EEG) is utilized, ignoring the complementarity between multi-modal physiological signals. Second, these methods aim to completely match the signal features between different domains, which is difficult due to the extreme individual differences of EEG. To solve these problems, we introduce the complementarity of multi-modal physiological signals and propose a new method for cross-subject ER that does not align the distribution of signal features but rather the distribution of spatio-temporal relationships between features. We design a Variational Bayesian Heterogeneous Graph Neural Network (VBH-GNN) with Relationship Distribution Adaptation (RDA). The RDA first aligns the domains by expressing the model space as a posterior distribution of a heterogeneous graph for a given source domain. Then, the RDA transforms the heterogeneous graph into an emotion-specific graph to further align the domains for the downstream ER task. Extensive experiments on two public datasets, DEAP and Dreamer, show that our VBH-GNN outperforms state-of-the-art methods in cross-subject scenarios.

Published as a conference paper at ICLR 2024
VBH-GNN: VARIATIONAL BAYESIAN HETEROGE- NEOUS GRAPH NEURAL NETWORKS FOR CROSS-
SUBJECT EMOTION RECOGNITION
Chenyu Liu'l, Xinliang Zhou'!, Zhengri Zhu", Liming Zhai Ziyu Jia Yang Liu'
School of Computer Science and Engineering, Nanyang Technological University
School of Electronic and Information Engineering, Beijing Jiaotong University
"School of Computer Science, Central China Normal University
"Institute of Automation Chinese Academy of Sciences
{chenyu0C3, xinliang001 @ee ntu. edu. sg, chengrizhu@bia edu. cn, @mingghaai@com cenu. edu. en, jia. z
com, yangliuentu edu. sg
ABSTRACT
The research on human emotion under electroencephalogram (EEG) is an emerg- ing field in which cross-subject emotion recognition (ER) is promising but challenging task. Many approaches attempt to find emotionally relevant domain- invariant features using domain adaptation (DA) to improve the accuracy of cross- subject ER. However, two problems still exist with these methods. First, only single-modal data (EEG) is utilized, ignoring the complementarity between multi- modal physiological signals. Second, these methods aim to completely match the signal features between different domains, which is difficult due to the extreme individual differences of EEG. To solve these problems, we introduce the com- plementarity of multi-modal physiological signals and propose new method for cross-subject ER that does not align the distribution of signal features but rather the distribution of spatio-tempon relationships between features. We design a Variational Bayesian Heterogeneous Graph Neural Network (VBH-GNN) with Relationship Distribution Adaptation (RDA). The RDA first aligns the domains by expressing the model space as posterior distribution of heterogeneous graph for a given source domain. Then, the RDA transforms the heterogencous graph into an emotion-specifer graph to further align the domains for the downstream ER task. Extensive experiments on two public datasets, DEAP and Dreamer, show that our VBH-GNN outperforms state-of the-art methods in cross-subject scenarios.
INTRODUCTION
Emotion is a complex physical and psychological state that plays vital role human decision- making, behavior, and interpersonal interaction (Cabanac (2002)). In recent years, emotion recog- nition (ER) has become increasingly important in fields such as diagnosis of depression and human- computer interaction (Shneiderman al. (2016); Zhou et al. (202)). To study emotions, researchers usually record changes in body language (Coulson) (2004)), voice (Zeng et al.) (2007), expression (Ekman (984)), and physiological signals (Gunes et al. (201T)) after inducing emotions in subjects. Since external cues are easy to control, they are not guaranteed to reflect the actual emotional state of the subject. contrast, physiological signals cannot be disguised, capturing subjects' potential reactions that reveal real human emotions (Mühl et al. (2014).. Electroencephallggaan (EEG), as high-resolution and effective physiological signal, therefore widely used in emotion recognition (ER) (Calvo & D MMelo (0100: Liu et al. (2024)).
Compared to conventional ER, cross-subject ER is undoubtedly more appealing but also presents greater challenges. First, EEG has high individual differences. Since emotion comprehensive reflection of human physiology and psychology, the differences in anatomical structures, biological rhythms, personalities, and psychological states among subjects can lead to different responses to
Co-first Authors, "Corresponding Authors

---

Published as a conference paper at ICLR 2024
EEG
ECB
E0G
G5R
Cubeewoux System
Mydret p
www.aoaaoaann www
Carclovssurirr System
Norvous Systom
Perceptual System
Hoterogeneity
Interactivity
Figure I: Heterogeneity and intricate interactivity. The heterogeneity means the difference among the signals from different modalities. The intricate interactivity refers to the multi-modal signals belonging to multiple physiological systems that dynamically cooperate to regulate human emotions.
the same emotional stimuli (Wan et al. 2021 Second, EEG typically has low signal-to signal-tonoois noise ratio due to internal and external factors. Internally, EEG often contains substantial biological noise from muscle movements. Externally, non-invasive EEG acquisition techniques limit the sampling rate and are susceptible to interference. These reasons restrict the robustness and generalizability of cross-subject ER in practical applications.
Existing work has demonstrated that multi-modal physiological signals can improve the accuracy of ERR Additionally, domain adaptation (DA) approaches have been explored for EEG-based cross- subject ER. However, no studies have yet combined multi-modalities and DA for cross-subject ER due to the intricate interactivity of multi-modal signals (see Figure I (b)). Current methods have two limitations: 1) they fail to capture the spatio-temporal relationships between modalities while considering their heterogeneity, and 2) they do not utilize such spatio-temporal relationships to align different domains.
For the first problem, although most methods heterogeneity (see Figure I (a)) of multi-modal data (Lan et al. (2020): Ma et al. (2019); Mittal et al. (2020); Wu al. (2024 2023a): Wang al. (2022). ). they ignore the spatio-temporal relationship between the modalities during the feature fusion stage. Other methods combine
modality-speciinn feature extractors to capture the
modil signals to capture spatio- temporal relationships (Zhang et al. (2020); Wu et al. (2022) 2023b); Liu et al.
(2023); et al.) 2 ) but the heterogeneity between modalities is neglected in the feature ex- traction process. For the second problem, existing DA methods (She et al. 2023); Peng et al. (2022) are stuck on completely matching the EEG feature distributions between different domains, which is difficult because individual differences. They ignore utilizing the spatio-temporal relationships of modalities between subjects to build connections among domains.
To solve these problems, we introduce new approach to align source and target distributions by multi-modal spatial-temporal relationships to achieve cross-subject ER. We propose Variational Bayesian Heterogencous Graph Neural Networks NBBH-NNN, which integrate modalities tempo- ral and spatial Relationship Distribution Adaptation (RDA) between domains and ER into one frame- work. Specifically, the RDA contains Bayesian Graph Inference (BGI) and Emotional Graph Trans- form (EGT). The BGI models multi-modal relationships as heterogeneous graphs (HeIG) and aligns the relationships of domains via the edge distribution of HetG based on the Variational Bayesian the- orem. EGT transforms the HetG into emotion-speciffe graphs (EmoG), further aligning the source and target domains while differentiating the relationships of modalities under different emotions. After the joint constraints of these two steps, the VBH-GNN can infer the comain-invariant multi- modal spatio-temporal relationships between source and target domains and utilize this relationship to weight the signal feature and feed to the classifier for cross-subject ER. We perform extensive quantitative and qualitative experiments multiple datasets, and the results demonstrate that our VBH-GNN significantly outperforms previous ofate-of-the-arr methods.
We make the following main contributions: 1) introduce novel deep learning architecture named VBH-GNN that combines multi-modal physiological signals and domain adaptation (DA) for more accurate cross-subject ER. 2) propose Relationship Distribution Adaptation (RDA) to align source and target domains through mudti-modal spatio-temporal relationship distributions. 3) develop Bayesian Graph Inference (BGI) to model and align intricate interactivity of modali-
2

---

Published as a conference paper at ICLR 2024
Wav Node
Spatial RDA
Temporal RDA
888 Source
Node to Edge
Node to Edge
M
Classifier
BGI
EGT
BGI
EGT
8 Target
V
Graph Allention
Graph Attention
Figure 2: Flowchart of VBH-GNN.
ties in source and target domains. 4) we design a graph learning module named Emotional Graph Transform (EGT) optimize the aligned domain distribution for downstream ER tasks.
2 PRELIMINARIES
Mulit-modal Domain Adaptation. This paper focuses on the scenario where the source and target domains have several modal types (i.e., EEG, Electrocardiography (ECG), Galvanic Skin Response (GSR), etc.). We denote the source domain D. { (X : X.EG Y.) where (X.EG XxECG) are multi-modal data with label Y,. For the target domain, donate limited number of labeled target data by Die { ,,.,, "r X Yu.) }. and unlabelled target data D., {( 1.. XECG)) The ER model is trained on D. and Di. and evaluated Dut.
Heterogeneous Graph. We define a HetG as G = (V, E, o, vh), where V and E denote the node set and edge set. Φ > Tv maps the node to its corresponding type in node type set Tv. and similarly v E + TE stands for the edge mapping. For a HetG, [Tvl + TTll 2.
3 VARIATIONAL BAYESIAN HETEROGENEOUS GRAPH NEURAL NETWORKS
3.1 OVERVIEW
As shown in Figure there are four components included in our cross-subject ER process: Wav to Node, 2) Spatial RDA and Temporal RDA, and 3) Classifier.
Wav-to-Node employs the same setup in (Jia et al. (202T)), transforming multi-modal signals into node embeddings with modal-specific networks to capture the heterogeneity of modalities. accepts multi-modal signals D. and Dir from the source and target domain as input and converts them into node embeddings Vs, Vi BBBNN,
V1. U ((X),
(I)
where X; and Xi, represents the signal of modality i in two domains, and fi(-) represents the modality-specifie feature extraction network of modality i.
RDA is the core component of VBH-GNN. It accepts node embeddings for domain alignment and updates the weights of node embeddings. VBH-GNN contains Temporal RDA and Spatial RDA, which perform inference and alignment of relationship distributions in the temporal and spatial dimensions. The details of RDA will be explained in Section 3.2 The overview process of RDA is follows:
(V', Vi) = SRDA(V,, Vi.)
(V.", V'' TRDA(FIRANS (AANfIAAAS((ENN/(// (VV, VD (V., Vi.))
where SRDA(-) and TRDA(-) represent Spatial RDA and Temporal RDA. feNN is a ferw-forward network. fIRANS() is operation that transforms spatial nodes into temporal nodes.
Classifier takes the output of the RDA and predicts labels for domains with classification network:
(V', Y() fa(V." Vil
Loss of VBH-GNN contains two types of loss: the RDA Loss for aligning the source domain and target domain and the prediction loss of the classifier. The final loss function is formulated as
LVBUIGNN
CSRDA ARCTRDA + AgLeDCE + A&CTACE
3

---

Published as a conference paper at ICLR 2024
Relationship Distribution Adoptation (RDA)
V.
V,
Edge to Distribtion
Edge to Distribution
Coupling
Node
PQHICEE
NiZ
perpani
to Edge
BGI
BGI LOKK
EGT
Ni Z
EGT ne para
Attention
PIHeSGIE))
Edge to Distribution
Edge to Distribution
Relationship distribution adaptation (RDA). It contains -odeee, Bayesian Graph Inference (BGI), Emotional Graph Transform (EGT), and Graph attention (GA).
Figure 3:
where LSRDA and LIRDA are loss of Spatial RDA and Temporal RDA (will be further explained in Section (.2.3), LSBCE and LтвCE are Binary Cross-Entrope Loss for source and target domain classification. À1 X2, λ3, and da are loss weights, which are all set to in the experiments.
3.2 RELATIONSHIP DISTRIBUTION ADAPTATION
RDA implements two of the most critical functions of VBH-GNN, the modeling of multi-modal relationships and the alignment of domains. As shown in Figure it contains four procedures:
Node-to-Edge accepts node embeddings input. It denotes the multi-modal spatio-temporal rela- tionship as a HetG by generating an edge embedding between every two nodes through convolu- tional layer Semb(.).
E., Eu. = Jamb (V. Vi((.,)) (i.1 [L,m]ii! =
(6)
where V,, Vi xD. represents the input node objects, V. VaL. BxN. D2x denotes the combination of two node embeddings, and Eg, Fis E R BBNNx D. represents the edge embedding set.
finds the latent relationship distribution of multi-modal physiological signals shared by the source and target subjects. It couples the intricate interactivity between modalities and expresses the HetG edge distribution of the target domain the posterior distribution of source domain HetG edges based on the Variational Bayesian theorem. The source and target domains are aligned via BGI Loss (see Figure 4b)). The BGI process as follows:
BGI
BaG BGI(E,, Eq.)
where ZHe RR Bx is sampling of HetG edge distribution.
EGT distinguishs between the latent relationship distribution founded by BGI in different emotions. transforms the HetG into EmoG that discriminates different emotion distributions the source and target domains (see Figure (c)). The EGT process is as follows:
Zemo EGT(Z E., Ex))
(8)
where Zimo BBNN the sampling EmoG edge distribution of source and target domains.
Graph Attention (GA) is an attention module that updates the node embedding weights Specif- ically, the output of EGT transformed T(-) as an adjacency matrix and multiplied with node embedding Vevit that passes through linear layer f(-). The final output of RDA is defined as
Vivie T(Z vie) {(Vevie)
3.2.1 BAYESIAN GRAPH INFERENCE (BGI)
BGI models the alignment of the HerG edge distributions in the source and target domains Variational Bayesian inference process. First, define the intricate interactivity of multi-modal
4

---

Published as a conference paper at ICLR 2024
signals the coupling of an infinite number of physiological system relationships. Specifically, an infinite number edges {e" E N] exist between node U; and üj. and the likelihood of an edge existing is very small Vn N,p" + We therefore define the distribution eig as
P(e,.) = Σ BER()")
lim pl, n n
(10)
Pi.
where (ER()) (-) is the Bernouli distribution. (IN(.) is the Binomial Distribution representing the coupling of infinite edges between two nodes. Pi.j is the parameters of the Binomial distribution computed by the neural network. From this, we define the prior HetG edge distribution from the source domain follows:
P(HerG)E.) p.)
(11)
where ps E Bx and n -+ 00.
Unlike VRNN (Chung et al. (2015). which applies an RNN encoder to estimate the parameters of the approximate posterior distribution, the posterior distribution of HetG edges cannot be computed similarly due to the presence of infinite n. According to De Moivre-Laplace theorem in (Sheynin) 1977 Liu & Jia (2022)), the Binomial Distribution (n, Pt.5)) can be infinitely approximated by Gaussian distribution N (mpiy, npt, Pt.j)) with
cc. To further circumvent the straight- forward parameterization of pi. and parameterize a Gaussian distribution A (ūa,j of,)), where Bi. and āi.j are calculated from Ei E Eu by neural network. First, we define an intermediate variable Aigi
1
A =
~ ((Dis) +
(12)
1- 25...
where Xi is an approximation used to prevent explosions in the calculation. e is small hyperpa- rameter that is the lower bound for Xi.j. We define a Gaussian distribution as follows:
1 / 4
(13)
P(e.s) ~ NNP. pi.))
(14)
P(ecs) is Gaussian proxy the Binominal distribution Pi,) with minimal constant divergence. We define the HetG edge distribution of the target domain as the posterior distribution as follows:
P[HetG Ea) ~ A (mm, pm (1 ppa)
(15)
where pic BxN. From this, we can apply re-parametrization (Kingma & Welling (001)) to draw samples form the distribution HetG edges:
ZHer Vun(1 mu) c pi
(16)
3.2.2 EMOTIONAL GRAPH TRANSFORM (EGT)
The core concept of EGT that it transforms innumerable intricate relationships into recognizable notion of emotion in both source and target domains. EGT learns Gaussian variable conditioned on the HetG edge to perform weighted transformation on the HetG edges, and the final graph obtained is the EmoG representing the informative emotion representation in each domain. We assume such Gaussian variable follows:
NevilZZenn (Znua X Pavit, Zuag dovte)
(17)
where ZHaG represents the HetG edge sampling from Eq. 16 and is utilized as a conditional restriction that prohibits the EmoG edge from acting arbitrarily when certain HetG edge sample values are near zero. Pavit and σ.vit represent the parameters of Gaussian distributions computed from two domains through the neural network during the EGT procedure. Then, we transform HetG by using the sampling of Gaussian variables Novie as weights:
7Emol Zitt Jevit x ZMaG flsvte) x Zuta
(18)
5

---

Published as a conference paper at ICLR 2024
3.2.3 LOSS OF RDA
Loss of RDA measures the difference between both HetG and EmoG edge distribution of the source and target domains. It contains parts:
CRDA
LEGI
LEGT
(19)
BGI Loss is the loss of the BGI procedure. Its practical meaning is the Kullback-Leibler Divergence (KLD) between the prior and posterior distributions of the HetG edges:
Lbc =
Eu)] K LDD(BIN P.)lI Mie - [()) (20)
However, since the prior edge distribution of HetG is Binomial distribution BIN(n, which contains an infinite parameter n, LRGI is computationally intractable. Therefore, we approximate minimizing the loss as minimizing its upper bound. The closed-form solution irrelevant to n is calculated as follows:
Mu + + (1-14)) log - Mrt + /ms2 /
Min (aaGi) Min(Jue log
(21)
Ps +
Ps + p. 2.2/2 +
EGT Loss is a further alignment, which forces the informative emotion representation of domains transformed from HetG also remain consistent. It is defined as follows:
LEGT KLD,N, ZneG E, || ZBRG Eic}
(22)
where N₂ Heu and Nu. Zeeo are the Gaussian variables in Eq. 17 We abbreviate the mean and the standard deviation of these Gaussian variables as ps (mn) and og (o Therefore, EGT loss could be expressed as
Min(CreT) = Min(2 + one (NI mn)²
(23)
dle + (0. cc
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Dataset. Our experiments were conducted on DEAP (Keelstra al. (001 and DREAMER (So- leymani et al. (201T)). They are both multi-modal physiological signal datasets tailored for human emotion analysis. Both datasets recorded multiple physiological signals from subjects while watch- ing videos (DEAP contains EEG, Flectrooculogram (EOG), Electromyogram (EMG), and GSR, and DREAMER contains EEG and ECG). These videos are designed to elicit different emotional responses characterized by valence (high/low) and arousal (high/low). Subjects will evaluate the valence and arousal with a given scale (DEAP from to and DREAMER from to 5). Detailed dataset information arousal scores are divided into two categories bounded by median number (For DEAP, is 5; For DREAMER, it is 3); train and predict valence and arousal, respectively.
be found in Appendix A. We adopt general ER paradigm: valence and
Training strategy. We adopt the ceave-onesubjectt-out paradigm to divide the source and target domains. The target domain contains one subject, and the remaining subjects are the source domain. This process is repeated for each subject, and the results of all subjects are averaged as the final result. We followed the supervised DA paradigm in (Wang et al. (2022) to divide the training and testing sets. Specifically, The data of the target subject is divided into 5 folds, where one fold is the training set used as the labeled target domain, another fold is the validation set, and the remaining folds are used as the testing set. The average of all five experiments' results is the target subject's final result. The evaluation metrics we use are accuracy and Fl score (Explained the Appendix A.2
Cropping strategy. To expand the data volume, our experiments used a cropped strategy Schiirmeister et al. (2077), which crops the data of each trail into 4s non-overlapping segments. Note that our cropping is done strictly after splitting the training and testing set. Since physiological signals are time-series signals, there is correlation between neighboring segments. If they appear separately in the training and testing sets, it will cause data leakage. We make all models follow the same cropping strategy prevent abnormally high accuracy and F1 scores caused by data leakage.

---

Published as a conference paper at ICLR 2024
Table I: Performance comparison of all methods in terms of accuracy and fl score (in %) for classification tasks (the greater the better) on DEAP and DREAMER. The best, second-best, and third-best results are highlighted in 1, 2, and 3, respectively.
DEAP REG. BOG, RMG and GSR)
DREAMER (RRG and RCG)
Baseline
Valence
Aruusal
Valence
Accuracy
Score
Accuracy
F] Score
Accuracy
Ei Score
Accuracy
FL Score
DOCNN Song et all
20180
59.52 1
61.13
1387
65.97 1:1
70.8 115
64.94 2:93
63.05 11M
62.19 3:225
56.22 :: 2:44
EEGNet
Lawhern al
2016
62.28
66.67
63.33
67.62
62.02
57.56
64.71
58.76 1:32
MM-Rest STST Ma ei 2010
68.42 1:
65.95 1133
67.69 144
58 2::
66.23 =: 13:55
60.55 11:00
67.87 1:407
27066 4:4
Net Ola
68.54
61.65
ACRNN
Tao al
2020
65.15 175
64.77 11.29
64,02 6.3
05.39 1.8
56.05 9:1
59.96 1047
63.8
55,88 a:1
MIGNN
67.46 11.11
63.03 13:9
64.77 1M
67.24
66.66 à 9.51
66.24
63.35 1:430
RAINDROP
7hung
M121
66.06 1011
63.7 200
65.59 1M
64 79
65.74
62.17 m0ch
65.85 7:
62,44
HelEmmttionNi
Ja
2021
169.1 1::
57.84 1281
669.0 1:574
62.75 76 1A2
+69.57 800
68.22 001
669.37 6M
57.21 7:335
TSeeption
'75.0%
63,83
76.86
65.48
69.93
64.76
MEKT Zhong & Wm
20200
64.24 t00
61.72
63.24 2
61.9 7M
54.21 3:8
67.26 11:21
63.62 7:70
70.92 676
MMDA-VAE Wang
$70,39 1.3
167.7% M0N
970.66 2:382
64.11 1::0
669.36
66.38 4:37
P71.32
68.48 4
ITSR Peng al:
65.53 km
57.19 1:M
62.75 1:15
64.88
66.77 7M4
66.7
61.7 2:445
66,44 1.12
DCCA [|t el 10221
66.81 5:339
67.19 11.3
65.15 2:4)
61.08 1120
67,47 2:00
63.21 555
64.93 1:4.0
64.81 1:335
SST-AGCN
1023
68.25
67.32
66,94
68.89
67.84
69.93
65.78
67.88
MSADA (She et 2l:
20125
66.5 7.5
63.65 1331
67.11 6:675
61.79 ; 9:38
64.32 9:30
60.0 : 6:33
65.00 4-433
65.88 135
VBHI-GNN
73.5
771.3 9036
71.21 1:4
133
70.64
73338
Models and Hyperparameters. We compared VBH-GNN with two types of baselines. The first type is non-DA methods that focus on the inference of signal relations. i.e.. HetEmotionNee (Jia al. (202T). ) The other type is the DA method, i.e., MEKT (Zhang & Wu (2020)). By comparing these two types of baselines, we hope to demonstrate the ability of VBH-GNN to fully use the multi- spatial-temporal relationships to improve the cross-subject ER accuracy. All models are trained and tested in the same experimental environment, where all conditions are kept constant except for the hyperparameters of models. The specific environment and parameter settings of VBH-GNN can be found in Appendix A.4
4.2 COMPARISON WITH PRIOR ART
The test accuracy and F1 score over the target domain are shown in Table [ Compared to modal fusion methods, VBH-GNN significantly improves performance on both datasets. DGCNN (Song ct al. (2018) and EEGNet (Lawhern et al. (2018) use CNNs to extract the signal's 's spatial fea- tures. ACRNN (Tao et al. (2020) and TSception (Ding et al. (2022)) include temporal and spatial relational inference components to extract more abundant features and therefore get higher ac- curacy than DGCNN and EEGNet. Compared to them, MM- SSMM (Ma et al. (2019) infers the interactivity between modalities through the combination of residual network and LSTM, effec- tively utilizes the spatio-temporal relationships of multi-modal signals and therefore achieves better results. Based on MM-ResLSTM, HetEmotionNet, and SST-EmotionNet also employ modality- specific feature extractors to learn the heterogeneity of multi-modal signals, thus obtaining excellent results. MTGNN (Wu et al. (2020)) and RAINDROP (Zhang et al. (2021)) are not suitable for physiological signals such as EEG and ECG, they all achieve low accuracy. None of the above models can learn general spatio-temporal relationships and thus perform less than VBH-GNN in cross-subject scenarios.
Compared to the existing DA methods, VBH-GNN still demonstrates superior performance. MEKT extracts features in the tangent space of the signal and aligns the features through joint probability distribution optimization. JTSR (Peng et al. (002)) proposes a joint optimization method to op- timize label classification while aligning the feature distribution, and MSADA (She et al. (2023) weighs the signal while aligning the features to improve the accuracy. These methods cannot infer and utilize the spatio-temporal relationships of multi-modal signals, thus limiting their performance. In contrast, DCCA (Fei et al. (2022)) utilizes different feature extractors to capture the heterogene- ity of different modalities and infer the spatial relationships of the modalities, thus achieving better results. SST-AGCN-DA (Gu et al. (2023) further infers the spatio-temporal relations through spa- tial and temporal modules and thus achieves better accuracy than DCCA. But SST-AGCN-DA fuses all the modalities directly, as compared to MMDA-VAL VAE, which uses different VAEs for different modalities to extract the features before feature fusion, thus achieving higher accuracy. However, all the above methods do not attempt to align the spatio-temporal relationship of multi-modal signal
7

---

Published as a conference paper at ICLR 2024
Table Results of ablation experiments on both datasets. means to remove this loss and means to use minimal weight for the loss. The table shows the results when the weight is set to 0.1.
DEAP
DREAMER
Loss
Arousal
Valence
Arousal
Valence
Accuracy
FI Score
Accuracy
F1 Score
Accuracy
Fl Score
Accuracy
Fl Score
BGI Loss
41.38
42.07
42.07
41.78
40.25
42.16
42.32
41.79
BGI Loss
43.45
43.64
44.28
40.75
44.83
40.31
43.98
42.59
EGT Loss
61.37
63.47
61.54
64.3
61.21
62.87
60.76
62.46
1. EGT Loss
62.1
63.27
62.44
60.23
61.32
63.92
62.88
61.3
ALL Loss
73.5
71..3
71.21
771.5
770.44
69.66
73338
69.08
Table 3:
Modality-deficien experiments on two datasets. "_" denotes removing one modality;
"ALL" indicates that all modalities are retained. The number in () indicates the number of channels.
DEAP
DREAMER
Modality
Arousal
Valence
Modality
Arousal
Valence
Accuracy
FI Score
Accuracy
Fl Score
Accuracy
FI Score
Accuracy
FI Score
GSR
N.A.
N.A.
N.A.
N.A.
GSR (N.A.)
N.A.
N.A.
N.A.
N.A.
EOG
55.08
52.51
59.12
62.23
EOG (N.A.)
N.A.
N.A.
N.A.
N.A.
EMG (2)
56.87
60.74
52.61
50.84
EMG (N.A.)
N.A.
N.A.
N.A.
ECG (N.A.)
N.A.
N.A.
N.A.
N.A.
ECG (2)
56.35
53.55
55.21
59.06
EEG (5)
61.21
62.73
61142
62.03
EEG (BCG
56.35
53.55
55.21
59.06
EEG (32)
65.7
65.06
64.88
64.57
EEG (14)
65.51
62.08
63.59
63.31
ALL (37)
773.5
711.3
711.1
'71.55
ALL (16)
70.64
699666
73.38
69.08
features between the domains. The RDA included in VBH-GNN finds a more general cross-domain distribution by aligning Nomain-invariaat multi-modal ppatio-tempora relationships while circum- venting individual differences of signal features. As result, VBH-GNN achieves better results on the cross-subject ER task than the above models.
4.3 ABLATION STUDIES
We conducted ablation studies to evaluate the effects of two main components: BGI loss and EGT loss. Table showed the results the two datasets. We compared the effect of the following four scenarios on the model performance: 1) removing the BGI loss, 2) removing the EGT loss, 3) the BGI loss obtaining a smaller weight, and the EGT loss obtaining a smaller weight. The results from the two datasets show that the effects of BGI loss and EGT loss on model performance exhibit similar trends. When the BGI loss wholly removed or given a smaller weight, VBH-GNN only achieves an accuracy of around 40% on both valence and arousal for two datasets. This suggests that the BGI loss determines whether the model converges or not or whether the model can learn the spatio-temporal relationship distribution of modalities. For EGT loss, its effect on the model is to determine the degree of convergence. This effect is more pronounced in the DEAP dataset due to modalities' more complex spatio-temporal relationships. In summary, EGT can help the model find the cquilibrium (saddle) points between BGI and the downstream classification task, and the best accuracy is obtained when the two losses are used together.
4.4 MODALITY-DEFICISS EXPERIMENTS
We conducted modality-deficinn experiments verify the rationality of our motivation for using multi-modal signals. As shown in Table 3 "All" denotes using all modalities, and "-EEG" denotes removing EEG signals. The results of all modalities show significant improvement compared to complete signals. Individual physiological signals lack sufficient information about spatial-temporal relationships, and therefore all perform worse. Since the EEG has the most significant number of channels and contains the wealthiest spatial-temporal relationships, it is still better than all the other signals. The model showed relatively stable results on the DEAP dataset because the DREAMER dataset has fewer modalities. In summary, information than other signals, and 2) VBH-GNN can take advantage of complementarity between multi-modal signals by inferring and utilizing spatial-temporal relationships.
conclude that 1) EEG can provide more practical
8

---

Published as a conference paper at ICLR 2024
(a)RAW
(b).ftter BGI
Ac)er BGT
5.
15
00
bearce 2os
-50
Tarpet
-55
-50
-25
5.0
75
Figure 4: The relationship distribution of source and target domain samples. The left figure is the raw distribution, and the middle and right are the distributions after BGI and EGT procedures.
Positive Emotion
Tisa
Negelive Emolion
Figure 5:
Visualization of the spatio-temporal relationship of the brain inferred by VBH-GNN on DEAP, where T denotes the thh time segment.
4.5 DISTRIBUTION VISUALIZATION
We apply T-distributed stochastic neighbor embedding (t-SNE) to visualize the distribution multi- modal signals. As shown in Figure we show the raw distribution and distribution after BGI and EGT. The raw distributions of the two domains have a low degree of coupling and form a high coupling state after BGI. EGT divides the clustering centers of the two emotion categories in the source and target domains into two. Thus we demonstrate that BGI can align the source and target domains, and EGT can make specific emotion distinctions.
4.6 INTERPRETABILIT ANALYSIS
We further verified the interpretability of the relationship after the EGT to visualize the spatio- temporal relationships. As shown in Figure remove some neighboring nodes for readability), we demonstrate the spatio-temporal relationships under different emotions. The intracranial rela- tionships shown as high correlations between the frontal lobe (front part) and frontal cortex (middle front part) under positive emotions; the central sulcus (middle part) was highly correlated under negative emotions, which is consistent with the findings in (Min et al. (2022)). The rela- tionships between modalities are shown by the correlation between the heart and the central sulcus under positive emotions and the correlation between the heart and the right prefrontal cortex (right front part) under negative emotions, which is also consistent with the findings in (Lichtenstein et al. Kreibig Kreibig (2010)).
CONCLUSION
We propose the VBH-GNN for cross-subject ER using multi-modal signals. This the first time emotional knowledge transfer is achieved by aligning the spatio-temporal relationships of multi- modal signals between domains. The VBH-GNN achieves the inference and alignment of multi- modal spatial-temporal relationships in domains through RDA. Experiment results show that VBH- GNN can improve ER accuracy in cross-subject scenarios and achieve promising results. In addition, the interpretability analysis shows that the distribution of relationships inferred by VBH-GNN is in accordance with the existing physiological findings, which helps reveal the potential working mechanism of VBH-GNN.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This research is supported by the National Research Foundation (NRF) Singapore, NRF Investiga- torship URRF-NRF10-200200000
REFERENCES
Michel Cabanac. What is emotion? Behavioural processes, 60(2):69-83, 2002.
Rafael A Calvo and Sidney D D'Mello. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions affective computing, 1(1):18-37, 2010.
Junyoung Chung. Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. Advances in neural information pro- cessing systems, 28, 2015.
Mark Coulson. Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of nonverbal behavior, 28: 17-139, 2004.
Yi Ding, Neethu Robinson, Su Zhang, Qiuhao Zeng, and Cuntai Guan. Tsception: Capturing tem- poral dynamics and spatial asymmetry from eeg for emotion recognition. IEEE Transactions on Affective Computing. 2022.
Paul Ekman. Expression and the nature of emotion. Approaches emotion, 3(19):344, 1984.
Cheng Fei, Rui Li, Li-Ming Zhao, Ziyi Li, and Bao-Liang Lu. A cross voos-modality deep learning method for measuring decision confidence from eye movement signals. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp- 2342-3345, IEEE, 2022.
Rong-Fe Fei Gu, Rui Li, Wei-Long Zheng, and Bao-Liang Lu. Cross-subject decision confidence estimation from eeg signals using spectral-spatian ppatial-temporal adaptive gen with domain adaptation. In 2023 International Joint Conference on Neural Networks (LJCNN), pp. 1-8. IEEE, 2023.
Hatice Gunes, Björn Schuller, Maja Pantic, and Roddy Cowie. Emotion representation, analysis and synthesis continuous space: survey. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp. 827-834. IEEE, 2011.
Ziyu Jia, Youfang Lin, Xiyang Cai, Haobin Chen, Haijun Gou, and Jing Wang. Sst-emotionnet: Spatial-spectral-tempee
based attention 3d dense network for eeg emotion recognition. In Pro- ceedings of the 28th ACM international conference on multimedia, pp. 2909-2917, 2020.
Ziyu Jia, Youfang Lin, Jing Wang, Zhiyang Feng, Xiangheng Xie, and Caijie Chen. Hetemotionnet: two-stream heterogeneous graph recurrent neural network for multi-modal emotion recognition. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1047-1056, 2021.
Diederik Kingma and Max Welling. arXiv:1312.6116 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and loannis Patras. Deap: A database for emotion analysis; using physiological signals. IEEE transactions affective computing. 3(1):18-31 31, 2011.
Sylvia Kreibig. Autonomic nervous system activity in emotion: A review. Biological psychology, 9443)1394-421, 2010.
Yu-Ting Lan, Wei Liu, and Bao-Liang Lu. Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism. In 2020 International Joint Confer- ence on Neural Networks (IJCNN), pp. 1-6. IEEE, 2020.
Vernon J Lawhern, Amelia Solon. Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent Lance. Eegnet: a compact convolutional neural network for beg-based brain-core computer interfaces. Journal of neural engineering, 15550005 2018.
10

---

Published as a conference paper at ICLR 2024
Antje Lichtenstein, Astrid Oehme, Stefan Kupschick, and Thomas Jürgensohn. Comparing two emotion models for deriving affective states from physiological data. Affect and emotion in human-computeer interaction: From theory applications. pp. 35-50, 2008.
Chenyu Liu, Xinliang Zhou, and Yang Liu. Eened: End-to-end neural epilepsy detection based on convolutional transformer. In 2023 IEEE Conference on Artificial Intelligence (CAI), pp. 368- 371. IEEE, 2023.
Chenyu Liu, Xinliang Zhou, Yihao Wu, Ruizhi Yang, Liming Zhai, Ziyu Jia, and Yang Liu. Graph neural networks eeg-based emotion recognition: A survey. arXiv preprint arXiv:2402.01 2024.
Yuchen Liu and Ziyu Jia. Bstt: A bayesian spatial-temporal transformer for sleep staging. In The Eleventh International Conference on Learning Representations, 2022.
Jiaxin Ma, Hao Tang, Wei-Long Zheng, and Bao-Liang Lu. Emotion recognition using multimodal residual Istm network. In Proceedings the 27th ACM international conference mullimedia, pp. 176-183, 2019.
Jungwon Min, Kaoru Nashiro, Hyun Joo Yoo, Christine Cho, Padideh Nasseri, Shelby Bachman, Shai Porat, Julian F Thayer Catie Chang, Tae-Ho Lee, et al. Emotion downregulation targets inte- roceptive brain regions while emotion upregulation targets other affective brain regions. Journal of Neuroscience, 42(14))29773 2985, 2022.
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1359-1365 2020.
Christian Mühl, Brendan Allison, Anton Nijholt, and Guillaume Chanel. survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain-Compute Interfaces, (2):6 84, 2014.
Yong Peng, Honggang Liu, Wanzeng Kong, Feiping Nie, Bao-Liang Lu, and Andrzej Cichocki. Joint eeg feature transfer and semi-superee semi-supervised cross-subject emotion recognition. IEEE Transactions on Industrial Informatics, 2022.
Joana Pinto, Ana Fred, and Hugo Plácido da Silva. Biosignal- multimodal emotion recogni- tion in valence-arousal affective framework applied to immersive video visualization. In 2019 41 Annual International Conference of the IEEE Engineering Medicine and Biology Society (EMBC), pp. 3577-3583. IEEE, 2019.
Robin Tibor Schirmeister, Jost Tobias Springenberg. Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, 38(11):5391-54) 2017.
Qingshan She, Chenqi Zhang, Feng Fang, Yuliang Ma, and Yingehun Zhang. Multisource associate domain adaptation for cross-subject and cross-session eeg emotion recognition. IEEE Transac- tions on Instrumentation and Measurement, 2023.
Oscar B Sheynin. Laplace's theory of errors. Archive for history of exact sciences, 17(1):1-61, 1977.
Ben Shneiderman, Catherine Plaisant, Maxine Cohen, Steven Jacobs, Niklas Elmqvist, and Nicholas Diakopoulos. Designing the user interface: strategies for effective human-cou comannc interaction. Pearson, 2016.
Mohammad Soleymani, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. A multimodal database for affect recognition and implicit tagging. IEEE transactions on affective computing. 3(1):42-55,
2011.
Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. Eeg emotion recognition using dy- namical graph convolutional neural networks. IEEE Transactions on Affective Computing, 11(3): 532-541, 2018.
11

---

Published as a conference paper at ICLR 2024
Wei Tao, Chang Li, Rencheng Song, Juan Cheng, Yu Liu, Feng Wan, and Xun Chen. Eeg-based emotion recognition via channel- wise attention and self attention. IEEE Transactions on Affective Computing, 2020.
Zitong Wan, Rui Yang, Mengjie Huang, Nianyin Zeng, and Xiaohui Liu. A review on transfer learning in eeg signal analysis. Neurocomputing, 421:1-14, 2021.
Yixin Wang, Shuang Qiu, Dan Li, Changde Du, Bao-Liang Lu, and Huiguang He. Multi-modal domain adaptation variational autoencoder for eeg-based emotion recognition. JEEE/CAA Journal of Automatica Sinica, 99):1612-1626 2022.
Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang. Wenxiu Sun, Qiong Yan, and Weisi Lin. Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling. In Proceedings of European Conference of Computer Vision (ECCV), 2022.
Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, Jinwei Gu, and Weisi Lin. Neighbourhood representative sampling for efficient end-to-end video quality assessment. IEEE TPAMI, 2023a.
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment user generated con- tents from aesthetic and technical perspectives. In International Conference on Computer Vision (ICCV), 2023b.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang. Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-bench: A benchmark for general- purpose foundation models on low-level vision. In ICLR, 2024.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 753-763, 2020.
Zhihong Zeng, Maja Pantic, Glenn Roisman, and Thomas S Huang. A survey of affect recognition methods: audio, visual and spontaneous expressions. In Proceedings of the 9th international conference on Multimodal interfaces, pp. 126-133, 2007.
Wen Zhang and Dongrui Wu. Manifold embedded knowledge transfer for brain-computer interfaces. IEEE Transactions Neural Systems and Rehabilitation Engineering, 28(5): 2020.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Repre- sentations, 2021.
Xiaowei Zhang, Jing Pan, Jian Shen, Zia Ud Din, Junlei Li, Dawei Lu, Manxi Wu, and Bin Hu. Fus- ing of electroencephaalogrmn and eye movement with group sparse canonical correlation analysis for anxiety detection. IEEE Transactions on Affective Computing, 13(2):958 111588971,,11 2020.
Xinliang Zhou, Chenyu Liu, Liming Zhai, Ziyu Jia, Cuntai Guan, and Yang Liu. Interpretable and robust ai in eeg systems: A survey, arXiv preprint 23X1:23344.1755, 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DATASETS
DEAP is a multimodal dataset always used to analyze human affective states. There are electroen- cephalogram (EEG) and peripheral physiological signals collected from 32 participants. Each partic- ipant viewed 40 one-minute-lowe excerpts of music videos and rated them based on arousal, valence, like/dislike, dominance, and familiarity. Besides that, considering the potential correlation between the brain and facial part, frontal face videos were also collected for 22 of the 32 participants. The selection of stimuli was guided by a novel method, which involved retrieving videos based on affec- tive tags from the last.fm website, detecting vidco highlights, and employing online assessment tool for evaluation.
In experiment, 32 volunteers watched subset of 40 music videos mentioned earlier. During the session, EEG and physiological signals were recorded from each participant, and they also rated the videos based arousal, valence, like/dislike, dominance, and familiarity. Additionally, frontal face videos were recorded for a portion of the participants. The data set has 40 channels, covering 4 different modes. Channels 1 32 are used collect participant EEG signals as modality 1. Channels 33 and 34 are used to collect horizontal and vertical EOG signals as modality 2. As modality 3, channels 35 and 36 are used collect Zygomaticus EMG and Trapezius EMG signals, respectively. Channel 37 is used to collect the GSR signal as modality 4. Channels 38, 39, and 40 collected the Respiration belt signal, Plethysmograph signal, and Temperature signal, respectively, which are not used in our experiences. The data was downsampled to 128Hz.
Dreamer a multi modal database containing electroencephalogram (EEG) and electrocardiogram (ECG) signals recorded during affect elicitation via audio-visual stimuli. Signals were collected from 23 participants, who self-assessed their affective state after each stimulus regarding valence, arousal, and dominance. The experiment used wearable, portable, and wireless equipment to capture signals, aiming to allow effective computing methods to be available in everyday applications. The Emotiv EPOC wireless EEG headset and the Shimmer2 ECG sensor for ECG were used for EEG.
With film clips, participants were exposed to audio and visual stimuli to evoke emotional reactions, while EEG and ECG data were recorded. The dataset comprised 18 carefully selected film clips intended to elicit emotions. These clips consisted of cut-out scenes from various films that evoke many emotions. Each of the 18 film clips targeted one of the following nine emotions: happiness, amusement, excitement, disgust, anger, fear, sadness, calmness, and surprise. The length of the film clips was between 65 to 393s (M 199s)
With the variable duration in the film clips and the allowed time for a specific emotion to become dominant, this approach aimed to capture the most prominent emotional responses elicited by the stimuli, facilitating more focused and reliable analysis of the EEG and ECG signals.
A.2 EVALUATION METRICS
Accuracy is metric used to evaluate the performance of classification models, particularly in the context of binary or multiclass classification tasks. It measures the overall correctness of the model's predictions by calculating the proportion of correctly classified samples out of the total samples in the dataset. The definition of Accuracy is as follows: Accuracy = (Number of Correctly Classified Samples) (Total Number of Samples). In this formula, the Number of Correctly Classified Samples refers to the count of samples for which the model's predictions match the actual accurate labels. Total Number of Samples is the total count of samples in the dataset. Accuracy is typically expressed as a percentage, with values ranging from 0% (worst performance, indicating that the model cor- rectly classified none of the samples) to 100% (best performance, indicating that the model correctly classified all samples). However, it may not be the most informative metric in imbalanced datasets, where one class significantly outnumbers the others. In such cases, other metrics like Precision, Recall, and F1 Score may be more informative for assessing model performance, they focus specific aspects of classification accuracy.
F1-Score is metric used to evaluate the performance of binary classification models. It combines two metrics, Precision and Recall, with the goal of providing a balanced assessment of model's accuracy and completeness. The definition of the F1 Score is as follows: F1 Score = (Precision
13

---

Published as a conference paper at ICLR 2024
X Recall) (Precision Recall). Precision represents the proportion of correctly predicted positive samples out of all predicted positive samples. Precision is calculated as: Precision (True Positives) / (True Positives False Positives). Recall represents the proportion of correctly predicted positive samples out of all actual positive samples. Recall is calculated as: Recall = (True Positives) / (True Positives False Negatives) The F1 Score is harmonic mean of Precision and Recall, and its values range from O to I, with O indicating the worst performance and I indicating the best performance. The strength of the Fl Score lies in its ability to provide balanced assessment of a model's accuracy and capture positive samples, especially when dealing with imbalanced class distributions in the dataset. It is a commonly used performance metrie, particularly for evaluating binary classification models, as it balances Precision and Recall and offers comprehensive understanding of model's performance.
A.3 BASELINES
Our baseline consists of two classes: non-DA methods that focus on inferring signal relationships
(Song et al.] (2018); Lawhern et al. (2; Ma et al. (2019); Tao et al. (2020); Ding et al. (2022)
Wu et al. (2020); Zhang et al. (2021) ): Jia et al. (2020 202 and DA methods (Zhang & Wu 2020
Wang et al. )2022) Peng al. (2022 She et al. (2023); Fei et al. (2022): Gu et al. (2023)
DGCNN Song et al. (0118) ) is deep learning model designed for EEG emotion recognition using dynamical graph convolutional neural networks. It constructs graph representations dynamically, conducts graph convolution operations to capture localized relationships, and applies sorting-bat -base pooling to condense graph dimensions and extract pivotal features. Moreover, it acquires global features for diverse applications, including graph classification,, node classification, and link predic- tion. DGCNN exhibits flexibility and adaptability, rendering it apt for various tasks encompassing distinct types of graph structures.
EEGNet (Lawhern al. (2018)) a compact convolutional neural network for EEG-based BCIs. EEGNet is widely utilized in both BCI applications and neuroscience research. It harnesses ID convolutions to capture temporal patterns, thus facilitating real-time processing adeptly. EEGNet stands out for its efficacy in tasks such as EEG classification and segmentation, owing to its efficient temporal pattern recognition capabilities.
MM-Res (M et al. (2019)) is a sophisticated deep learning architecture crafted for process- ing data from multiple modalities. MM-Resl amalgamates the strengths of residual connec- tions and Long Short-Term Memory (LSTM) layers. By effectively leveraging these components, MM-ResLSTM adeptly captures relationships and features spanning diverse modalities. This model fosters interactions between modalities, facilitating tasks such as multimodal sentiment analysis, in- ference, and mooss-modal data fusion. By harnessing complementary information from varied data sources, MM-ResLSTM, enhances overall performance across spectrum of applications.
ACRNN Tao et al. (2020) is an attention- based convolutional recurrent neural network (ACRNN) designed to enhance emotion recognition accuracy by extracting more discriminative features from EEG signals. ACRNN employs a channel-wise attention mechanism to allocate weights to differ- ent channels dynamically. Additionally, it utilizes a CNN to capture spatial information encoded within EEG signals. To delve into temporal dynamics, ACRNN integrates extended self-attention mechanisms within recurrent neural network (RNN). This allows for the recalibration of impor- tance based on intrinsic similarities within EEG signals over time. This holistic approach enables ACRNN analyze EEG data and thoroughly enhance emotion recognition accuracy.
TSception (Ding et al. (2022) is a multi-scale convolutional neural network that can classify emo- tions from EEG with its temporal dynamics and spatial asymmetry. Comprising asymmetric spatial, dynamic temporal, and high-level fusion layers, TSception simultaneously learns discriminative rep- resentations across time and channel dimensions. The dynamic temporal layer uses multi-scale ID convolutional kernels to capture temporal and frequency representations in EEG signals. Mean- while, the asymmetric spatial layer capitalizes on the distinctive asymmetrical EEG patterns associ- ated with emotions, facilitating the acquisition of discriminative global and hemisphere representa- tions. This comprehensive architecture equips TSception with the ability to extract nuanced features from EEG data, enabling robust emotion classification.
14

---

Published as a conference paper at ICLR 2024
MTGNN (Wu al. 2020)) stands out specialized framework for multivariate time series in- ference, distinguishing itself from traditional methods by autonomously learning uni-directional relationships among variables. Its innovative design integrates mix-hop propagation layer and dilated inception layer, enabling the capture of both spatial and temporal dependencies within the data. By combining graph convolution, graph learning, and temporal convolution, MTGNN offers a comprehensive approach to modeling complex interactions in multivariate time series data, effec- tively utilized in fields like energy and traffic.
RAINDROP (Zhang et al. 2021) is a graph neural network designed for analyzing irregularly sampled and multivariate time series data, often found in healthcare, biology, and climate science. It uniquely handles time series data challenges by representing each data point as a separate sensor graph, capturing the dynamic relationships between sensors over time. RAINDROP excels in esti- mating the underlying structure of these sensor networks and uses this knowledge to make accurate predictions, outperforms existing methods significantly, even complex scenarios where sensors are omitted, proving especially effective in classifying time series data and understanding temporal patterns.
AST-EmotionNet (Jia et al. (2020)) represents groundbreaking approach to emotion recognition, (SST) paradigm within a 3D dense network architecture. Its primary strength is scamlessly integrating spatial, spectral, and temporal features within unified network framework. Additionally, it incorporates a 3D) attention mechanism to explore discrimina- tive local patterns dynamically.
leveraging a ppaatialspectral-temppee
HetEmotionNet (Jia et al. (2021)) is a pioneering two-stream heterogeneous graph neural network engineered to fuse multi-modal physiological signals for enhanced emotion recognition. It com- prises both the -tmpprall and the spatial-spectral stream, adept at integrating spatial-spectral- temporal domain features within a cohesive framework. Each stream is carefully designed, incor- porating the graph transformer network to handle heterogeneity, the graph convolutional network to capture correlations, and the gated recurrent unit for capturing domain dependency.
MEKT Zhhng & Wu (2020) offers a novel method for transferring knowledge, particularly in EEG trial analysis. Initially, MEKT aligns the covariance matrices of EEG trials within the Riemannian manifold, effectively capturing underlying geometric structures. Between the source and target domains, it extracts features within the tangent space, optimizing domain adaptation by minimizing the joint probability distribution change. MEKT ensures that the geometric structures inherent to both domains are preserved while adapting. Remarkably, MEKT is versatile, capable of managing single multiple source domains, and is computationally efficient, making it suitable for practical applications in EEG data analysis.
MMDA-VAE WWang al. (2022)) presents novel approach to learning shared cross-domail latent representations of modal data. This method employs a multi-modal variational autoencoder (MVAE) to build data from modalities into unified space. By integrating adversarial learning and cycle consistency regularization, the method effectively minimizes distribution discrepancies among different domains within the shared latent representation layer. As a result, it facilitates knowledge transfer across domains, enabling the extraction of meaningful insights from multi-modal data while preserving cammain-specffc characteristics.
JTSR (Peng et al. (2022) optimizes the shared subspace projection matrix and target labels si- multaneously. By jointly optimizing these components, the model achieves superior performance in emotion recognition tasks. Furthermore, it identifies spatial-frequeng activation patterns within critical EEG frequency bands and brain parts associated with cross-subject emotion expression. This analysis is facilitated by examining the learned shared subspace, enabling a quantitative understand- ing of the underlying patterns that contribute to emotion recognition enhancements.
MSADA (She et al. (2023)) utilizes multisource associate domain adaptation (DA) network, ac- counting for domain-specific and domain-invariant features. Initially, distinct parts are established for multiple resource domains, assuming shared low-level features among EEG datasets. Subse- quently, domain-speciff features are extracted employing the one-to-one associate DA approach. Weighted scores of specific sources are determined based distribution distance, enabling the derivation of different source classifiers with exact weighted scores. Finally, EEG emotion recogni- tion tests are conducted across different datasets.
15

---

Published as a conference paper at ICLR 2024
DCCA (Fei al. (2022)). based Deep Canonical Correlation Analysis (CDCCA), works by transforming each modality separately and coordinating the multi-modals into a hyperspace using specific canonical correlation analysis constraints. In the proposed test, eye movement signals are the sole inputs used during the test phase, while the knowledge derived from EEG signals is inte- grated during the training stage. The model processes these eye movement signals, allowing it infer human decision confidence levels, thus enabling the application of eye movement signals in scenarios traditionally reliant on more complex and costly EEG signal processing.
SST-AGCN-DA (Gu et al. )202)) is a deep learning model designed for cross-subject decision con- fidence estimation. Unlike the traditional SST-AGCN model, which is focused on specific subjects and may exhibit decreased efficiency in cross-subject scenarios, the SST-AGCN-DA is tailored for more common application scenarios involving various subjects. The model incorporates domain adaptation techniques to enhance its applicability and accuracy across different subjects. It demon- strates superior performance experimental comparisons, particularly cross-subject confidence estimation experiments conducted on EEG dataset during a text-based decision-making experi- ment.
A.4 IMPLEMENTATION DETAILS
For our experiments, utilized computational environment fortified with specific software and hardware configurations. On the software front, our setup was rooted in Python version 3.8.11, complemented by PyTorch (version 1.8.0) and NumPy (version 1.20.2). Hardware- our system was powered by an Intel (R) Xeon (R) CPU E5-2620 v4 clocked at 2.10 GHz, buttressed by substantial 256 GB of RAM, and accelerated by a GeForce Tesla P40 GPU. for the configuration of the training hyper-parameters, the details are listed in Table 4
Hyper-parameter
Value
HiddenDim
128
GraphDim
128
Sequence Dim
4 x sample rate
Node Embedding Dim
128
Edge Embedding Dim
128
Epoch
500
Time Series
32
Weight Decay
le -
Learning Rate
3e
eps (c)
1c
Dropout
0.5
Semb Conv]
0.1 x sample rate
Jemb Convl Channel
8
Table 4: The Configuration of Hyper-paramettes for Training.
16


Reviews:
[{"review_id": "Z1IaAnHLVv", "reviewer": "Reviewer_QvEM", "summary": "This paper discusses the emerging field of research on human emotion using electroencephalogram (EEG) data, with a focus on cross-subject emotion recognition (ER). The challenges in this area include the neglect of multi-modal physiological signals and the difficulty in matching signal features across different domains. To address these issues, the authors propose a novel approach called Variational Bayesian Heterogeneous Graph Neural Network (VBH-GNN) with Relationship Distribution Adaptation (RDA). This method does not align the distribution of signal features but rather focuses on the distribution of spatio-temporal relationships between features. Through extensive experiments on DEAP and Dreamer datasets, the VBH-GNN with RDA demonstrates superior performance compared to state-of-the-art methods.", "questions": "1. After constructing the emotional graph, did the author only perform a single convolution layer by multiplying an adjacency matrix with node embeddings? Did the author consider using multiple convolution layers or using the more expressive graph neural network?\n2. Due to the necessity of employing Bayesian graph inference to construct multiple graphs in the author's method, I am concerned about whether the performance benefits outweigh the increased computational burden.\n3. In Table 1, VBH-GNN achieves optimal accuracy, but the F1 scores consistently demonstrate poor performance. Does this observation imply the class imbalance issue in the predicted results of the method?\n4. The authors used leave-one-subject-out paradigm to divide the source and target domains is unrealistic. In this case, the target domain is actually a validation set, which is not the real domain adaptation setting.\n5. The heterogeneity (e.g., EEG, ECG) in this paper can be regarded as multi-variant time series data. Please clarify the difference between the heterogeneity and multi-variant time series. The experiments should also include the baseline methods for learning multi-variant time series (e.g., learning a graph structure to represent the spatio relationships in multi-variant time series).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This article offers an in-depth analysis of the current challenges within the emerging domain of human emotion recognition using electroencephalogram (EEG), with a specific emphasis on cross-subject emotion recognition (ER).\n2. Author introduces the novel VBH-GNN method, and conducts comprehensive experiments to showcase the model's competitive performance.", "weaknesses": "1. It is essential to provide a more detailed explanation of how each component contributes to addressing the proposed issue, e.g., clarify the necessity of graph neural network and the specific problem it aims to solve.\n2. The readability and visual appeal of the process diagram for RDA in Figure 2 could be improved.\n3. The experimental setting of dividing source and target domains is unrealistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfe0FYWZmY", "reviewer": "Reviewer_Pxr2", "summary": "This paper designs a Variational Bayesian Heterogeneous Graph Neural Network (VBH-GNN) with Relationship Distribution Adaptation (RDA) for cross-subject emotion recognition (ER) that does not align the distribution of signal features but rather the distribution of spatio-temporal relationships between features. Extensive experiments demonstrate the superiority of the method.", "questions": "1. What is the difference between Spatial-RDA and Temporal-RDA, and what roles do they play in this task?\n2. Are the weights of each loss function in Formula 5 the same? Are the weights of each loss function considered?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method is novel and intuitive.\n2. The experiments have clearly demonstrated the effectiveness of the proposed method.", "weaknesses": "1. The quality Figure 2 needs to be improved.\n2. Statistical results in tables will be more convincing. In addition, the optimal results in Table 1 should all be highlighted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sZZAPWWlm1", "reviewer": "Reviewer_D38r", "summary": "The paper addresses cross-subject emotion recognition using EEG data. It proposes a Variational Bayesian Heterogeneous Graph Neural Network (VBH-GNN) with Relationship Distribution Adaptation (RDA) to align spatio-temporal relationships between multi-modal physiological signals, instead of aligning individual features. The method outperforms existing approaches in experiments on two public datasets.", "questions": "- The paper assumes an infinite number of edges between nodes with probabilities $p_n​$ tending to zero in section 3.2.1. Could the authors clarify a bit more on the reasoning behind the assumption?\n- Could the authors clarify $\\epsilon$ in equation 12 in the main paper. They mention it is a \"very small hyperparameter.\" Does this mean that it is close to 0?\n- Did the authors confirm that for Table 1, the comparable methods are all using the same experimental conditions as the authors (e.g., cropped experiment, experiment setup, modalities used).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors come up with an interesting method to combat heterogeneity across patients by using multiple modalities. \n- The authors provide a rigorous proof and interpretability of their method.", "weaknesses": "- In Table 1, there is a typo. I believe DEAT should read DEAP. Additionally, Mathod should be Method. Additionally, expanding Table 1 by including which modalities were used for each baseline would be more comprehensive. \n- Expanding Table 3 by seeing the effect of all modalities used would be more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T96Ts8nvxe", "reviewer": "Reviewer_zn1U", "summary": "The authors propose a Domain adaptation method for a cross-subject emotion recognition task by aligning spatial-temporal relationships of multimodal physiological signals. The method is implemented as follows: the model contains temporal and spatial Relationship Distribution Adaptation (RDA) components, each of which represents the multimodal spatio-temporal relationships as edge distributions of a heterogeneous graph and aligns them twice.", "questions": "Is the BCI in Figure 2 trying to represent BGI?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Exploiting the correlation of multimodal physiological signals to address individual differences in cross-subject emotion recognition is a simple but often overlooked detail in the past. Multimodal data has been shown to provide more information compared to single modalities, and making full use of multimodal data can improve model performance. In other words, it is a common approach to utilise the complementary of multimodal data for feature fusion to generate better feature representations. However, VBH-GNN adopts a different perspective, i.e., exploiting the relationship between multimodal data to solve the problem of individualised differences. This is like in NLP, different languages may be completely different in pronunciation and writing, but there must be similar correlations in semantic structure. This multimodal relationship will be better represented in the field of physiology, because various physiological signals correspond to the physiological system interactions of the human body. In the final experimental section (4.6) the authors also provide a good demonstration of the hypothesis that there is cross-subject similarity in the inter-signal relationships, despite individual differences in signals.\n\nDomain alignment via heterogeneous graph edge distributions is a novel idea in the field of BCI.DA has been used relatively rarely in EEG tasks, but most of the concerns are about finding domain invariant features. These features are often signal-level features. The authors argue that domain invariant features tend to trap the model in sub-optimal solutions due to individual differences in physiological signals. Therefore VBH-GNN looks for domain-invariant distributions (although in a sense the spatio-temporal relationship distribution is also a signal feature, I think there is a difference between the feature representation and the distribution), which has not been seen in previous EEG tasks.\n\nBayesian graph inference (BGI) is a generalised method. It implements a distribution with infinite parameter n in a neural network by an ingenious method and backpropagation of this infinite parameter n in network by an upper bound. It provides a feasible method to implement infinite parameters in neural networks as well as to perform backpropagation.", "weaknesses": "The explanation of the (Emotional graph transform) EGT step lacks depth. Although the authors demonstrate in their experiment (4.5) the difference between EGT and BGI, which is able to transform an intermodal heterogeneous graph into a more emotion-specific graph, the motivation for this step is not clear enough to me, and it seems more like a step based on experimental attempts to determine what to do; in other words, the authors seem to know what has to be done and how it should be done, but are unable to explain why it allows HetG to be transformed in an emotionally weighted way. what has to be done and how it should be done, but are unable to explain why doing so allows HetG to undergo an emotionally relevant weighting transformation. I think it should be that EGT creates an Attention-like effect between the original input and the HetG weights, and in training this ATTENTION tends to notice the HetG edges that are more emotionally relevant. I think the authors should experimentally demonstrate what makes EGT work and provide a more direct explanation in the paper.\n\nLack of a flowchart of the overall model. This paper contains a large number of formulas that are difficult to read, and coupled with the lack of a flowchart of the overarching model, I had a hard time imagining what the complete model would look like, how the temporal and spatial RDA components would be linked, and how the heterogeneous edges would be generated. Although the authors used formulas to explain the steps, this piling up of formulas in the presence of a large number of formulas rather made it difficult for me to understand the framework of the model, at least for me I would have liked a clearer flowchart as a guide.\n\nThe choice of Baseline is rather narrow. Although several baselines are included, they are basically methods in the BCI area. There are many DA methods in other fields, such as Maximum Classifier Discrepancy proposed in CVPR and a series of methods derived from it. I think adding more diversity of DA methods to compare and analyse can make the results more convincing, as DA is a relatively uncommon method in the BCI domain. Because multimodal data in other area do not necessarily correlate across subject as well as physiological signals, so comparison with methods in other area can demonstrate the applicability of VBH-GNN in the field of physiological signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)