Correlation-Driven Multi-Modality Graph Decomposition for Cross-Subject Emotion Recognition

Authors: chenlong  gao
qian  chen
teng  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:50:23

Abstract: Multi-modality physiological signal-based emotion recognition has attracted increasing attention as its capacity to capture human affective states comprehensively. Due to multi-modality heterogeneity and cross-subject divergence, practical applications struggle with generalizing models across individuals. Effectively addressing both issues requires mitigating the gap between multi-modality signals while acquiring generalizable representations across subjects. However, existing approaches often handle these dual challenges separately, resulting in suboptimal generalization. This study introduces a novel framework, termed Correlation-Driven Multi-Modality Graph Decomposition (CMMGD). The proposed CMMGD initially captures adaptive cross-modal correlations to connect each unimodal graph to a multi-modality mixed graph. To simultaneously address the dual challenges, it incorporates a correlation-driven graph decomposition module that decomposes the mixed graph into concordant and discrepant subgraphs based on the correlations. The decomposed concordant subgraph encompasses consistently activated features across modalities and subjects during emotion elicitation, unveiling a generalizable subspace. Additionally, we design a Multi-Modality Graph Regularized Transformer (MGRT) backbone specifically tailored for multimodal physiological signals. The MGRT can alleviate the over-smoothing issue and mitigate over-reliance on any single modality. Extensive experiments demonstrate that CMMGD outperforms the state-of-the-art methods by 1.79% and 2.65% on DEAP and MAHNOB-HCI datasets, respectively, under the leave-one-subject-out cross-validation strategy.

Main paper: 
 introduction: 
 Multimodal physiological signals, such as electroencephalography (EEG) and peripheral physiological signals (PPS), reflect the cognitive processes of the human [11, 62] . Recognizing emotions from these modalities has attracted increasing attention for various scenarios [15, 67] since humans find it hard to conceal genuine emotions reflected by these signals [61] . Although these signals have been widely used in emotion recognition, capturing generalizable multimodal patterns across diverse subjects remains a grand challenge which hinders their practical applications in real life [29, 48] . Principal Challenges. As depicted in Figure 1 (a), the distributions of different modalities within the same individual are inconsistent, highlighting the primary aspect of intrinsic multi-modality heterogeneity. Furthermore, Figure 1 (b) demonstrates that the distributions of multi-modality signals are inconsistencies across individuals, defining the secondary aspect of cross-subject divergence.To effectively tackle these challenges, bridging the gap between multi-modality signals and establishing a generalizable representation across individuals is crucial. Unfortunately, the coupled nature of these dual challenges exacerbates the complexity of devising isolated solutions. As a result, singular approaches aimed at addressing either the heterogeneity or the divergence fail in the context of cross-subject emotion recognition utilizing multimodal physiological signals, leading to suboptimal performance.Previous studies have primarily relied upon shared representational spaces to obtain subject-independent features for crosssubject scenarios, including robust feature decomposition [10, 25, 50] , and selection [29, 66, 76] . However, these approaches necessitate specialized expertise and may not be optimal for diverse tasks. Prior or late fusion of multimodal signals through deep networks have shown progress [2, 5, 20] , but cannot wholly resolve issues of multi-modality heterogeneity and cross-subject divergence, limiting their generalizability. Moreover, they also fail to fully leverage the inherent structural information within physiological signals. Recently, transfer learning-based approaches concentrate on generalizability [21] [22] [23] [24] . Nevertheless, most methods necessitate calibration data from the target subject, often unavailable in realworld scenarios. Overall, simultaneously and effectively addressing multi-modality heterogeneity and cross-subject divergence remains an open challenge that this work aims to tackle. The Proposed Solution. This study introduces a novel unified framework, namely the Correlation-Driven Multi-Modality Graph Decomposition (CMMGD). In detail, the CMMGD initially maps physiological signals within each modality onto graphs. Subsequently, fine-grained adaptive cross-modal correlations between modalities are developed, forming a multi-modality mixed graph. A pivotal step in this framework involves the decomposition of the mixed graph into concordant and discrepant subgraphs driven by the learned correlations. The concordant subgraph contains channels activated consistently across modalities and subjects during emotion elicitation, thereby delineating a generalizable subspace. Specifically, this subspace is devised to address the primary multimodal heterogeneity while mitigating cross-subject divergence. Additionally, the discrepant subgraph conveys modality-intrinsic activations, such as muscle activity and respiration patterns. Finally, a cross-rebalance fusion mechanism is devised to fuse features from the concordant and discrepant subgraphs in a balanced manner, realizing precise emotional state prediction.Within the CMMGD framework, we design a novel backbone specifically for multimodal physiological signals, termed the Multi-Modality Graph Regularized Transformer (MGRT). More precisely, the MGRT incorporates a strategy of localized graph regularization, which is applied in parallel with global multi-modality attention. Such a concurrent methodology effectively addresses the problem of over-smoothing [26] -a known challenge arising from the sequential application of graph convolution and attention layers [4, 14, 26] , as well as issues related to small graph scales in physiological signals [18, 49] . Furthermore, integrating local and global features minimizes the risk of excessive reliance on any modality and in turn, significantly augments the generalizability. Contributions. The principal contributions are detailed as follows:(1) We propose a novel multi-modality correlation-driven graph decomposition module to learn a generalizable space that simultaneously addresses the dual challenges of multi-modality heterogeneity and cross-subject divergence.(2) We develop a novel MGRT backbone specifically for multimodal physiological signals, mitigating the over-smoothing issue and avoiding over-reliance on any single modality, thus further promoting generalizability.(3) We establish the CMMGD framework to integrate the above innovations. Comprehensive experiments on two benchmark datasets demonstrate the superiority of the CMMGD framework over the state-of-the-art methods. 
related work: 
 Physiological signal-based emotion recognition has a longstanding history [33, 41, 42] , aiming to identifying human affective states using the dimensional model [46] , which conceptualizes emotions along the dimensions of arousal and valence. The valence dimension describes whether an emotion is positive or negative, whereas arousal refers to its intensity. Multi-modality Emotion Recognition. Previous studies have synergistically integrated multi-modal data like EEG and PPS to enhance emotion recognition performance [39, 68] . Compared to unimodal approaches, these multimodal methods have shown superior performance [32, 37, 53, 74] . Existing fusion methods adopt the concatenation or attention mechanisms to combine features from different modalities [8, 55, 64] . However, these methods do not explicitly address the correlations among modalities and the variations within each modality, potentially leading to suboptimal performance [12] . To learn inter-modality relationships, correlationbased fusion [70, 71] , canonical correlation analysis [73] and graphbased models [13] have been proposed. However, their characterizations of cross-modal correlations remain coarse. The proposed CMMGD represents each modality as an unimodal graph, capturing adaptive and fine-grained cross-modal correlation to build a mixed graph. It further decomposes this graph into concordant and discrepant subgraphs, providing a more precise representation. Cross-Subject Emotion Recognition. There are two basic validation strategies: subject-dependent and subject-independent. The former is more common and has shown better performance [8, 55, 64] . The latter requires generalization across subjects, remaining a challenge [2, 5, 20] . Robust feature selection methods, including feature decomposition [10, 25, 50] and channel selection [29, 66, 76] partially mitigate cross-subject divergence but require domain expertise. Recently, transfer learning improves generalizability [21] [22] [23] [24] . However, most of them operate within domain adaptation [19] , which necessitates calibration data from the target subject. Moreover, the above methods fail to address multi-modality heterogeneity explicitly. In comparison, the proposed CMMGD framework reveals a generalizable subspace across modalities and subjects without additional target data, providing a unified solution to multimodal heterogeneity and cross-subject divergence. Graph Transformer Architecture. Representing physiological signals as graphs have gained popularity since graph structures can preserve natural spatial and functional connectivity among electrodes [13, 43, 65] . The graph-based methods serve as the foundation [51] . Among these, graph transformers are recent advancements that have shown promise in capturing dependencies within and across modalities [4, 47, 63] . Rampasek et al. [44] introduce a robust and versatile graph transformer with linear complexity.Jiang et al. [14] propose a graph transformer specifically for emotion recognition. Nevertheless, the utilization of small-scale graphs in emotion recognition [18, 49] exacerbates the challenge of oversmoothing [26] , a concern that most existing graph transformer methodologies do not explicitly address. We introduce a Multi-Modality Graph Regularized Transformer (MGRT) backbone designed to mitigate over-smoothing while enhancing generalizability. 
preliminaries: 
 Problem Formulation. The CMMGD model M CMMGD aims to predict emotional states leveraging multimodal physiological signals, specifically, EEG and PPS: Y = M CMMGD 𝑋 𝑒 , 𝑋 𝑝 . The notation Y signifies the emotional states on the valence or arousal dimension. The pair 𝑋 𝑒 , 𝑋 𝑝 alludes to a multimodal sample, where 𝑋 𝑒 ∈ R 𝑐 𝑒 ×𝑇 and 𝑋 𝑝 ∈ R 𝑐 𝑝 ×𝑇 denote the EEG and PPS data. Here, 𝑐 𝑒 and 𝑐 𝑝 represent the number of EEG and PPS channels, while 𝑇 signifies the temporal duration. To enhance clarity, the subscripts 𝑒 and 𝑝 in the following paragraphs specifically pertain to EEG and PPS, respectively. Theoretical Insights. In the domain of cross-subject emotion recognition, involving 𝐾 + 1 subjects, it requires to adopt the leave-onesubject-out (LOSO) cross-validation strategy [16] , which trains the model on 𝐾 visible subjects with set of distributions {P 𝑖 |𝑖 = 1, . . . , 𝐾 }, and validates it on the left-out Q. We aim at minimizing the error on the left-out subject 𝜀 Q (ℎ), by leveraging the distributions of visible subjects. To this end, we provide theoretical insights into the generalization error of cross-subject emotion recognition: TheoRem 3.1. Let Q and {P 𝑖 |𝑖 = 1, . . . , 𝐾 } be distributions over space X, H be a class of hypotheses corresponding to this space, and {𝜑 𝑖 } 𝐾 𝑖=1 be a collection of non-negative coefficients with 𝑖 𝜑 𝑖 = 1. Let O be a set of distributions, such that for every S ∈ O, we have:𝑖 𝜑 𝑖 𝑑 HΔH (P 𝑖 , S) ≤ max 𝑖,𝑗𝑑 HΔH (P 𝑖 , P 𝑗 ).(1)Then, for any ℎ ∈ H :EQUATIONwhere the 𝜀 Q (ℎ) is the error for a hypothesis ℎ on the left-out subject, 𝜆 𝜑 is the error of an ideal joint hypothesis which could be neglected. 𝑑 HΔH (P, Q) is H -divergence which measures the difference between two distributions. The proof of this theorem has been given in [1, 36] .The above Theorem 3.1 explores the generalization error 𝜀 Q (ℎ) of the cross-subject emotion recognition. The reduction of the second term 𝑖 𝜑 𝑖 𝜀 P 𝑖 (ℎ) can be accomplished through supervised emotional state recognition loss L emo introduced in Section 4.3. The last term 1 2 max 𝑖,𝑗 𝑑 HΔH (P 𝑖 , P 𝑗 ) entails aligning the distributions from visible subjects. Note that, we focus on multimodal signals where P is the distribution of multimodal samples. We leverage an alignment loss L aln introduced in Section 4.2 to jointly align multimodal features, thus minimizing the last term in (2) .  
the proposed cmmgd framework: 
 The proposed Correlation-Driven Multi-Modality Graph Decomposition (CMMGD) framework is designed to jointly address the dual challenges of multi-modality heterogeneity and cross-subject divergence. We subsequently detail each component in conjunction with the overall architecture in Figure 2 .Encoding Spatial and Functional Connectivity. The MGRT initially represents each modality as graphs [13, 64] , and further captures both intra-and inter-modality dependencies. Two types of connection encoding are considered to maintain prior EEG and PPS structures: (1) Spatial encoding: It is established based on the physical proximity of the electrodes within the standard 10-20 system of EEG [18] . The linear distances Ω ∈ R 𝑐 ×𝑐 between 𝑐 electrodes can be computed as Ω 𝑖,𝑗 = ∥𝜔 𝑖 -𝜔 𝑗 ∥ 2 , where 𝜔 ∈ R 𝑐 ×3 denotes the 3D coordinates of the electrodes. ( 2) FC encoding: The functional connection is constructed using the mutual information [12, 13] , which captures both linear and non-linear relationships. The functional connection matrix Φ ∈ R 𝑐 ×𝑐 is determined by Φ 𝑖,𝑗 = 𝑚∈𝑋 𝑖 𝑛∈𝑋 𝑗 log 𝑝 (𝑚,𝑛) 𝑝 (𝑚)𝑝 (𝑛) . We form the EEG adjacency matrix 𝐴 𝑒 = 𝜆 Ω Ω + (1 -𝜆 Ω )Φ, where 𝜆 Ω is a hyperparameter. For PPS, only the functional connection is utilized 𝐴 𝑝 = Φ, since the spatial encoding is not applicable. To ensure sparsity and robustness, we further reserve merely the top 30% strong connections. Embedding Temporal Dynamics. Deriving from previous study [28] , one-dimensional convolutional network 𝑓 𝑡 is incorporated to capture the temporal patterns. The temporal features are 𝐻 Incorporating recent advancements in graph transformers facilitates the comprehensive perception of both localized and global features within and across modalities. The graph transformer integrates graph convolution and self-attention layers sequentially [14, 56] . However, this sequential application is not optimal for small-scale graphs derived from physiological signals. To explain, We revisit the structure of the typical graph convolution layer [17] . A typical GCN layer is 𝐻 gcn = 𝜎 D -1 2 Ã D -1 2 𝐻𝑊 gcn , where the normalized adjacent matrix Ã = 𝐴 + 𝐼 , the degree matrix D𝑖𝑖 = 𝑗 Ã𝑖 𝑗 , and 𝜎 denotes the activation function. The matrix 𝑊 gcn ∈ R 𝑑 ×𝑑 signifies the linear transformation. For small-scale graphs, the GCN layer might oversmooth features [26] or excessively suppress local features, creating a bottleneck. The sequential application of GCN and self-attention layers further exacerbates these issues as the self-attention layer partially smooths the features.To address these issues, we devise a novel GR operation, where the ℓ-th GR operation for EEG is defined as:EQUATION)The PPS graph regularization operation is defined analogously, yielding output 𝐻(ℓ )𝑝,gr . Subsequently, GR and MMSA are simultaneously employed to derive the final output of the ℓ-th MGRT layer as: EQUATIONTo further reveal a generalizable space that can address the multimodality heterogeneity and cross-subject divergence, this section elucidates the decomposition of multimodal physiological signals into concordant and discrepant subgraphs driven by correlations. Learning Multi-modality Mixed Graph. We first construct a mixed graph to connect EEG and PPS modalities, aiming at capturing intricate adaptive cross-modal relationships. Γ ∈ R 𝑐 𝑒 ×𝑐 𝑝 ×2 denotes cross-modal correlation between the 𝑖-th EEG and the 𝑗th PPG channel, where eachΓ 𝑖 𝑗 ∼ 𝑝 Γ 𝑖 𝑗 𝐻 (𝐿 𝑑 ) 𝑒,𝑖 , 𝐻 (𝐿 𝑑 ) 𝑝,𝑗is sampled from their vertex features:Γ 𝑖 𝑗 = Softmax 𝑝 𝑖 𝑗 + 𝑔 𝜏 , where 𝑝 𝑖 𝑗 = 𝐻 (𝐿 𝑑 ) 𝑒,𝑖 𝐻 (𝐿 𝑑 ) 𝑝,𝑗 𝑊 𝑐 + 𝑏 𝑐 .(5) The 𝑊 𝑐 ∈ R 2𝑑 ×2 , 𝑏 𝑐 ∈ R 2 are learnable parameters. 𝑔 ∈ R 2 is a vector of independent and identically distributed (i.i.d.) samples drawn from a standard Gumbel distribution, and 𝜏 is the temperature parameter that governs the smoothness of Γ. The entire correlation process is differentiable [12, 38] . We adopt a curriculum learning approach by gradually annealing 𝜏 after each epoch to facilitate the convergence [69] .We assign the first dimension of Γ 𝑖 𝑗 to signify the presence of correlation between the 𝑖-th EEG channel and the 𝑗-th PPS channel, denoted as 𝑍 𝑖 𝑗 , and the cross-modal correlation matrix 𝑍 ∈ R 𝑛 𝑒 ×𝑛 𝑐 . Larger values indicate stronger cross-modal correlation.The multi-modality mixed graph G mixed amalgamates the EEG and PPS graphs based on their correlation matrix 𝑍 . G mixed consists of 𝑐 𝑒 + 𝑐 𝑝 vertices with features 𝑋 mixed = 𝐻 (𝐿 𝑑 ) 𝑒 𝐻 (𝐿 𝑑 ) 𝑝 ∈ R (𝑐 𝑒 +𝑐 𝑝 ) ×𝑑 . The adjacency matrix 𝐴 mixed ∈ R (𝑐 𝑒 +𝑐 𝑝 ) × (𝑐 𝑒 +𝑐 𝑝 ) is constructed by concatenating 𝐴 𝑒 , 𝐴 𝑝 , and 𝑍 as described above.Correlation-Driven Channel Ranking. As elucidated in Section 1, the primary challenges in cross-subject multimodal emotion recognition arise from the multi-modality heterogeneity and cross-subject divergence. We aim to address these challenges concurrently by decomposing the mixed graph into concordant and discrepant subgraphs driven by the cross-modal correlations.The decomposition commences by ranking the channels. We assess the overall significance of each channel with all other channels in the opposing modality, represented as 𝜉 𝑒,𝑖 = 𝑗 𝑍 𝑖 𝑗 and 𝜉 𝑝,𝑗 = 𝑖 𝑍 T 𝑖 𝑗 , where 𝜉 𝑒,𝑖 and 𝜉 𝑝,𝑗 denote the score of the 𝑖-th EEG and the 𝑗-th PPS channel, respectively, with 𝜉 𝑒 ∈ R 𝑐 𝑒 and 𝜉 𝑝 ∈ R 𝑐 𝑝 . The top-𝜌 channels are selected for the concordant subgraph, while the remaining channels are categorized as discrepant channels:EQUATION)where idx con denotes the index of the concordant channels, while idx dis pertains to the discrepant channels. Here, 𝜌 functions as a hyperparameter.The concordant subgraph G con and the discrepant subgraph G dis are subsequently derived from G mixed . The adjacency matrices 𝐴 con and 𝐴 dis can be obtained from 𝐴 𝑢 by extracting the rows and columns corresponding to the concordant and discrepant subgraphs. Similarly, the features 𝑋 con and 𝑋 dis can be derived from 𝑋 𝑢 .EQUATIONwhere [•, •] denotes the row and column selection by indices. The whole decomposition process is described with pictures in Figure 2 , where the outside circle represents the discrepant subgraph, while the circular area inside represents the concordant subgraph. The Divergence Loss. A divergence loss is introduced to promote the acquisition of diverse representations, as elaborated in Section 3The distance correlation (dCor) [52, 75] is utilized since it can quantify the dependencies between the concordant and discrepant subgraphs with no assumption of linearity or normality. We randomly sample 𝑛 𝑠 vertices from the concordant and discrepant subgraphs, 𝑛 𝑠 ≤ min 𝑐 𝑒 , 𝑐 𝑝 . The features of the sampled vertices are denoted as 𝑆 con , 𝑆 dis ∈ R 𝑛 𝑠 ×𝑑 respectively. The divergence loss L div is:EQUATION) V 2 𝑛 𝑠 (𝑆 con , 𝑆 dis ) is the empirical distance covariance. V 2 𝑛 𝑠 (𝑆 con , 𝑆 con ) and V 2𝑛 𝑠 (𝑆 dis , 𝑆 dis ) are the empirical variances. The Alignment Loss. Furthermore, we follow Section 3 to introduce an auxiliary alignment loss L aln aimed at aligning the distributions of the visible subjects. We represent the features of the mixed graph of the 𝑖-th sample as 𝑋 <𝑖> mixed . In the model implementation, each batch contains samples from all visible subjects. The alignment loss within a batch B can be expressed as:EQUATION)where𝜇 = 1 | B | | B | 𝑗=1 GMP 𝑋 < 𝑗 > mixed ∈ R 𝑑 .The GMP(•) ∈ R 𝑑 represents adopting the global mean pooling operation.Emotional State Prediction. Naturally, the concordant and discrepant subgraphs are not equally noteworthy since the former provides more generalizable features. A cross-rebalance mechanism is introduced to assess the importance of these two subgraphs precisely, assigning weight factors 𝛼 1 and 𝛼 2 to the two subgraphs, respectively. The final prediction of emotional states is given by:EQUATION)EQUATION)where 𝜛 (•) denotes the readout function implemented by the Equilibrium aggregation method [3] . The classifiers 𝑓 𝑐 con and 𝑓 𝑐 dis consist of two linear layers. The 𝐻 (𝐿 𝑓 ) con and 𝐻(𝐿 𝑓 )dis are the final high-level features of the concordant and discrepant subgraphs after 𝐿 𝑓 feature extractors. We adopt a weak version of MGRT as the feature extractor, where treating the multi-modality channels as a single modality reduces the model complexity, namely the GRT layers. Cross-Rebalance Mechanism. The unresolved issue pertains to the computation of the weight factors 𝛼 1 and 𝛼 2 through the crossrebalance mechanism. We balance theie significance as follows:EQUATION)where the vector 𝜓 1 ∈ R 2 evaluates the importance from the concordant perspective, and 𝜓 2 ∈ R 2 does so from the discrepant perspective. 𝑓 𝑔 con and 𝑓 𝑔 dis are two-layer linear layers. The vector 𝛼 = 1 2 (𝜓 1 + 𝜓 2 ) ∈ R 2 is the ultimate cross-rebalance weight. The weight factors 𝛼 1 and 𝛼 2 are the first and the second dimension of 𝛼. A cross-rebalance loss L cross is introduced to ensure the consistency of the significance assessments between two perspectives:EQUATIONwhere 𝐷 KL (𝑥, 𝑦) = 𝑖 𝑥 𝑖 log 𝑥 𝑖 𝑦 𝑖 is the Kullback-Leibler divergence. Model Training. The final loss function L comprises the divergence loss L div , alignment loss L aln , cross-rebalance loss L cross , and supervised emotion recognition lossL emo = - | B | 𝑖=1 Y 𝑖 log Y 𝑖 ,where Y denotes the ground truth. L is formulated as:EQUATION)where 𝜆 div , 𝜆 aln , and 𝜆 cross denote the hyperparameters.  
experimental evaluation 5.1 datasets and experimental setup: 
 Table 1 presents the statistics of the DEAP [18] and MAHNOB-HCI [49] datasets, which are two widely used benchmark multimodal physiological datasets for emotion recognition. We split each trial into 4-second segments with no overlap following [35] . This strategy increases the number of samples and forces the model to learn more robust short-time features. During all the experiments, we leverage the leave-one-subject-out cross-validation.Comparison of CMMGD. Table 2 and Table 3 present the comparison results. From an overarching perspective, the CMMGD framework attains superior performance on both datasets, surpassing the state-of-the-art methods. It excels in nearly all emotion dimensions. These outcomes substantiate the efficacy of the proposed CMMGD framework in handling multi-modality heterogeneity and cross-subject divergence. From the results, methods integrating multimodal signals (Lower part of the Table 2 and 3 ) lead to superior performance compared to those relying solely on EEG (Middle part of the Table 2 and 3 ).The highest overall performance of EEG-only methods is 64.46%, achieved by EEGFuseNet [31] . The multimodal methods demonstrate enhanced performance. Among these multimodal methods, the proposed CMMGD framework showcases the most superior overall performance of 68.59% on DEAP and 66.88% on MAHNOB-HCI, representing improvements of 1.76% and 2.65% over the secondbest method, respectively. These enhancements further emphasize the effectiveness of the CMMGD framework. Comparison of the MGRT Backbone. We proceed to evaluate the effectiveness of the proposed MGRT backbone. The comparative results are outlined in Table 4 , encompassing two latest graph transformers, namely GraphGPS [44] and EmoGTs [14] , as well as two traditional graph-only methods, GCN [17] , GraphConv [40] , and Transformer [59] . EmoGTs is the latest graph transformer network devised for multimodal emotion recognition.From Table 4 , the graph transformer-based methods outperform graph-only and transformer-only approaches. Among these architectures, our MGRT gains the highest overall performance. This result is due to the parallel design of MMSA and GR, which alleviates the over-smoothing issue and mitigates over-reliance on any single modality, thereby enhancing generalizability.Ablation Study of Dropping Discrepant Subgraph. It is acceptable to discard the discrepant subgraph Wang et al. [60] as they convey less consistent information across modalities and subjects. The results of utilizing only the concordant channels are presented in Table 5 . The performance decreases when discrepant features are discarded, proveing that both the concordant and discrepant features are essential. The concordant features alone are insufficient to capture the full dynamics of emotion. Ablation Study of Fusion Strategies. We introduce a cross-rebalance schema for integrating concordant and discrepant subgraphs. To validate the effectiveness of this fusion approach, we conduct an ablation study comparing it with three commonly used methods: summation, maximum, and concatenation. The results are presented in Table 6 , and the proposed cross-rebalance fusion mechanism exhibits superior performance by adaptive assigning weights to the concordant and discrepant subgraphs. Ablation Study of Auxiliary Losses. We proceed with an ablation study to assess the effectiveness of L div and L aln , which serve as supplementary losses aimed at enhancing cross-subject generalizability. Table 7 presents the detailed results and the findings demonstrate that omitting either L div or L aln results in a performance decline, validating the effectiveness of the proposed auxiliary losses in enhancing cross-subject generalizability.A series of sensitivity analyses are performed to assess the influence of hyperparameters, including the number of MGRT layer and GRT layer, augmentation ratio, and hidden dimension. The hyperparameters can be determined based on results in Figure 3 .We employ the t-SNE technique [58] to visualize the distribution of both temporal and decomposed features, aiming to gain insights into the feature space. In Figure 4 (a), the temporal features display a dispersed distribution, with a noticeable gap between EEG and PPS. However, within each modality, it is not possible to distinctly separate the concordant and discrepant channels. In contrast, the decomposed features, depicted in Figure 4 (b), exhibit a clearly dispersed distribution with the concordant and discrepant channels distinctly separated, highlighting that the CMMGD can learn concordant and discrepant representations.In Figure 5 , we visualize the activation of each channel in the EEG signals averaged on all samples of the left-out subject during the validation process. The red regions indicate the highly activated brain regions, while the blue regions are less activated. We mark partially consistent highly activated brain regions across subjects with circles, indicating that the proposed CMMGD framework can effectively capture robustness brain activation patterns.  
conclusions: 
 This study proposes the CMMGD framework to effectively confronts the challenges posed by multi-modality heterogeneity and cross-subject divergence in cross-subject multimodal emotion recognition and offers a unified framework that simultaneously mitigates these issues. By decomposing multimodal signals into concordant and discrepant representations, the CMMGD facilitates a comprehensive analysis of the data. A cross-rebalance fusion mechanism is introduced to adaptively fuse each subgraph in a balanced manner. Additionally, CMMGD contains a specifically devised MGRT (a) Temporal features. backbone that can capture both local and global information in multimodal physiological signals.Our work presents a promising step towards solving the critical challenge of cross-subject generalization for multimodal contents. One limitation remains the requirement of complete data for each modality. Future work will explore the potential of interpolating missing data, or dealing with noisy data, to enhance the robustness of the CMMGD framework. Moreover, we will investigate the potential of the CMMGD framework in other multimodal tasks. 
framework, termed correlation-driven multi-modality graph decomposition (cmmgd).: 
 The proposed CMMGD initially captures adaptive cross-modal correlations. It connects each unimodal graph to a multimodal mixed graph. To simultaneously address the dual challenges, it incorporates a correlation-driven graph decomposition module that decomposes the mixed graph into concordant and discrepant subgraphs based on the correlations. The decomposed concordant subgraph encompasses consistently activated features across modalities and subjects during emotion elicitation, unveiling a generalizable subspace. Additionally, we design a Multi-Modality Graph Regularized Transformer (MGRT) backbone specifically tailored for multimodal physiological signals. The MGRT can alleviate the over-smoothing issue and mitigate over-reliance on any single modality. Extensive experiments demonstrate that CMMGD outperforms the state-of-the-art methods by 1.79% and 2.65% on DEAP and MAHNOB-HCI datasets, respectively, under the leaveone-subject-out cross-validation strategy. 
 
 Appendix:  


Decision: Based on the ratings of reviewers, I suggest to accept the paper .

Reviews: review: paper_summary: paper_summary: The paper presents a novel framework, CMMGD, addressing multi-modality heterogeneity and cross-subject divergence in the context of emotion recognition from physiological signals. The authors propose a correlation-driven approach to decompose multimodal signals into concordant and discrepant subgraphs, aiming to capture consistent and variable patterns across subjects. The framework includes a specialized backbone, MGRT, designed to handle the unique challenges of small-scale physiological graph data. The CMMGD framework demonstrates improved performance over existing methods on the DEAP and MAHNOB-HCI datasets.
strength_weakness: strength_weakness: (1) The CMMGD framework innovatively addresses the dual challenge of multi-modality heterogeneity and cross-subject divergence, which are critical issues in cross-subject emotion recognition.
(2) The introduction of the MGRT backbone is a significant contribution, providing a specialized solution for handling multimodal physiological signals and mitigating over-smoothing.
(3) The paper includes a comprehensive set of experiments and ablation studies that thoroughly validate the effectiveness of the proposed framework and its components.
(4) The cross-rebalance fusion mechanism is a thoughtful addition that adaptively fuses concordant and discrepant subgraphs, potentially leading to more accurate emotion recognition.
(5) The authors provide a clear and detailed explanation of the methodology, along with a well-structured presentation of experimental results and analyses.
limitations: limitations: (1) The paper lacks a detailed discussion on the computational complexity and efficiency of the proposed framework, which is crucial given the graph-based nature of the approach.
(2) The theoretical justification for certain design choices, such as the specific architecture of the MGRT backbone or the choice of auxiliary losses, could be more rigorously explained.
(3) The paper could benefit from a more thorough comparison with state-of-the-art methods, particularly those that also aim to address cross-subject generalization in emotion recognition.
(4) The authors should provide more insight into how the framework handles missing or incomplete data, which is often a concern in real-world physiological signal datasets.
(5) The paper does not sufficiently address potential limitations or the robustness of the framework to noisy data, which is an important consideration for practical applications.
paper_summary: paper_summary: The paper introduces CMMGD as a new unified framework that initially captures adaptive cross-modal correlations between different modalities like EEG and PPS. It then decomposes these multimodal signals into concordant and discrepant subgraphs based on the learned correlations. This decomposition is intended to isolate consistently activated features across modalities and subjects during emotion elicitation, which are believed to form a generalizable subspace. Additionally, the study develops a Multi-Modality Graph Regularized Transformer (MGRT) backbone specifically tailored for multimodal physiological signals, which helps mitigate over-smoothing and avoid over-reliance on any single modality, thereby promoting generalizability.
strength_weakness: strength_weakness: 1. The proposed CMMGD framework offers a fresh perspective on addressing multimodal heterogeneity and cross-subject divergence by decomposing multimodal graphs. This approach allows for a more nuanced analysis of emotional states compared to traditional fusion methods.
2. The use of a graph transformer architecture with local and global attention mechanisms within the MGRT backbone seems well-suited to capture both local and global features from multimodal data, potentially improving model performance.
3. The introduction of an alignment loss and a divergence loss further enhances the model's ability to generalize across subjects, which is a significant advantage for real-world applications where subject variability is high.
limitations: limitations: The conceptual framework is complex, making it difficult for readers outside the field of machine learning and signal processing to grasp all the nuances involved in the methodology.
paper_summary: paper_summary: The paper presents a novel Correlation-Driven Multi-Modality Graph Decomposition (CMMGD) framework for cross-subject emotion recognition using multi-modality physiological signals. CMMGD addresses multi-modality heterogeneity and cross-subject divergence by capturing adaptive cross-modal correlations and decomposing them into concordant and discrepant subgraphs. The methodology is clearly explained with the inclusion of diagrams and pseudocode. Extensive experiments demonstrate the superiority of CMMGD.
strength_weakness: strength_weakness: 1. The overall logic of the paper is clear, and the method is well-structured. The inclusion of the main framework and pseudocode adds clarity to the explanation of the proposed method.
2. The experiments are comprehensive and include specific tests that highlight the unique contributions of the proposed method.
limitations: limitations: 1. The authors need to compare more extensively with recent works, including discussing them in the section Introduction and considering them as stronger baselines, such as [1] and [2], but we recognize that there are more relevant studies beyond these.
  - [1] Liu, Chenyu, Xinliang Zhou, Zhengri Zhu, Liming Zhai, Ziyu Jia, and Yang Liu. "VBH-GNN: Variational Bayesian Heterogeneous Graph Neural Networks for Cross-subject Emotion Recognition." In The Twelfth International Conference on Learning Representations.

  - [2] Zhong, Xiaolong, Fei Wu, Zhong Yin, and Gang Liu. "An Attention-Enhanced Retentive Broad Learning System for Subject-Generic Emotion Recognition from EEG Signals." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2310-2314. IEEE, 2024.

2. Have the authors considered the impact of individual differences on consistency and inconsistency?

3. During the decomposition of the multi-modality mixed graph, how do the authors ensure that the resulting subgraphs accurately represent concordant and discrepant features?