Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 04:01:05

Abstract: Multimodal emotion recognition in conversation (MERC) seeks to identify the speakers' emotions expressed in each utterance, offering significant potential across diverse fields. The challenge of MERC lies in balancing speaker modeling and context modeling, encompassing both long-distance and short-distance contexts, as well as addressing the complexity of multimodal information fusion. Recent research adopts graph-based methods to model intricate conversational relationships effectively. Nevertheless, the majority of these methods utilize a fixed fully connected structure to link all utterances, relying on convolution to interpret complex context. This approach can inherently heighten the redundancy in contextual messages and excessive graph network smoothing, particularly in the context of long-distance conversations. To address this issue, we propose a framework that dynamically adjusts hypergraph connections by variational hypergraph autoencoder (VHGAE), and employs contrastive learning to mitigate uncertainty factors during the reconstruction process. Experimental results demonstrate the effectiveness of our proposal against the state-of-the-art methods on IEMOCAP and MELD datasets. We release the code to support the reproducibility of this work (currently it is uploaded as the "complementary material" within the review system and will be made public following the completion of the review process).

Main paper: 
 introduction: 
 Emotion is one of the crucial characteristics of human behavior [19] . Experienced psychiatrists can assess emotions by observing an individual's behavior, which serves as a key indicator for understanding their inclinations and responses. As human-computer interaction (HCI) advances, the capability to discern emotions from dialogues using multimodal information is becoming increasingly significant [29, 38] . This process is commonly referred to as multimodal emotion recognition in conversation (MERC). The multimodality herein includes different modal information such as the speaker's language, tone, facial expression, body movement and so on [6, 35] . From a modeling perspective, a conversation consists of a sequence of utterances. Each utterance contains one or more modalities of information and is linked to speaker information. The target of MERC is to identify the emotion category of each utterance by analyzing the available information and contextual cues.Compared with the emotion recognition in non-dialogue scenarios [10] , MERC necessitates a specific emphasis on modelling the speakers involved in the dialogue. Also unlike the analysis of single-modal information [9] , the processing of multimodal information demands the utilization of distinct processing techniques to extract meaningful information from various modalities. Different modalities of information need to be synthesized to facilitate the comprehensive analysis of a conversation. For example, when a speaker utters the word "ok" with a tone of helplessness, solely relying on textual cues may not fully convey the speaker's emotional state. By taking into account factors such as intonation and tone assist in inferring the underlying feeling of sadness expressed by the speaker. Efficient integration and utilization of multimodal information play a crucial role in enhancing the precision of emotion recognition during conversations [31] .Current research methodologies regarding MERC can be classified into two main categories: non-graph-based method [13, 24, 25] and graph-based method [3, 9, 12, 14, 21] . Non-graph-based method typically utilizes recurrent neural networks (RNN) or long short-term memory (LSTM) to capture contextual information, while the output utterance representations are used for label classification. However, these methods encounter challenges in modeling long-range dependencies because of issues in information propagation and gradient vanishing problems [9] . Graph-based method typically uses a graph to depict a conversation, with each utterance represented as a node and the relationships between utterances shown through edge weights or connections between nodes. Contextual information is captured through graph convolutions, and the resulting node embeddings are fed into subsequent classification steps [19] .Graph-based methods can be further divided into standard graphbased and hypergraph-based methods. Standard graph-based method [9, 12, 14, 21] is a typical graph-based method, which represents textual information in utterances as nodes and captures contextual relationships by connecting nodes with various types of edges within a specific window size. For the standard graphbased methods, the pairwise connection approach fails to depict the actual physical structure of MERC accurately. Additionally, as the number of graph convolution layers rises, the training time and storage requirements increase exponentially. It can also result in oversmoothing of the graph and redundancy of nodes, potentially leading to inaccurate assessments [27] .Hypergraph-based method changes the point-to-point connection to a hyperedge connection structure that more closely fits the model [3] . Hypergraph is a special graph structure capable of capturing high-order correlations, enabling the exploration of more intricate relationships [1] . By linking multiple modalities within a single utterance and connecting all nodes of the same modality using hyperedges, the hypergraph-based method can achieve outstanding performance improvement. Nevertheless, the fixed fully connected hypergraph structure still results in information redundancy, graph smoothing and slow convergence, especially when processing long-distance conversations [37] .To address the aforementioned issues in existing hypergraphbased methods, we propose a multimodal fusion framework via hypergraph autoencoder and contrastive learning named HAUCL for MERC, which is applicable to multimodal data and capable of adaptively adjusting hypergraph connections. The framework consists of five modules: (1) unimodal encoding, (2) hypergraph construction, (3) hypergraph convolution, (4) hypergraph contrastive learning, and (5) classifier. The unimodal encoding module is designed to generate modality-independent representations. For the hypergraph construction module, it firstly forms an initial fully connected hypergraph structure. Then, a variational hypergraph autoencoder (VHGAE)-based approach is introduced to realize adaptive adjustment of the hypergraph. In this paper, we develop VHGAE to map the hypergraph to the latent space to obtain node and hyperedge by sampling from space, and then learn new connections via Gumbel-Softmax [16] . The aforementioned procedures exhibit a degree of randomness. To minimize the influence of random factors, two parameter-sharing paths are established through the utilization of contrastive learning techniques: Two VHGAEs reconstruct the hypergraph, and the reconstructed hypergraphs are utilized in the subsequent hypergraph convolution module to learn the embeddings along with contextual information. Then, point-to-point hypergraph contrastive learning module is applied to the obtained two hypergraphs, where nodes corresponding to each other in different hypergraphs are considered positive sample pairs to ensure model stability. Conversely, other nodes are treated as negative sample pairs to enhance the learning of more distinctive embeddings. Finally, the learned embeddings are fed into the classifier module for emotion category prediction.The main contributions of this paper are summarized as follows:• We propose a joint learning framework based on hypergraphs, which achieved synergistic optimization of hypergraph reconstruction, contrastive learning, and emotion recognition, leading to globally optimal performance. Specifically, VHGAE is integrated into MERC to adaptively adjust the hypergraph, while Gumbel-Softmax is devised to mitigate data overflow. • We utilize contrastive learning to mitigate the impact of uncertainty in the sampling process and the Gumbel-softmax learning process of VHGAE, enhancing the robustness and stability of the model. • Extensive experiments conducted on two mainstream MERC datasets, IEMOCAP and MELD, validate the effectiveness of our work. The results showed that our proposal performed superiorly compared to the state-of-the-art methods in accuracy and weighted F1 score. 
multimodal emotion recognition: 
 Regarding the non-graph learning methods, BC-LSTM captures contextual information from surrounding utterances in three different modalities by using three independent bidirectional LSTM networks, and the output utterance representations are used for label classification [25] . However, this method lacks the usage of speaker information and thus is not applicable to multi-person conversation scenarios. DialogueRNN utilizes three gate recurrent units (GRUs) to track the global context, speaker state, and emotion state throughout the entire dialogue, which effectively integrates speaker modeling, contextual modeling, and emotion modeling [24] .To mimic the human reasoning process, DialogueCRN introduces reasoning modules to integrate the factors that make emotions happen [13] .The standard graph-based methods, such as DialogueGCN [9] , represent textual information in utterances as nodes and capture contextual relationships through different types of edges connecting nodes within a given window size. The multimodal graph convolutional network (MMGCN) develops DialogueGCN by further incorporating audio and video modalities into the model [14] . To address the challenge of cross-modal interaction in information fusion within ERC, MIMMN introduces a multi-view network that leverages complementary information from all modalities. It dynamically balances the relationships between all modalities during the fusion process [32] . MM-DFN [12] uses a dynamic fusion mechanism to fully understand the context relationship between multiple modalities and reduce the redundancy between modalities. COG-MEN [17] utilizes graph neural network (GNN) to leverage both local and global information in a conversation. GraphMFT [21] not only designs a multimodal fusion method based on graphs but also utilizes multiple graph attention networks (GATs) to capture the intra-modal contextual details and inter-modal complementary information. M3NET [3] introduces the hypergraph into the field of MERC. Through simple fully connected structures and randomly initialized edge weights, significant improvement in prediction accuracy and time efficiency has been achieved by multiple hypergraph convolutions. A hypergraph acts as an extended version of the standard graph learning, specifically designed to extract high-order correlations within the data [1] . The examples of hypergraphs are shown on the left side of Figure 1 , with the corresponding standard graphs shown on the right side. The circular dots represent five nodes, i.e., from 𝑉 1 to 𝑉 5 . Curves with the same color form a hyperedge, and there are three hyperedges 𝑒 1 , 𝑒 2 , 𝑒 3 in total. In a hypergraph, connections are not limited to pairwise relationships as in a standard graph. Hyperedges can link multiple nodes together, and a single node can be linked by multiple hyperedges simultaneously. Meanwhile a hypergraph can include multiple types of hyperedges, representing multiple meanings. In this paper, we create a hypergraph in which all utterances linked to the same speaker are grouped together on a hyperedge, while also connecting similar modality into another hyperedge. This structure closely resembles the physical structure of certain models, capturing higher-order correlations and minimizing information loss during the modeling process. The effectiveness of hypergraph learning in solving the association problem of multimodal data has already been verified in various applications, such as including recommendation system [33] , video segmentation [34] , sleep stage classification [23] , and drug-target interaction prediction [28] .Regarding hypergraph convolutions, the learning process involves aggregating node information onto connected hyperedges with varying weights, followed by sending messages from the hyperedges back to the connected nodes. This process is not constrained by distance, thereby mitigating the limitations of message transmission during the process [7] . These benefits are particularly pronounced in long-distance transmissions [8] . Therefore, the hypergraph learning process is anticipated to be effective in the MERC task, as speakers frequently discuss topics that are distant from the current conversation, utilizing long-distance cues. 
methodology: 
 In this paper, we model the MERC task as follows: a conversation contains a sequence of utterances 𝑢 𝑖 (𝑖 = 1, ..., 𝑁 ). 𝑁 is the number of utterances. Each utterance 𝑢 𝑖 consists of textual, acoustic, and visual modality, represented as 𝑢 𝑖 = {𝑢 𝑡 𝑖 , 𝑢 𝑎 𝑖 , 𝑢 𝑣 𝑖 }, respectively. Meanwhile, each 𝑢 𝑖 is spoken by a corresponding person 𝑠 𝑖 . By integrating the speaker information, an utterance can be denoted as 𝑣 𝑖 = (𝑢 𝑖 , 𝑠 𝑖 ). The goal of a MERC task is to predict the emotion label for each utterance 𝑣 𝑖 based on the given multimodal information. The overall framework of our proposed HAUCL is illustrated in Figure 2 . It includes unimodal encoding, hypergraph construction, convolution, contrastive learning and classifier.This module involves extracting essential information from raw visual, textual, and acoustic modalities data. Following the approach outlined in M3NET [3] , features from visual modalities are extracted using DenseNet [15] or 3D-CNN [36] , depending on the adopted dataset. Features from acoustic and textual modalities are extracted using the OpenSmile toolkit [5] and the RoBERTa large model [22] respectively.As mentioned above, incorporating contextual information is crucial for emotion category prediction in conversations. To enhance discourse feature representation, we employ various encoding methods tailored to the characteristics of different modalities. Specifically, we utilize GRU network [4] to encode context information for the textual modality, while acoustic and visual information is encoded using two fully connected multilayer perceptrons (MLPs). To facilitate the information fusion across modalities, we normalize the encoded dimension to a unified 𝑑 dimension as below:𝑈 𝑎 𝑖 = 𝑊 𝑎 𝑢 𝑎 𝑖 + 𝑏 𝑎 𝑖 (1a) 𝑈 𝑣 𝑖 = 𝑊 𝑣 𝑢 𝑣 𝑖 + 𝑏 𝑣 𝑖 (1b) 𝑈 𝑡 𝑖 = 𝑊 𝑡 ← -→ 𝐺𝑅𝑈 (𝑈 𝑡 𝑖 -1 , 𝑢 𝑡 𝑖 , 𝑢 𝑡 𝑖+1 ) +𝑏 𝑡 𝑖 (1c)where 𝑢 𝑖 is the input of unimodal encoding. 𝑈 𝑖 is the output of the model with the dimension 𝑑. 𝑎, 𝑣, 𝑡 stands for visual, acoustic, and textual modalities respectively. 𝑊 and 𝑏 are trainable parameters. Speaker information is a critical factor that affects the performance of the MERC task. We firstly encode the speaker information into vectors 𝑠 𝑖 in one-hot form as:𝑆 𝑖 = 𝑊 𝑠 𝑠 𝑖 + 𝑏 𝑠 𝑖 (2)Next, we integrate them into the modality information by:EQUATIONThe output of this module is 𝑉 𝑖 , which is the feature embeddings with modality-independent context awareness and speaker information.This module is composed of three stages: structure initialization, VHGAE, and hypergraph reconstruction.We represent a conversation with continuous utterances through hypergraph G = (V, E), where each node 𝑣 ∈ 𝑉 represents a unimodal utterance and each hyperedge ℎ ∈ 𝐻 captures multimodal dependencies. Each utterance's modality is represented by a node in a hypergraph, i.e., 𝑉 𝑡 𝑖 for the textual modality, 𝑉 𝑎 𝑖 for the acoustic modality, and 𝑉 𝑣 𝑖 for the visual modality.We design two distinct types of hyperedges in this paper: the first one involves connecting every node in a modality to form a hyperedge , i.e., it includes {𝑉 𝑣 1 , 𝑉 } by joining the three modalities of an utterance. Similar to the standard graphs, the incidence matrix for hypergraphs can also be defined as H ∈ R 3𝑁 ×𝑀 , where 𝑁 and 𝑀 is the number of nodes and hyperedges, respectively. We define 𝐻 𝑖,𝑗 to determine the presence of node 𝑖 in hyperedge 𝑗 as:EQUATION3.2.2 VHGAE. The fully connected hypergraph generated in the structure initialization stage may lead to redundancy in the subsequent update process, impeding the classification of subtle differences. To mitigate this challenge, we introduce VHGAE to reconstruct the hypergraph, aiming to identify the most appropriate hypergraph structure. VHGAE comprises of three processes: encoder, sampler, and decoder. The structure of VHGAE is illustrated in Figure 3 .Encoder: It aims to project the hypergraph into a representation consisting of sets of nodes and hyperedges. This projection can facilitate the subsequent decoding process of the non-Euclidean structure, as highlighted in the original paper that proposed variational graph auto-encoders (VGAE) [20] .In our proposed method, we follow the VHGAE framework and utilize a hypergraph neural network (HyperGNN) to perform hypergraph convolution on the original hypergraph. This convolution operation produces embeddings for both the nodes 𝑣 and the hyperedges 𝜖 as:EQUATIONWe utilize the obtained embeddings to encode the mean 𝜇 and variance 𝜎 vectors for each type 𝑘 ∈ (𝑣, 𝜖). This encoding process involves applying linear transformations and activation functions, as described by the following equations:EQUATION)𝜎 𝑘 = 𝜎 1 𝑊 𝑘 2 (𝜎 (𝑊 𝑘 𝜎 (𝑘) + 𝑏 𝑘 𝜎 )) + 𝑏 𝑘 2 (6b)where 𝑊 𝑘 and 𝑏 𝑘 are learnable parameters specific to the type 𝑘. The activation functions 𝜎 and 𝜎 1 correspond to the ReLU and Softplus functions, respectively. Through encoding the mean and variance vectors with node and hyperedge embeddings, we can effectively capture and represent the crucial information regarding the hypergraph's structure within a latent space. These encoded vectors will play a pivotal role in the subsequent stages, enabling the generation of meaningful and relevant outputs.Sample: To incorporate the reparametrization trick, we utilize sampling in the latent space to obtain new nodes and hyperedge embeddings. The sampling process introduces stochasticity while ensuring differentiable computations during the training phase.To generate the new embeddings, we use the mean 𝜇 𝑘 and variance 𝜎 𝑘 vectors obtained from the encoder process by Equations 6a and 6b. The reparametrization trick involves sampling from a standard normal distribution 𝛿 ∼ 𝑁 (0, 1) and scaling it by the standard deviation 𝜎 𝑘 . The obtained sample is then added element-wise to the mean vector 𝜇 𝑘 to obtain the new embedding 𝑚 𝑘 by:EQUATIONwhere ⊙ represents the element-wise product between 𝜎 𝑘 and 𝛿. By incorporating the sampled noise 𝛿 into the latent space representation, we introduce randomness to the model, while maintaining differentiability for efficient optimization. The obtained embeddings 𝑚 𝑘 serve as the updated representations for the output nodes or hyperedges, capturing the variability and uncertainty within the hypergraph structure.Decoder: This process aims to reconstruct the hypergraph from the latent space representation. By leveraging the updated embeddings obtained from the encoder process, we can recover the connection structure of the new hypergraph through a series of operations.First, we calculate the matrix ℎ 𝑖 by taking the dot product between the transpose of 𝑚 𝜖 and 𝑚 𝜎 by:EQUATIONwhere 𝑚 𝑇 𝜎 represents the inverse of 𝑚 𝜎 . Next, we apply the Gumbel-Softmax function to the matrix ℎ with a temperature coefficient 𝜏 to introduce stochasticity:EQUATIONIn the above equation, Gumbel_Softmax is a function that applies the Gumbel-Softmax relaxation. To prevent data overflow, we incorporate the addition of a constant 𝑝 to the Gumbel-Softmax operation and subsequently apply the softmax function.After conducting the softmax operation, The obtained matrix ℎ has two columns, representing a distribution over the hypergraph connections. We extract the first column of matrix ℎ, which corresponds to the incidence matrix of the new hypergraph G 0 = (V, E 0 ).By obtaining the connection structure of the new hypergraph through the decoder, we can reconstruct the relationships and connections between nodes and hyperedges. This reconstructed hypergraph can then be further utilized for various downstream tasks.Loss function: The loss function in VHGAE consists of primary components designed to produce a reconstructed hypergraph that closely resembles the original one. Specifically, the first component measures the Kullback-Leibler (KL) divergence between the distributions of the latent variables (nodes and hyperedges) and their corresponding prior distributions. The second component quantifies the difference in connection structure between the newly generated hypergraph and the original hypergraph. The VHGAE's loss function L 𝑔 is defined as follows:EQUATIONwhere KL(𝑚 𝜎 , 𝜎) measures the KL divergence between the distribution of the sampled latent variables 𝑚 𝜎 and the prior distribution 𝜎. Similarly, KL(𝑚 𝜖 , 𝜖) represents the KL divergence between the distribution of the sampled hyperedge embeddings 𝑚 𝜖 and the prior distribution 𝜖. The third term CE(ℎ 0 , ℎ) denotes the cross-entropy loss function, quantifying the connection structure difference between the original hypergraph ℎ 0 and the generated hypergraph ℎ. This component ensures that the generated hypergraph closely matches the original hypergraph regarding the distribution of connections.By minimizing this loss function, VHGAE aims to learn an effective latent space representation that captures the essential characteristics of the hypergraph while preserving its connection structure.With the new hypergraph G 0 = (V, E 0 ), we first perform node convolution by aggregating node features to update the hyperedge embeddings. The aggregation stage facilitates the integration of information from neighboring nodes into the hyperedge representation. Following the update of the hyperedge embeddings, we proceed to the hyperedge convolution stage, where hyperedge messages are disseminated to the nodes. This operation enables the information propagation from hyperedges to their incident nodes. For each hyperedge 𝜖 ∈ E 0 , we aggregate the embeddings of its incident nodes 𝑣 according to a predefined aggregation function by:EQUATIONwhere 𝑛 𝜖 represents the updated embedding for the hyperedge 𝜖 and 𝑛 𝑣 denotes the embedding of the node 𝑣. The aggregation function Agg combines the embeddings of the incident nodes to generate the new hyperedge embedding. For each node 𝑣 ∈ V, we aggregate the messages from its incident hyperedges 𝜖 using a predefined aggregation function similar to Equation 11. By performing the node and hyperedge convolutions, we can effectively propagate information and update the embeddings in the hypergraph G 1 = (V 0 , E 0 ). This reformulated solution enables the capture of the relationships and interactions between nodes and hyperedges, facilitating a more comprehensive understanding of the hypergraph structure.In order to mitigate the instability inherent in the sampling and decoding processes, we devise a dual-path scheme within our model. The primary objective is to minimize the dissimilarity between corresponding points in two hypergraphs G (1)1 = (V (1) 0 , E 0 ) and G (2) 1 = (V (2)0 , E 0 ), which are obtained through the progression of VHGAE and convolution. Concurrently, we aim to maximize the distance between each point and other points within the embedding space.Within the context of the two hypergraph views, pairs of vertices that correspond to one another are regarded as positive pairs, whereas the remaining vertex pairs are considered negative pairs. The embedding of the 𝑖-th vertex in the two views is denoted as𝑣 (1) 𝑖 ∈ V (1) 0 and 𝑣 (2) 𝑖 ∈ V (2)0 . The contrastive loss L 𝑐𝑙 between V (1) 0 and V(2) 0 is:EQUATIONHere, V 0 denotes the set of vertices and |V 0 | signifies the cardinality of V 0 . The term 𝑓 (𝑣 (1)EQUATION𝑖 ) is calculated as:𝑓 (𝑣 (1) 𝑖 , 𝑣 (2) 𝑖 ) = -log 𝑞 (𝑣 (1) 𝑖 , 𝑣 (2) 𝑖 ) 𝑞 (𝑣 (1)𝑖 , 𝑣 Here, 𝜏 is a temperature parameter and 𝑔(, ) denotes the cosine similarity function. Considering that the function 𝑔(, ) is not symmetric, we average the positive and negative aspects. Specifically, 𝑖≠𝑗 𝑞(𝑣 𝑖 ) represents a positive pair in the other graph. By minimizing the combined loss function L 𝑐𝑙 , the similarity between corresponding points is expected to increase while enhancing the distance between each point and other points within the embedding space. This approach promotes alignment and discrimination of the embeddings, thereby yielding more stable and meaningful representations of the hypergraph structure.After acquiring contextual knowledge, we perform a fusion process on the node embeddings of the two hypergraphs G (1)1 and G (2)1 to obtain G 2 = (V 2 , E 2 ). This process aims to integrate the information from the two hypergraphs into a unified representation.Following that, we concatenate the node embeddings of the three modalities that belong to the same utterance, resulting in a comprehensive representation. Specifically, let {𝑣 𝑡 𝑖 , 𝑣 𝑎 𝑖 , 𝑣 𝑣 𝑖 } ∈ V 2 denote the node embeddings of the hypergraphs corresponding to the textual, acoustic, and visual modalities, respectively. We concatenate these embeddings to obtain a fused representation by:EQUATIONThe Concatenate function combines the embeddings of the three modalities into a unified vector, allowing for the integration of multiple sources of information. The fused representation 𝑣 𝑖 encompasses a broader range of information, enhancing subsequent analysis and prediction tasks by providing a more comprehensive input.Given the fused representation 𝑣 𝑖 for an utterance, the formulas for predicting the emotion label are as follows:EQUATION)EQUATIONIn these formulas, 𝑊 2 and 𝑊 3 are weight matrices, 𝑏 2 and 𝑏 3 are bias vectors, 𝑣 𝑖 is the processed output of 𝑣 𝑖 using the ReLU activation function, 𝑃 𝑖 is the probability distribution over the emotion labels, and 𝑦 𝑖 is the predicted emotion label. 𝜏 represents the dimension corresponding to the emotion labels. By applying these formulas, we can predict the emotion label for each utterance based on the fused representation 𝑣 𝑖 and the learned parameters 𝑊 2 , 𝑊 3 , 𝑏 2 , and 𝑏 3 .We use categorical cross-entropy loss with 𝐿2 regularization term to define the error loss between the predicted emotion category and the true label during the training process as below:EQUATIONwhere 𝑁 represents the number of dialogues in a dataset. 𝑐 (𝑠) represents the number of utterances in dialogue 𝑠. It is worth noting that each dialogue can have a different number of utterances. 𝑃 𝑖,𝑗 denotes the predicted probability distribution of emotion labels for utterance 𝑗 in dialogue 𝑖, while 𝑦 𝑖,𝑗 represents the expected class EQUATIONwhere the hyperparameter weights 𝜆 𝑔 and 𝜆 𝑐𝑙 control the importance of the generalized adversarial loss and the contrastive loss, respectively. 
experiment 4.1 datasets: 
 In this paper, we conduct experiments on two popular multimodal datasets in the field of MERC: the interactive emotional dyadic motion capture database (IEMOCAP) [2] and multimodal emotionlines dataset (MELD) [26] .• IEMOCAP: It contains videos of two-way conversations with 10 actors (5 male and 5 female). IEMOCAP records the tone and power of speech, facial expressions, torso posture, head position, gestures, transcripts, and gaze in a duo session. In this paper, we use facial expressions, the tone and power of speech and transcripts. The emotions in this dataset are artificially classified into six categories: happy, sad, neutral, angry, excited, and frustrated. We use 120 dialogues containing 5,810 utterances for training and validation, while the remaining 31 dialogues with 1623 utterances for testing. • MELD: It is a multimodal dataset for emotion recognition in multi-party conversations, containing textual, acoustic and visual modalities for ERC, selected from Friends TV series. This dataset includes seven emotions: neutral, surprise, fear, sadness, happiness, disgust, and anger. We use 1,153 dialogues with 11,098 utterances for training and validation, while the rest 280 dialogues with 2610 utterances for testing. It is worth noting that IEMOCAP dataset features a fixed set of two speakers engaging in multiple rounds of conversation, whereas MELD dataset may involve multiple speakers but with fewer utterances per conversation. Meanwhile, the emotion distribution within MELD dataset is imbalanced, with a significantly higher proportion of "neutral" emotions compared to other emotional categories, comprising nearly half of the dataset. These characteristics pose significant challenges to the model's stability.We perform all experiments on an NVIDIA GTX 1050Ti with Win11 operating system. The versions of Pytorch and cuda are 2.1.2 and 11.8, respectively. Adam optimizer is used for training. We set the batch size as 12 and the learning rate as 0.0001 on both datasets. The hyperparameter 𝜏 of Gumbel-softmax in Equation 9 is 0.1 . The number of hypergraph convolutions is 1. More details regarding the main parameters can be found in Table In order to validate the performance of the proposed method HAUCL in the MERC task, we introduce the ten state-of-the-art methods for comparison: (1) non-graph learning: LSTM [25] , Dia-logueRNN [24] , and DialogueCRN [13] ; (2) standard graph learning: DialogueGCN [9] , MMGCN [14] , DIMMN [32] , MMDFN [12] , COGMEN [17] and GraphMFT [21] ; and (3) hypergraph learning: M3NET [3] . More details regarding the baseline methods can be found in Section 2.For validation, we adopt the most mainstream evaluation metrics in this field: accuracy (Acc.) and weighted F1 score (WF1).Table 2 summaizes the performance of different methods tested on IEMOCAP and MELD datasets. The results show that our proposed HAUCL achieves superior performance in terms of the overall accuracy and weighted F1 score. In detail, compared with M3NET, which achieves the second-best performance, HAUCL enhances the accuracy and WF1 by 0.43% and 0.57% respectively on MELD dataset and by 1.29% and 1.15% on IEMOCAP dataset. The ability of HAUCL to dynamically modify the connection structure of the hypergraph helps in reducing information redundancy, particularly in IEMOCAP dataset with a high average utterance per conversation. Additionally, the use of hypergraphs helps prevent excessive smoothing, reducing the risk of excessive smoothing occurring in standard graph-based methods. Compared with non-graph learning methods, our proposed HAUCL can demonstrate significant enhancement in long-distance information transmission and multimodal information fusion, resulting in satisfactory accuracy and weighted F1 score performance.We select the following four main parameters in HAUCL for sensitivity analysis tested on MELD dataset. Figures 4a and 4b show the ratio of hypergraph reconstruction loss and contrastive learning loss to the total loss, respectively. Figure 4c represents the number of convolution layers passed by the new hypergraph obtained by reconstruction to learn the contextual information. Figure 4d shows the effect of batch size. Similar trends are also observed on IEMOCAP data.The weight of the hypergraph reconstruction loss 𝜆 𝑔 : It reflects the deviation from the original graph over the total loss (See Equation 18). Higher values of 𝜆 𝑔 indicate that the reconstructed hypergraph closely resembles the original graph. As shown in Figure 4a , when 𝜆 𝑔 is set to 0.5, our method demonstrates optimal performance in accuracy. Meanwhile, deviating from this optimal value, either towards larger or smaller values, results in a decline in the overall performance.The weight of contrastive learning loss 𝜆 𝑐𝑙 : Similar with 𝜆 𝑔 , as the value of 𝜆 𝑐𝑙 increases, the method will increasingly focus on the differences between the two hypergraphs derived from the two paths. Conversely, when the dissimilarity between the two hypergraphs diminishes, our proposal's capability to withstand interference strengthens. However, when this value is excessively large, it will impact the loss of emotion recognition, i.e., when 𝜆 𝑐𝑙 exceeds 1.1, there is a degradation in accuracy performance as plotted in Figure 4b .Hypergraph layer 𝐿: Figure 4c demonstrates that increasing the number of covolutional layers in a hypergraph does not necessarily lead to enhanced accuracy performance. A large value of 𝐿 not only amplifies the model's complexity and runtime, but also risks oversmoothing, potentially complicating the differentiation of emotions with similar characteristics. When 𝐿 is 5, there is a sharp decrease in accuracy performance, indicating an over-smoothing phenomenon.Batch size: The selection of batch size is a crucial factor that impacts the performance of recognition [11, 18] . Given the nonuniform distribution of MELD dataset, employing a batch size that is too small can render the model susceptible to the interference of small samples, leading to significant gradient fluctuations and convergence challenges. Conversely, an excessively large batch size may prompt the model to overly generalize, potentially compromising accuracy. As depicted in Figure 4d , the best performance is attained with a batch size of 12.For a more comprehensive analysis of the effectiveness of our proposed method HAUCL, we conduct ablation experiments from three different aspects: the impact of (1) speaker embedding, (2) VHGAE and contrastive learning, and (3) contrastive learning in terms of accuracy performance. The results are summarized in Table 3 .Impact of Speaker Embedding: Speaker embedding can distinguish the input features from different speakers. Existing research has shown that incorporating speaker information can enhance the accuracy of emotion recognition tasks [14] . The exclusion of speaker embedding ("w/o SE" in Table 3 ) results in a decrease in accuracy, with a degradation of 1.11% and 0.81% observed on IEMO-CAP and MELD datasets, respectively. These findings indicate that incorporating person modeling can enhance the model's performance in the MERC domain.Impact of VHGAE and Contrastive Learning: We utilize VHGAE for dynamic hyperedge selection to minimize redundancy and employ contrastive learning to mitigate random errors. In Table 3 , "w/o GCL" indicates the direct fusion of two hypergraph convolutions without the inclusion of VHGAE and contrastive learning module. The results demonstrate the effectiveness of our proposed HAUCL: It can improve the accuracy performance by 0.78% and 0.43% on IEMOCAP and MELD datasets respectively. 3 refers to the model that incorporates hypergraph and VHGAE without the integration of contrastive learning. The experimental results indicate a 0.98% enhancement in accuracy on IEMOCAP dataset and a 0.58% improvement on MELD dataset. These results verify that the contrastive learning module effectively controls the random fluctuations of VHGAE, enhancing model accuracy while reducing information redundancy.In order to demonstrate the discriminability of nodes, we present the node representations acquired through our proposed method HAUCL and the M3Net (the second-best method in Table 2 ) on MELD dataset. To visualize these representations in a more comprehensive manner, we employ t-SNE [30] method for dimensionality reduction, transforming the obtained nodes into three dimensions. Furthermore, we assign distinct colors to indicate the true labels of the nodes. By comparing the two figures in Figure 5 , it is evident that the data points depicted in Figure 5a (our method) exhibit greater separation, resulting in a more discriminative segmentation. As aforementioned, the representations derived from the proposed HAUCL exhibit reduced redundancy and enhanced discriminability, thereby enabling the attainment of superior outcomes. 
conclusion: 
 In this paper, we propose a joint learning framework based on hypergraph learning to improve the performance of MERC. This framework aims to address the issue of excessive redundancy stemming from the fully connected structure of graphs or hypergraphs. The proposed method HAUCL effectively integrates hypergraph adaptive reconstruction and contrastive learning, which reduces information redundancy and enhances accuracy. Experimental results verify the superiority of our proposed method against state-of-theart ones. In the future, we expect to integrate external knowledge, such as large language models (LLM), into our framework. By focusing on linear labels such as valence-arousal-dominance (VAD) in dimensional emotion space, we aim to substitute classification labels with the goal of enhancing machines' comprehension of human behavior.  
 
 Appendix:  


Decision: Test-Time Adaptation for Multimodal Sentiment Analysis

Reviews: review: paper_summary: paper_summary: The paper proposes VHGAE, Variational HyperGraph Autoencoder, to mitigate limitations of redundancy, excessive smoothing within existing graph networks while modelling long-distance dependencies across modalities to predict dominant emotion in Multimodal Emotion Recognition in conversation (MERC). The paper proposes a joint learning framework based on HyperGraphs coupled with Contrastive Learning to achieve improved performance towards MERC task. 

Experiments and ablation studies were conducted on the MELD and IEMOCAP datasets.
strength_weakness: strength_weakness: HyperGraph Construction, HyperGraph Convolution, HyperGraph Contrastive Learning are the key contributions of the paper. Motivation and the need to capture long-distance dependencies across modalities for tasks such as MERC is well grounded. Utilizing hypergraph structure to overcome the limitation of pairwise relationships in a standard graph structure is well-motivated and well-explained in the paper.
limitations: limitations: While the HyperGraph construction aims to capture long-distance dependencies across modalities and the details of hypergraph creation are explained in the paper, there are a few questions/limitations that could be addressed.

1. Details about the computational complexity of the proposed work have not been discussed in the paper. The time taken for the proposed HyperGraph Construction (for an utterance with N feature nodes in each modality) could help understand the feasibility of the proposed work, compared to existing cross-attention based approaches.

2. Ablation studies with details about hypergraph creation in cases of corrupted/missing information within modalities can help to understand the robustness of the proposed graph-based approach.

3. Above mentioned details regarding the performance of the utilized hypergraph convolution in corrupted/missing information scenarios within modalities can help to understand the robustness of the proposed approach.
paper_summary: paper_summary: The paper proposes a novel framework to perform Multimodal emotion recognition using an hypergraph. The idea is first to encode each modality in a traditional way, then to build an hypergraph connecting nodes within each modality, and cross-modality for each utterance. A second graph is generated through a Variational Autoencoder, in order to perform some feature selection, and then the two representations are used to predict emotions for each utterance in the dialogue. Results show a better all-round performance compared to other SOTA baselines.
strength_weakness: strength_weakness: - The introduction clearly motivates the proposed methods and claims, providing a good overview on why the authors want to test the hypergraph idea on Multimodal Emotion Recognition.

- Autoencoding the hypergraph structure could be and interesting and novel idea, and in general the methodology section is quite clear and not too difficult to follow, despite the methods used not being straightforward.

- The benchmarking datasets used are appropriate for the task, as well as the experimental setup and evaluation metrics. I appreciate the report of the F1 score for each emotion descriptor for IEMOCAP (it should have been done for MELD too). The all-round average performance is better than other graph-based baselines.
limitations: limitations: - I find the literature review a bit lacking on recent methods based on Transformers and discriminative LLMs (such as BERT), which are indeed used here as encoders. It seems from the description that non-graph methods use mostly CNN and LSTM, but this is not true anymore recently. The authors may consider reviewing the latest SOTA methods on the analyzed datasets, for instance from paper with code benchmark lists.

- I am not sure the GRU is really needed for text encoding, as RoBERTa is already capable of capturing sequential information from the text, and can be eventually finetuned.

- In section 3.2.1, it is not really clear to me the connection rationale, why connecting every node in a modality or all the modalities of an utterances, and not for instance to connect according to other dialogue information. I also wonder whether a graph built in this way is really useful, as cross-modality information or intra-modality information can be captured by other mechanisms (for instance an attention layer). This also because, later, contrastive learning is applied between the original and reconstructed graph structures, and I wonder why similar information such as similar speakers or emotion may get distanced away as not represented in the hypergraph topology.

- Regarding the analysis, it would be better to show Figure 5 in 2D, as it is not too clear to see the class separation in 3D. I also wonder, related to the previous point and the discussion in section 4.4, what is the actual achievement of the hypergraph reconstruction, as it seems that the best performance is achieved when the reconstruction loss and the contrastive learning loss weights are set in order to enforce more close reconstruction, and therefore it is not so clear why the autoencoder at first place.
paper_summary: paper_summary: This paper introduces hypergraph autoencoder and contrastive learning to MERC. Experimental results on IEMOCAP and MELD demonstrate the effectiveness of this method.
strength_weakness: strength_weakness: Advantages:
1.	This paper is different from existing graph-based works in MERC. It combines graph-based autoencoder [1] and graph-based contrastive learning [2] and applies them to MERC. 

[1] Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).

[2] You, Yuning, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. "Graph contrastive learning with augmentations." Advances in neural information processing systems 33 (2020): 5812-5823.
limitations: limitations: Disadvantages:

2.	However, this combination does not bring noticeable performance improvement. For example, in Table 2, this paper brings less 1% performance improvement for both datasets. Meanwhile, the ablation studies in Table 3 further shows that each component appears to be of little use.

3.	Please compare the training times and the model size with existing works. Maybe you can emphasize your advantages in terms of less training time and small model size rather than performance.

4.	Meanwhile, this paper draws the basic idea of graph-based autoencoder [1] and graph-based contrastive learning [2]. We would like to see some structural modifications specific to MERC rather than directly applying existing methods to MERC.

5.	There seems to be a writing error: “The second type of hypergraph creates a hyperedge {V_i^a, V_v^t, V_i^t}” => {V_i^a, V_i^v, V_i^t}

6.	Where N and M is the number of nodes and hyperedges => M=N(the number of nodes)+3 (the number of modalities)? Please specific the relationship between N and M.

[1] Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).

[2] You, Yuning, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. "Graph contrastive learning with augmentations." Advances in neural information processing systems 33 (2020): 5812-5823.