Multimodal Emotion Recognition Calibration in Conversations

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:03:56

Abstract: Multimodal Emotion Recognition in Conversations (MERC) aims to identify the emotions conveyed by each utterance in a conversational video. Current efforts focus on modeling speaker-sensitive context dependencies and multimodal fusion. Despite the progress, the reliability of MERC methods remains largely unexplored. Extensive empirical studies reveal that current methods suffer from unreliable predictive confidence. Specifically, in some cases, the confidence estimated by these models increases when a modality or specific contextual cues are corrupted, defining these as uncertain samples. This contradicts the foundational principle in informatics, namely, the elimination of uncertainty. Based on this, we propose a novel calibration framework CMERC to calibrate MERC models without altering the model structure. It integrates curriculum learning to guide the model in progressively learning more uncertain samples; hybrid supervised contrastive learning to refine utterance representations, by pulling uncertain samples and others apart; and confidence constraint to penalize the model on uncertain samples. Experimental results on two datasets show that the CMERC significantly enhances the reliability and generalization capabilities of various MERC models, surpassing the state-of-the-art methods.

Main paper: 
 related work: 
 Context modeling in ERC: Contextual information in conversations provides significant clues for emotion analysis [59] . According to emotion dynamics in conversations [10] , the ERC model requires modeling both context-and speaker-sensitive dependencies [57] , including recurrent-based network [21, 29, 38] , transformer-based network [24, 32, 51] , and graph-based network [11, 52, 58] . However, modeling contextual interactions among different modalities remains a significant challenge. Recent research efforts [22, 31] have explored the modeling of intra-and cross-modal interactions within a graph framework to capture contextual clues. Nevertheless, textual cues prove insufficient for understanding deep emotions [17] .Multimodal fusion: As multi-modality draws nearer to real-world application scenarios, MERC has been garnering growing research attention in recent years [53] . It integrates modalities like audio and visual cues alongside text to better grasp conveyed emotions [65] .Multimodal fusion in MERC aims to combine information from different modalities, including aggregation-based methods like concatenation [17, 60] , tensor product [35, 37] , attention networks [49, 62] . However, aggregation-based fusion methods overlook the complex interactions between modalities, resulting in insufficient utilization of contextual information [20] . Recently, researchers have explored graph-based fusion methods to capture intra-and intermodal interactive information [4, 20, 22, 44, 71] . For instance, Hu et al. [22] investigated intra-and cross-modal interactions in graph networks for contextual clue capture. Despite the progress, the reliability of these methods remains largely unexplored.Uncertainty estimation: Uncertainty estimation is crucial for reliable predictions [36] . Various models have been proposed to address uncertainty, including Bayesian neural networks [7, 26] , Dropout [40] , and Deep ensembles [16, 28] . In classification tasks, prediction confidence is essential, but models often exhibit overconfidence due to the rapid growth of softmax probabilities [19] . To mitigate this, methods have been developed to calibrate confidence scores to reflect predictive uncertainty [39] . Some approaches aim to train well-calibrated models directly [18, 30, 33, 64, 66, 69, 73] , employing techniques like mixup [56] , label smoothing [43] , and focal loss [41, 42, 74] . Others rely on post-processing methods for calibration [14, 27, 46, 48] , with temperature scaling [12] being a prominent example, adjusting probabilities using a single scalar parameter. However, these methods fail to consider the relationship between various modalities or contexts, solely adjusting overall confidence without specific calibration for individual modalities or contexts. This limits their effectiveness in the MERC task. 
methodology: 
 In this section, we offer a comprehensive introduction to each component of the CMERC framework as illustrated in Fig. 2 .Let U = [u 1 , ..., u N ] be a conversation uttered by M ≥ 2 speakers, consisting of N utterances. Each utterance u 𝑘 is represented by a triplet  and x T 𝑘 ∈ R 𝑑 𝑡 denote the acoustic, visual, and textual features of u 𝑘 , respectively. MERC aims to predict the emotion label of each utterance u 𝑘 according to its context c 𝑘 = {𝑐 X 𝑘 , 𝑐 O 𝑘 }. 𝑐 X 𝑘 and 𝑐 O 𝑘 are the intra-speaker and inter-speaker contexts of the utterance u 𝑘 .x 𝑘 = {x A 𝑘 , x V 𝑘 , x T 𝑘 }. x A 𝑘 ∈ R 𝑑 𝑎 , x V 𝑘 ∈ R 𝑑 𝑣 ,Following Ghosal et al. [9] , we employ layer normalization and average operation on the last four hidden layers of the Roberta model [34] to obtain textual features. For acoustic and visual feature extraction, following Hu et al. [20] , Wen et al. [63] , we utilize OpenSmile [8] and a pre-trained DenseNet model [23] , respectively 1 .Considering the unreliable emotional inferences in MERC models, we propose a CMERC framework to calibrate them, as shown in Fig. 2 , which integrates three key calibrations: First, it employs CL Q 7 Y J m Q Y S x a O t R S t F M l W B R I c R h c b J n 4 4 a V Q W Z j E + / o 6 F Z 2 I n c X h a c i Z J q r r z l W P I 6 b P O Z P F T q 9 b R L 1 q 1 6 3 4 N d 8 u r x / U S 1 B B u X Y T 9 x n H O E E C j h w R B G J o w h I M G T 1 H q M N H S l w H B X G K U G j j A j 1 M k D a n L E E Z j N g L + p 7 R 7 q h k Y 9 o b z 8 y q O Z 0 i 6 V W k 9 L B M m o T y F G F z m m f j u X Ud E x J V K U 6 W g E = " > A A A C 1 3 i c j V H J S g N B E H 0 Z 9 z 3 q 0 c t g I n g K E 8 H l G P T i w Y O C c S G G 0 N P p x M b Z m O k R J Q R v 4 t U f 8 K p / J P 6 B / o X V b Q d c E O 1 h Z l 6 / q v e 6 q 8 p P A p k p z 3 s p O E P D I 6 N j 4 x O T U 9 M z s 3 P F + Y W j L M 5 T L u o 8 D u L 0 x G e Z C G Q k 6 k q q Q J w k q W C h H 4 h j / 2 J H x 4 8 v R Z r J O D p U 1 4 l o h q w b y Y 7 k T B H V K i 6 U z 0 K m z j k L e n v 9 V o + L f r l V L H k V z y z 3 J 6 h a U I J d + 3 H x G W d o I w Z H j h A C E R T h A A w Z P Q 1 U 4 S E h r o k e c S k h a e I C f U y S N q c s Q R m M 2 A v 6 d m n X s G x E e + 2 Z G T W n U w J 6 U 1 K 6 W C F N T H k p Y X 2a a + K 5 c d b s b 9 4 9 4 6 n v d k 1 / 3 3 q F x C q c E / u X b p D 5 X 5 2 u R a G D L V O D p J o S w + j q u H X J T V f 0 z d 1 P V S l y S I j T u E 3 x l D A 3 y k G f X a P J T O 2 6 t 8 z E X 0 2 m Z v W e 2 9 w c b / q W N O D q 9 3 H + B E d r l e p G Z f 1 g r V T b t q M e x x K W s U r z 3 E Q N u 9 h H n b y v 8 I B H P D m n z o 1 z 6 9 x 9 p D o F q 1 n E l + X c v w P 3 M Z b J < / l a t e x i t > L ce < l a t e x i t s h a 1 _ b a s e 6 4 = " E x t P Y Z / q 3 W P a p y 6 S F F / 8 q p l t U L A = " > AA A C 1 n i c j V H L S s N A F D 2 N r 1 p f r S 7 d B K v g q q S C j 2 X R j Q s X F e w D W i m T 6 b Q N z Y t k o p R S d + L W H 3 C r n y T + g f 6 F d 8 Y U f C A 6 I c m Z c + 8 5 M / d e O 3 S d W F r W S 8 a Y m Z 2 b X 8 g u 5 p a W V 1 b X 8 o X 1 e h w k E R c 1 H r h B 1 L R Z L F z H F z X p S F c 0 w 0 g w z 3 Z F w x 6 e q H j j S k S x E / g X c h S K S 4 / 1 f a f n c C a J 6 u Q L 2 2 2 P y Q F n 7 v h s 0 h n H k + 1 O v m i V L L 3 M n 6 C c g i L S V Q 3 y z 2 i j i w A c C T w I + J C E X T D E 9 L R Q h o W Q u E u M i Y s I O T o u M E G O t A l l C c p g x A 7 p 2 6 d d K 2 V 9 2 i v P W K s 5 n e L S G 5 H S x A 5 p A s q L C K v T T B 1 P t L N i f / M e a 0 9 1 t x H 9 7 d T L I 1 Z i Q O x f u m n m f 3 W q F o k e j n Q N D t U U a k Z V x 1 O X R H d F 3 d z 8 V J U k h 5 A 4 h b s U j w h z r Z z 2 2 d S a W N e u e s t 0 / F V n K l b t e Z q b 4 E 3 d k g Z c / j 7 O n 6 C + V y o f l P b P 9 4 q V 4 3 T U W W x i C 7 s 0 z 0 N U c I o q a u R 9 j Q c 8 4 s l o G j f G r X H 3 k W p k U s 0 G v i z j / h 0 A z Z Z q < / l a t e x i t > L s < l a t e x i t s h a _ b a s e = " z o Y U U N Z g N / B O n e o K r X g M J E j V z s = " > A A A C n i c j V H L S s Q w F D W v q k x R n A d A Z L A f d u H C h O i A D k M a o w b o k V K e N O P o D b v W T x D / Q v / A m R l A H Z S J + f e c J b C G M l e + / z L g D A N j y O j U M T k P z L p z w d U m R c t H g S J l k Y L k I Z S x a S q p Q t N N M s C g I x W F w s a X j h c i y U S v r V H Q i d h b L U m Z I q r r z l W P I b O O Q v L n V L q N d s r x / U L a j A r t E f c Y x T p C A o A E g R i K c A i G n J j O E j J a D k r i M k D R x g R m S F t Q l q A M R u w F f c o d T Z m P b a M z d q T q e E G a k L B M m o T y M s L N M / E C + O s d + S + O p Z N / B R c Q q n B P l + z X Y v C K T Z M D Z J q S g j q + P W p T B d T f v l S l y C E l T u M T i m e E u V F + t k z m t z U r n v L T P z V Z G p W n N L f C m b k D r v c Z z a N T q a X V v U a l u W l H P Y Z F L G G F r m O J r a x i x Z X + E B j h y s N c + v c f a Q A a z g G / L u X H p W W g = = < / l a t e x i t > L cPredictive Distribution< l a t e x i t s h a 1 _ b a s e 6 4 = " E x t P Y Z / q 3 W P a p y 6 S F F / 8 q p l tU L A = " > A A A C 1 n i c j V H L S s N A F D 2 N r 1 p f r S 7 d B K v g q q S C j 2 X R j Q s X F e w D W i m T 6 b Q N z Y t k o p R S d + L W H 3 C r n y T + g f 6 F d 8 Y U f C A 6 I c m Z c + 8 5 M / d e O 3 S d W F r W S 8 a Y m Z 2 b X 8 g u 5 p a W V 1 b X 8 o X 1 e h w k E R c 1 H r h B 1 L R Z L F z H F z X p S F c 0 w 0 g w z 3 Z F w x 6 e q H j j S k S x E / g X c h S K S 4 / 1 f a f n c C a J 6 u Q L 2 2 2 P y Q F n 7 v h s 0 h n H k + 1 O v m i V L L 3 M n 6 C c g i L S V Q 3 y z 2 i j i w A c C T w I + J C E X T D E 9 L R Q h o W Q u E u M i Y s I O T o u M E G O t A l l C c p g x A 7 p 2 6 d d K 2 V 9 2 i v P W K s 5 n e L S G 5 H S x A 5 p A s q L C K v T T B 1 P t L N i f / M e a 0 9 1 t x H 9 7 d T L I 1 Z i Q O x f u m n m f 3 W q F o k e j n Q N D t U U a k Z V x 1 O X R H d F 3 d z 8 V J U k h 5 A 4 h b s U j w h z r Z z 2 2 d S a W N e u e s t 0 / F V n K l b t e Z q b 4 E 3 d k g Z c / j 7 O n 6 C + V y o f l P b P 9 4 q V 4 3 T U W W x i C 7 s 0 z 0 N U c I o q a u R 9 j Q c 8 4 s l o G j f G r X H 3 k W p k U s 0 G v i z j / h 0 A z Z Z q < / l a t e x i t > L s Cross-entropy Loss Predictive Distribution Gold Distribution CE < l a t e x i t s h a _ b a s e = " E C h k p a E H T L J s I d E x J V K U W g E = " > A A A C i c j V H J S g N B E H Z z q c t g I n g K E H l G P T i w Y O C c S G G N P p x M b Z m O k R J Q R v t U f K p / J P B / o X V b Q d c E O h Z l / q v e q p P A p k p z s p O E P D I N j x O T U M z s P F + Y W j L M T L u o D u L x G e Z C G Q k k q q Q J w k q W C h H h j / J H x v R Z r J O D p U l o h q w b y Y k T B H V K i U z K m z j k L e n v V o + L f r l V L H k V z y z J h a U I J d + H x G W d o I w Z H j h A C E R T h A A w Z P Q U S E h r o k e c S k h a e I C f U y S N q c s Q R m M A v d m n X s G x E e + Z G T W n U w J U K W C F N T H k p Y X a a + K c d b s b n v d k / q F x C q c E / u X b p D X u R a G D L V O D p J o S w + j q u H X J T V f z d P V S l y S I j T u E x l D A y k G f X a P J T O t z E X m Z v W e w c b / q W N O D q H + B E d r l e p G Z f g r V T b t q M e x x K W s U r z E Q N u h H n b y v I B H P D m n z o z x p D o F q n E l + X c v w P M Z b J < / l a t e x i t > L cePseudo Label 0 Pseudo Label 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " C 3 O x W 7 a 7 h W Q j o t 0 f 6 C o j r L t H X x 4 = " > A A A C 1 n i c j V H L S s Q w F D 3 W 9 7 v q 0 k 1 x R n A 1 d A Z 8 L A f d u H C h 4 O i A D k M a o x b T B 2 m q S B l 3 4 t Y f c K u f J P 6 B / o U 3 s Y I 6 i K a 0 P T n 3 n p P c e 4 N U h p n 2 / Z c B Z 3 B o e G R 0 b H x i c m p 6 Z t a d m z / I k l x x 0 e K J T F Q 7 Y J m Q Y S x a O t R S t F M l W B R I c R h c b J n 4 4 a V Q W Z j E + / o 6 F Z 2 I n c X h a c i Z J q r r z l W P I 6 b P O Z P F T q 9 b R L 1 q 1 6 3 4 N d 8 u r x / U S 1 B B u X Y T 9 x n H O E E C j h w R B G J o w h I M G T 1 H q M N H S l w H B X G K U G j j A j 1 M k D a n L E E Z j N g L + p 7 R 7 q h k Y 9 o b z 8 y q O Z 0 i 6 V W k 9 L B M m o T y F G F z m m f j u X U 2 7 G / e h f U 0 d 7 u m f 1 B 6 R c R q n B P 7 l + 4 z 8 7 8 6 U 4 v G K T Z s D S H V l F r G V M d L l 9 x 2 x d z c + 1 K V J oY U U N Z g N / B O n e o K r X g M J E j V z 0 s = " > A A A C 1 n i c j V H L S s Q w F D 3 W 9 7 v q 0 k 1 x R n A 1 d A Z 8 L A f d u H C h 4 O i A D k M a o w b 7 o k 0 V K e N O 3 P o D b v W T x D / Q v / A m R l A H 0 Z S 2 J + f e c 5 J 7 b 5 C G M l e + / z L g D A 4 N j 4 y O j U 9 M T k 3 P z L p z 8 w d 5 U m R c t H g S J l k 7 Y L k I Z S x a S q p Q t N N M s C g I x W F w s a X j h 5 c i y 2 U S 7 6 v r V H Q i d h b L U 8 m Z I q r r z l W P I 6 b O O Q v L n V 6 3 5 L 1 q 1 6 3 4 N d 8 s r x / U L a j A r t 3 E f c Y x T p C A o 0 A E g R i K c A i G n J 4 j 1 O E j J a 6 D k r i M k D R x g R 4 m S F t Q l q A M R u w F f c 9 o d 2 T Z m P b a M z d q T q e E 9 G a k 9 L B M m o T y M s L 6 N M / E C + O s 2 d + 8 S + O p 7 3 Z N / 8 B 6 R c Q q n B P 7 l + 4 z 8 7 8 6 X Y v C K T Z M D Z J q S g 2 j q + P W p T B d 0 T f 3 v l S l y C E l T u M T i m e E u V F + 9 t k z m t z U r n v L T P z V Z G p W 7 7 n N L f C m b 0 k D r v 8 c Z z 8 4 a N T q a 7 X V v U a l u W l H P Y Z F L G G F 5 r m O J r a x i x Z 5 X + E B j 3 h y 2 s 6 N c + v c f a Q 6 A 1 a z g G / L u XTo design the curriculum for MERC models, we gauge the difficulty of each conversation across different modalities and contexts by measuring confidence levels after the removal operation. Intuitively, the MERC model faces challenges in making decisions based on reliable predictive confidence for uncertain samples, thereby hindering the learning process. Greater improvement on uncertain samples within a conversation, the more difficult it becomes, because the model's confidence becomes increasingly unreliable. As the number of uncertain samples in a conversation increases, the model's predictive confidence becomes less reliable, making it more challenging to grasp the emotion of the utterances. Subsequent experiments have also demonstrated that this difficulty may manifest in the increased entropy of the model's predicted distributions.EQUATIONEQUATIONwhere CF is the number of speakers take part in d 𝑖 and it acts as a smoothing factor. We utilize baby step training scheduler [55] to arrange conversations and organize the training process, described as 𝐿𝑖𝑛𝑒𝑠 1 -4 in Algorithm 1.Understanding the factors contributing to uncertain samples is pivotal for bolstering the reliability of MERC models during training. Consequently, we advocate for the adoption of a Hybrid SCL framework, which seamlessly integrates modality-and context-specific SCL components. This approach serves to discern uncertain samples caused by the removal of modalities or contextual cues from their counterparts, thereby capturing their nuanced correlations and distinctions. As a result, the MERC model can adeptly identify the underlying factors driving uncertainty during the training process, thereby enhancing its overall effectiveness.EQUATIONEQUATIONEQUATIONwhere H denotes the hidden representation of the model M. z indicates the set of pseudo labels, generated according to the process described in 𝐿𝑖𝑛𝑒𝑠 14 -17 of Algorithm 1. ℓ (★, ★) = 𝑒 𝑠𝑖𝑚𝑖 (★,★)/𝜏 , where 𝜏 is the temperature parameter. 𝑠𝑖𝑚𝑖 (★, ★) denotes the cosine similarity function. The calculation process of modality-and context-specific SCL, taking audio and intra-speaker context as an example, is detailed in 𝐿𝑖𝑛𝑒𝑠 18 -27 of Algorithm 1. o 𝑘 , H 𝑘 ← M x {A,V,T} 𝑘 , c {X,O} 𝑘 12 o A 𝑘 ← M x { [𝑚𝑎𝑠𝑘 ],V,T} 𝑘 , c {X,O} 𝑘 13 o X 𝑘 ← M x {A,V,T} 𝑘 , c { [𝑚𝑎𝑠𝑘 ],O} 𝑘 14⊲ Pseudo labeling for the utterance u 𝑘 . 2715 for 𝜉 = {A, X} do 16 if o 𝑘 [𝜑 ] < o 𝜉 𝑘 [𝜑 ] then 17 z 𝜉 𝑘 ← 1; Γ 𝑠 += o 𝜉 𝑘 [𝜑 ] -o 𝑘 [𝜑 ] 18 Γ A , Γ X ← 0, 0; ℓ A (+) , ℓ A (-) , ℓ X (+) , ℓ X (-) ← [ ], [ ], [ ], [ ]L 𝑚 , L 𝑐 ← -Γ A /B, -Γ X /BTo enhance the reliability of predicted confidences in MERC models, we utilize the difference in confidence increase after removal operations as the regularization constraint for mini-batch d 𝑖 .EQUATIONWe jointly train our proposed framework by minimizing the sum of the following four losses.EQUATIONwhere 𝛾 𝑚 , 𝛾 𝑐 , and 𝛾 𝑠 are tuned hyperparameters. L 𝑐𝑒 denotes the loss function for the MERC task, typically implemented as a crossentropy loss. Θ is the set of trainable parameters within the CMERC. 𝜆 is the coefficient of L 2 -regularization. 
experiments 4.1 datasets: 
 We evaluate the CMERC on two datasets: IEMOCAP [3] has dyadic conversation videos with ten speakers, featuring 7,433 utterances and 151 dialogues. Each utterance has one of six emotions. MELD [47] contains multiparty conversations collected from the 'Friends' TV series, having 1,433 conversations, 13,708 utterances, and 304 speakers. Each utterance holds one of seven emotions. Following Ghosal et al. [9] , the data splitting for datasets is detailed in Table 1 . As the IEMOCAP dataset lacks a predefined train/validation split, we allocate 10% of the training dialogues for validation.All re-implementation methods have released their source codes, ensuring identical settings as the original papers. For the CMERC, 𝛾 𝑚 , 𝛾 𝑐 , 𝛾 𝑠 , and 𝜏 are manually tuned for each dataset using hold-out validation2 . We adopt M3Net [4] as the Baseline in this paper. The reported results are the average score of 5 random runs on the test set. Our experiments are conducted on a single RTX 4090 GPU.Evaluation metric: Following Zhang and Li [71] , we utilize the weighted F1 score (w-F1) as evaluation metrics and we also report F1 scores per class. To evaluate model prediction reliability, aside from various typically used confidence estimation metrics such as Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Root Mean Square Calibration Error (RMSCE) [13] , Area Under the Receiver Operating Characteristic Curve (AUROC), and Area Under the Precision-Recall Curve (AUPRC) [75] , we suggest a novel metric called Confidence Enhancement Level (CEL) to measure the degree to which predictive confidence improves for test samples when certain modalities or contexts are removed. A lower CEL denotes more reliable predictions. Except for CEL, all metrics are represented in percentages (%).EQUATIONAggregation-based fusion: Concatenation: DialogueRNN [38] and DialogueGCN [10] ; Attention networks: CTNet [70] and SCMM [67] .Graph-based fusion: MMDFN [20] , MMGCN [22] , M3Net [4] , CMCF-SRNet [71] , and CORECT [44] . Furthermore, we also consider other model-agnostic confidence calibration methods suitable for multi-modal scenarios: T-Scale [12] , Ensemble [30] , CRL [41] , FMFP [74] , and CML [36] . These can be seamlessly integrated into the MERC task for a fair comparison.In our ablation study, we present variations of our proposed CMERC: "w/o CL" denotes without CL for MERC; "w/o HSCL" denotes without Hybrid SCL (HSCL); "w/o CC" denotes without Confidence Constraint (CC).In Table 2 , a significant proportion of uncertain samples is observed in various MERC models, exceeding 90% in some cases, especially when modalities and contexts are removed. Additionally, the impact of each modality or context cannot be underestimated. Interestingly, more advanced models like M3Net show an even higher proportion, despite enhancements in performance, highlighting the formidable difficulty in reliable confidence prediction for MERC models.Further reliability analysis: We investigate the reasons behind the generation of uncertain samples, focusing on the Baseline as a representative case. In Fig. 3 , we visualize the contribution of each modality and context to uncertain samples, as measured by MM-SHAP [45] . This reveals that on the one hand, a significant portion of uncertain samples arises from the model overly prioritizing the textual modality. On the other hand, uncertain samples can also be attributed to the model struggling to appropriately weigh the influence of disparate contexts, treating them indiscriminately. Because if the removed information is noise, the increase in predictive confidence is not a bad thing. In fact, the likelihood of increased predictive confidence resulting from noise removal is extremely low for the following reasons: (1) The granularity of removed information is substantial, operating at certain modalities or specific contextual cues rather than at the feature level. The datasets used are of high quality, with almost no instances of pure noise in certain modalities or contexts. (2) The determination of whether removed information is noise depends on the model. As models improve in understanding and denoising, such cases are expected to decrease, particularly as we explore advanced MERC models.(3) Assuming removed information is noise is probable, thus removing it with extremely low probability ideally shouldn't decrease model performance. However, the Baseline in Table 3 shows performance degradation even with a missing information rate as low as 0.001. This highlights that this assumption doesn't hold under such coarsegrained information removal operations.Table 4 compares our method with others3 , showing its superior W-F1 score and establishing a new state-of-the-art benchmark. Specifically, W-F1 scores rose by 2.37% and 1.48% for IEMOCAP and MELD, respectively. CEL decreased by 629.04 and 194.53 for IEMOCAP and MELD. Table 5 further demonstrates our method's superiority over other confidence calibration methods across various metrics. Importantly, we observed a performance improvement in the advanced MERC model. However, its unreliable confidence estimation has led to deteriorating effects, as evident from the increasing CEL and echoed by other metrics in Table 6 . Addressing this issue could further enhance the model's performance, as supported by Table 6 . This demonstrates that the CMERC enhances both the reliability and the generalization of various MERC models.In this section, we analyze the impact of various components within the CMERC. Ablation experiments in Table 4 show significant improvements across all components. Statistical analysis further confirms this, with a p-value ≪ 0.05 for the paired t-test.The effectiveness of CL is evident in guiding model learning [61] , potentially enhancing performance on uncertain samples, as confirmed by ablation results in Table 4 . However, the precondition for CL efficacy is an entropy-increasing system [1, 68] . To validate this, we compute the entropy of predicted distributions for uncertain samples and others. In with HSCL. The latter exhibits significantly clearer distinctions compared to the former, with silhouette coefficients [50] of 0.048 and 0.098, respectively. This suggests that capturing latent features contributing to uncertain samples aids in enhancing utterance representations. In Fig. 6 , we also visualize the intermediate representations of the Baseline with modality-specific or context-specific SCL. Their silhouette coefficients of 0.063 and 0.067 surpass those derived from the Baseline, highlighting the efficacy of HSCL and the complementary nature of its constituent components. Analysis of CC: Table 4 shows that CC significantly reduces CEL in the MELD dataset, but it falls short in lengthy conversations like IEMOCAP, possibly due to inefficiencies in the later conversation stages, as supported by Fig. 9 . In Fig. 7 , considering modalities and contexts in CC leads to CF value (Formula 4) distributions where smaller values correspond to higher density. This underscores CC's effectiveness on uncertain samples and improving model reliability across modalities and contexts in the IEMOCAP testing set. In Fig. 8 , we demonstrate improved performance on the IEMO-CAP validation set through the adjustment of hyperparameters, including 𝛾 𝑠 , 𝛾 𝑚 , and 𝛾 𝑐 . These hyperparameters exhibit an initial increase, followed by a decrease and eventual stabilization with minor fluctuations. Importantly, performance consistently surpasses the case where these parameters are set to zero, showcasing the CMERC's effectiveness across different hyperparameter settings.Our CMERC framework differs from the global calibration of predicted confidences, as it explicitly calibrates the MERC model across various modalities and contexts. In Table 7 , we demonstrate the enhanced performance of the CMERC, highlighting substantial advancements in generalization and reliability through individual pattern calibration. This enhancement is particularly noteworthy within the acoustic modality and intra-speaker context. The integration of various modalities yields complementary effects, with consistent findings observed across different contexts.In this section, we delve into the complementarity among three pivotal calibrations to elucidate the rationale of the CMERC framework. In Fig. 9 , CC performs well in mid-conversation, particularly in the 'Excited' emotion and utterances with ES types, where two consecutive utterances exhibit different emotions. CL demonstrates strength in 'Happy' and 'Angry' emotions, especially in utterances without ES types. HSCL exhibits proficiency in handling 'Neutral' emotions. Collectively, each module showcases strengths in different aspects, underscoring their complementary performance, as also evidenced by Table 4 .Many errors in our method stem from class imbalance, as evidenced by the low F1 scores of 24.69% and 32.31% for the 'Fear' and 'Disgust' emotions, respectively, in the MELD dataset. This phenomenon also constitutes a primary constraint on the performance of the MERC task, a fact supported by the results in Table 4 . Furthermore, we are also investigating cases where the CMERC-enhanced MERC model misclassifies samples correctly predicted by Baseline ‡ , totaling 89 samples in the IEMOCAP dataset. Notably, it struggles particularly in Short (48 samples) and Medium (34 samples) positions of conversations, with relatively better performance observed in Long positions (7 samples). This could be due to the benefits of CL, as CL exhibits outstanding performance in utterances at long positions in conversations, as depicted in Fig 9 . Misclassifications are distributed almost evenly across samples with and without ES types. This suggests that our method is not affected by the phenomenon of emotional shift, regardless of whether it is present or not.To evaluate the generalizability of our CMERC, we conduct experiments with MERC models, as presented in Table 6 . Noticeably, we observe a consistent decrease in CEL and an improvement in W-F1 scores across all methods. Additionally, similar enhancements are reflected in other confidence estimation metrics. These findings demonstrate that the CMERC significantly enhances the reliability and generalization capability of various MERC models.In Fig. 10 , we examine misclassified uncertain samples, where color intensity indicates the contribution of modalities or contextual cues. The Baseline tends to favor the textual modality, overlooking other modalities. Through calibration, improvements are observed. For instance, in the first utterance, the model correctly focuses on the visual modality to identify the emotion 'Sad'. Similarly, in the second utterance, emphasizing visual cues helps identify the 'Happy' emotion. For the third and fourth utterances, The Baseline treats inter-and intra-speaker contexts alike, while calibration introduces a preference. Specifically, in the fourth utterance, the model appropriately prioritizes inter-speaker context, recognizing the influence of others on the speaker's 'Frustrated' emotion. 
conclusion: 
 In this paper, we introduce a novel calibration framework CMERC designed to tackle the issue of unreliable predictive confidence in MERC models without altering their structures. It integrates CL to guide the model to learn progressively uncertain samples in an entropy-increasing environment; Hybrid SCL to separate uncertain  
: 
 network [44, 71] . Secondly, there is significant attention on the fusion of multimodal data, including aggregation-based methods like concatenation [17, 60] , tensor product [35, 37] , and attention networks [53, 67] , as well as heterogeneous graph methods [4, 20, 22] . Despite the above progress, the reliability of current MERC methods remains largely unexplored. In classification settings, a crucial aspect of reliability involves developing a robust confidence estimator [6, 36, 41] that accurately quantifies the probability of correct predictions. Such an estimator proves particularly valuable in highstakes situations [15] . In MERC, alongside the precise overall prediction confidence, it is crucial to consider the correlation between confidence and modalities or contextual cues, both intra-speaker (within the same speaker) and inter-speaker (between different speakers) contexts. Intuitively, the confidence of the target category should not increase when a modality or specific contextual cues (intra-or inter-speaker contexts) are removed in a MERC model, as the observed information becomes less comprehensive.However, empirical studies on existing methods reveal a counterintuitive trend -in some cases, the removal of certain modalities or specific contextual cues can lead to an increase in confidence as illustrated in Fig. 1 , defining these as uncertain samples. Especially, in multi-party conversations like MELD [47] , removing context poses a greater risk of unreliable predictions for MERC models compared to removing modality. Conversely, in dyadic conversations such as IEMOCAP [3] , removing modality is more likely to result in unreliable predictions than removing context. This contradicts the fundamental principle in informatics that "the essence of information is to eliminate uncertainty" [2] . This further hampers the reliability of models, making them susceptible to influence when a modality or specific contextual cues are corrupted, as they lack a trustworthy confidence estimator for decision-making. To delve into the underlying reasons, the advanced model M3Net [4] is examined as an example. The analysis reveals that its contradiction with the fundamental principle in informatics arises from its • We have conducted extensive empirical studies revealing that existing MERC methods fail to provide reliable confidence estimation for decision-making. • We propose the CMERC framework to calibrate MERC models for the first time, focusing on calibrating the training strategy, utterance representations, and the loss function. • Experimental results demonstrate that our CMERC can significantly enhance various baselines in generalization and reliability, surpassing the state-of-the-art MERC methods. 
 
 Appendix: Please refer to the supplementary materials for detailed hyperparameter settings.Please refer to the supplementary materials for a comparison of results against various large language models. 


Decision: Test-Time Adaptation for Multimodal Sentiment Analysis

Reviews: review: paper_summary: paper_summary: The paper proposes a three-pronged approach for improving model reliability for emotion recognition - curriculum learning, contrastive learning and confidence constraints on the loss function
strength_weakness: strength_weakness: - The framework proposed in the paper is very general and can be adopted to any emotion recognition model without any major changes to the architecture
- The paper has extremely robust and varied experiments and ablation studies to support the claims made regarding all three contributions
- The curriculum learning scheme seems to be novel, and provides a good way of measuring difficulty of multimodal samples
limitations: limitations: - The authors should expand on the details behind the difficulty measurement for curriculum learning provided in Sec 3.4, lines 278-284.
- The novelty of the proposed contrastive loss is not clear. While the experiments show that the contrastive loss clearly improves performance, projecting the loss as a primary contribution/novelty of the paper (line 134) does not seem warranted.
paper_summary: paper_summary: The paper proposes CMERC framework to calibrate Multimodal Emotion Recognition models to tackle the challenge of miscalibration in MERC models across various modalities and contexts. The paper performs studies to identify the at existing MERC methods fail to provide reliable confidence estimates. The method utilizes a Hybrid Difficulty Measurer to identify difficult modalities for a given utterance followed by a hybrid supervised contrastive learning to identify factors contributing to uncertain samples. Experiments were conducted on the MELD and IEMOCAP datasets. In addition to the Weighted-F1, the paper presents CEL (Confidence Enhancement Level) to validate the proposed approach.
strength_weakness: strength_weakness: The motivation that each modality contribute differently within an utterance is well-grounded. Identification of miscalibration in existing MERC models provided strong motivation towards presenting confidence estimates in addition to the widely used w-F1 score for multi-class classification datasets including MELD and IEMOCAP. Hence, the proposed Hybrid Difficulty Measurer is a strength of the work. 

Extensive ablation studies are provided to validate various components of the proposed method. 

Additionally, performance of existing MERC methods based on the proposed CMERC framework is presented to validate generalizability of the proposed work.
limitations: limitations: While the proposed Hybrid Difficulty Measurer sorts the input modalities based on the difficulty score formulated in the paper, the computational complexity of this additional difficulty measurement is not discussed. Additionally, the time taken for training/inference of existing benchmarks (Table 6) with and without CMERC framework would help demonstrate the tradeoff between computational cost v/s performance gains.
paper_summary: paper_summary: The paper proposes a novel calibration framework CMERC to calibrate MERC models without altering the model structure. CMERC integrates curriculum learning to guide the model in progressively learning more uncertain samples; hybrid supervised contrastive learning to refine utterance representations, by pulling uncertain samples and others apart; and confidence constraint to penalize the model on uncertain samples.
strength_weakness: strength_weakness: The results of the experiment are promising.
limitations: limitations: 1. The abstract mentions that the confidence in these model estimates increases when modal or specific context cues are disrupted, thus defining them as uncertain samples. However, what exactly do uncertain samples refer to? Do they refer to missing modalities? I am very confused by the author's expression.
2. To the best of knowledge of my knowledge, existing MERC methods mainly focus on three aspects: speaker-sensitive (graph-based networks), context-sensitive (recurrent-based networks and transformer-based networks), and distinguishing speakers-sensitive (DialogueRNN). The authors need to perform a more detailed classification of existing MERC methods.
3. I am very confused about what Fig. 1 is trying to express. The author does not explain Fig. 1 in detail in either the figure legend or the main text. I don't think Fig. 1 is necessary. Furthermore, the problem defined by the authors is very unclear. I strongly recommend that the authors rewrite the abstract and introduction.
4. How are the uncertain samples in Table 2 counted? Is it the number of misclassifications by the model? In addition, why some data are missing in the video modality condition in the MELD dataset?
5. Why is there no specific meaning given to the abscissa in Fig. 3?
6. The experimental setup of Table 3 is very ambiguous. The author mentions deleting any modality or specific context in the paper, but in Table 3 I am confused that the author does not explain any implementation details of the missing modality.
7. The experimental comparison results in Table 4 are unreasonable. None of the methods compared in the paper are aimed at scenarios where modalities are missing. The authors should compare with some missing modality methods. (e.g., GCNet [1]). Furthermore, why are the missing proportions of the experimental modal and feature contexts not given? I need to know clearly the performance of the model under different missing rates.
8. Since the author did a statistical significance analysis in the ablation experiment, I suggest that the author give the p-values of each emotion category in the supplementary material.
9. The experimental analysis of hyperparameters is relatively rough. The authors should fix one hyperparameter and then use 3D grid search to verify optimal solutions for other parameters.
10. Some academic terminology expressions are not standardized. The tensor product should be a fusion method rather than an aggregation method.
There are some typos errors as follows:
-	L46, P1, Firstly -> firstly
-	L48, P1, network -> networks
[1] Lian Z, Chen L, Sun L, et al. Gcnet: Graph completion network for incomplete multimodal learning in conversation[J]. IEEE Transactions on pattern analysis and machine intelligence, 2023.
paper_summary: paper_summary: This paper introduces a calibration framework for the task of Multimodal Emotion Recognition in Conversations (MERC). The proposed model enhances performance even in scenarios with missing modalities or contextual information.
strength_weakness: strength_weakness: The authors propose a generalizable model for handling incomplete information samples. The experiments are comprehensive and effectively demonstrate the validity of the proposed method. The  figures can help readers to understand this work.
limitations: limitations: The paper delves into many issues related to the MERC task, which may slightly confuse the logical flow. Focusing more on the points described in the abstract would enhance clarity.

The motivation of the paper needs to be reorganized. If the dataset is of the high quality, as mentioned later in the paper, instances of missing modalities or contextual information should be rare. Therefore, the significance of this method in such scenarios needs to be clarified. Validating the method on datasets of lower quality could better illustrate its importance.

Other comments:
1. The introduction's definition of uncertain samples and  Figure 1's results could be smoother. Which rate of missing information leads to the results in Figure 1? Would it be different with different rates?

2. The analysis of the reason for increased confidence after removing some information is not clear. Therefore, the utilization of confidence constraints might be confusing.

3. Are the  hyperparameters in the total loss may make adjustment difficult?

4. Figure 1 in the introduction mainly highlights the differences between two datasets, yet the trends seem consistent in MMDFN. Additionally, there is no significant difference between M3Net and MMGCN. What causes this discrepancy?

P.S. The paper also mentions models dependent on textual modality, an area where much related research has been conducted. Additionally, research on incomplete modality also exists. Exploring these works may be beneficial.