Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation
Authors: Cong-Duy T Nguyen
Thong Thanh Nguyen
Duc Anh Vu
Anh Tuan Luu
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal Sentiment Analysis
Contrastive Learning
TL;DR: We propose Supervised Angular-based Contrastive Learning for Multimodal Sentiment Analysis to improve discrimination and generalization of multimodal representation.

Abstract: The effectiveness of a model is heavily reliant on the quality of the fusion representation of multiple modalities in multimodal sentiment analysis. Moreover, each modality is extracted from raw input and integrated with the rest to construct a multimodal representation. Although previous methods have proposed multimodal representations and achieved promising results, most of them focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. Additionally, they fail to capture the significance of unimodal representations in the fusion vector. To address these limitations, we introduce a framework called Supervised Angular-based Contrastive Learning for Multimodal Sentiment Analysis. This framework aims to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector's modality. Our experimental results, along with visualizations on two widely used datasets, demonstrate the effectiveness of our approach.

Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representation
Cong-Duy Nguyen', Thong Nguyen², Duc Anh Vul, Luu Anh Tuan
"Nanyang Technological University, Singapore National University of Singapore, Singapore
nguyentz003entu, edu.
anhtuan. lwwentu..duuu
Abstract
(a) because a good movie
Happy
rising tone
(P)
The effectiveness of model is heavily reliant on the quality of the fusion representation of multiple modalities in multimodal sentiment analysis. Moreover, each modality is extracted from raw input and integrated with the rest construct multimodal representation. Al- though previous methods have proposed multi- modal representations and achieved promising results, most of them focus forming pos- itive and negative pairs, neglecting the varia- in sentiment scores within the same class. Additionally, they fail capture the signifi- cance of unimodal representations in the fusion vector. To address these limitations, intro- duce framework called Supervised Angular- based Contrastive Learning for Multimodal Sentiment Analysis. This framework aims enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector's modality. Our perimental results, along with visualizations on two widely used datasets, demonstrate the effectiveness of our approach.
(b) gond
High-pitched Hligh volume
Slightly Frown
0.6 (P)
makes him look good
Frown,
Heavy tone, emphasizes "good" -0.6 (N)
Shake the head
Example in CMU-Mosi dataset. All three transcripts of the samples contain the word "Good," " yet their sentiment scores span a range from highly positive 3.0 to slightly negative -0.6. (P) for positive class, (N) for negative class.
Table 1:
1 Introduction
audio/acoustic signals, and visual modalities (im- ages/videos). This stands in contrast to traditional machine learning tasks that typically focus on sin- gle modalities. such as text or voice (Tay et al.. 2017, 201 Sa.b).
Through internet video-sharing platforms, individ- uals engage daily exchanges of thoughts, experi- ences, and reviews. As a result, there is growing interest among scholars and businesses in study- ing subjectivity and sentiment within these opinion vidcos and recordings (Wei et al., 2023). Conse- quently, the field of human multimodal language analysis has emerged as a developing area of re- search Natural Language Processing. Moreover, human communication inherently encompasses multiple modalities, creating a heterogeneous envi- ronment characterized by the synchronous coordi- nation of language, expressions, and audio modali- ties. Multimodal learning leverages diverse sources of information, such language (text/transeripts),
As shown in Table 1, the primary objective of Multimodal Sentiment Analysis (MSA) is uti- lize fusion techniques to combine data from mul- tiple modalities in order to make predictions for the corresponding labels. the context of emotion recognition and sentiment analysis, multimodal fu- sion is crucial since emotional cues are frequently distributed across different modalities. However, previous studies, highlighted by (Hazarika al., 2022), have pointed that task-related infor- mation is not uniformly distributed among these modalities. Specifically, the text modality often exhibits a higher degree importance and plays
"Corresponding Author

---

Transcript
Video
Face
Sentiment
Voice
Expression
Score
jjust got linished watching an excellent movie called "Mars needs moms"
Enthusiastic Tone, High volume,
Extremely exciting
3.0 (P)
emphasizes "excellent"
storyline
Normal volume
Unemotional
0.6 (P)
Steady
god this is had
Angry.
Aggressive tone,
-3.0 (N)
Stroking the face
High volume
and the only explanation come up with makes this story kind of kind creepy Monotone, Normal volume, Hesitant
Frown
-1.0 (N)
Table 2: Example in CMU-Mosi dataset. The first two samples are positive, while the last two are negative. All four examples demonstrate diversity in sentiment representation. The first two examples belong the positive group. but the first example exhibits happiness across all three modalities, whereas the second example only indicates positivity through the phrase "ok". This pattern is similarly observed in the negative example pair (c,d).
a more pivotal role compared to the visual and au- dío modalities. While words undoubtedly carry significant emotional information (e.g.. words like "good", "great", "bad"), emotions are conveyed not only through words but also other communication channels, e.g. tone of voice facial expressions.
despite belonging to the same class, the examples possess varying sentiment levels. Particularly, the first positive example (2a) has a score of 3.0, while the second positive example has a score of 0.6, indicating significant difference. Similarly, the negative examples in (2c) and (2d) differ in senti- ment. Therefore, these representations should not be considered closely related in the feature space. Furthermore, the distance between the (2a-c) pair should be greater than that the (2a-d) pair, con- sidering the sentiment.
We introduce a fusion scheme called Triplet loss for triplet modality to enhance representations of partial modalities in the fusion vector. our study, we found that incorporating facial expres- sions and tone of voice is crucial in determining communication-reltte scores. Neglecting these modalities lead loss of valuable informa- tion. For instance, in Table 1, we observe that even though all three transcripts convey positive senti- ments about the film, the sentiment scores differ significantly. This difference arises from the varia- tions in facial expressions and tone of voice. When a modality is hidden, it alters the representation and makes it challenging to perceive the underlying sentiment. By considering the interrelation among triplet modalities, we propose self-supervised task bridge the representation gap between the com- plete input and missing modalities. The objective to ensure that an input with a concealed modality is more similar to the complete input than containing only one modality. This approach aims to improve the overall representation of multimodal data.
To address this challenge, we challenge the sumption that all positive samples should be repre- sented similarly in the feature space. In the process of sampling positive and negative pairs, we take into account the sentiment score difference between each sample. When the difference in sentiment score does not exceed a predefined threshold, we consider the input as positive. We propose Super- vised Angular Margin-based Contrastive Learning approach that enhances the discriminative repre- sentation by strengthening the margin within the angular space while capturing the varying degrees of sentiment within the samples. By not assum- ing uniform representation for all positive/negative samples in the feature space, our method effectively handles the differences between their sentiment lev- els.
Previous research often focused on contrastive learning techniques, forming positive and negative representation pairs, but overlooked distinctions within the same class. As illustrated in Table 2,
Our contributions can be summarized follows:
We propose a novel approach called Su- pervised Angular Margin-based Contrastive

---

Training objactives
Audio extractor
Lugen
Cinata
MAB v)
Vieual extractor
It was so well done
FUSION
PREDICTOR
Figure Implementation of our framework. The framework consists of three inputs for text, visual and audio modalities, finally, the whole model train with three objectives: Supervised Angular Margin-based Contrastive Learning and Triplet Modalities Triplet lossi.
Learning for Multimodal Sentiment Analysis. This method enhances the discriminative rep- resentation of samples with varying degrees of sentiment, allowing for more accurate clas- sification.
Some works such as (Wöllmer et al., 2013) and (Po- ria et al., 201 6a) have explored the utilization hybrid fusion techniques that integrate both early and late fusion methods. In addition, MAG-BERT model (Rahman et al., 2020) combines BERT and XLNet with Multimodal Adaptation Gate to in- corporate multimodal nonverbal data during fine- tuning. Besides, (Tsai al., 2020) introduced novel Capsule Network-based method, enabling the dynamic adjustment of weights between modal- itics. MMIM (Han et al., 2021) preserves task- related information through hierarchical maximiza- tion of Mutual Information (MI) between pairs of unimodal inputs and the resulting fusion of multi- ple modalities. Recently, several works (Ma et al., 2021a; Sun al., 2022: Zeng et al., 2022) have focused uncertainly solving the missing modali- ties problem.
We introduce self-supervised triplet loss that captures the generalized representation of each modality. This helps to bridge the representation gap between complete inputs and modaltyyy inputs. improving the overall multimodal fusion.
Extensive experiments were conducted on two well-known Multimodal Sentiment Anal- ysis (MSA) datasets, namely CMU-Mosi and CMU-Mosei. The empirical results and visu- alizations demonstrate that our proposed ap- proach significantly outperforms the current state-of-the-art models in terms of sentiment analysis performance.
Contrastive Learning (Chopra et al., 2005) has led to its wide adoption across various problems in the field and is widely applied in many appli- cations (Nguyen and Luu, 2021; Wu et al., 2022). Noteworthy training objectives such as N-Pair Loss (Sohn, 2016), Triplet Margin Loss (Ma et al., 2021b), Structured Loss (Song et al., 2016), and AreCSE (Zhang et al., 2022), directly apply the principles of metric learning. In supervised down- stream tasks, softmax- based objectives have shown promise by incorporating class centers and penaliz- ing the distances between deep features and their respective centers. Center Loss (Wen et al., 2016), SphereFace (Liu et al., 2017), CosFace (Wang al., 2018), and ArcFace (Deng al., 2018) have be- come popular choices in deep learning applications in computer vision and natural language processing. However, these loss functions are designed specifi- cally for classification tasks and are not appropriate
2 Related Work
Recently, multimodal research has garnered atten- tion because of its many applications and swift expansion (Wang et al., 2019; Nguyen al., 2022, 2023; Wei et al., 2022, 2024). In particular, mul- timodal sentiment analysis (MSA) predicts senti- ment polarity across various data sources, like text, audio, and video. Early fusion involves concate- nating the features from different modalities into single feature vector (Rosas al. 2013; Poria al. 2016b). This unified feature vector then employed as input. Late fusion entails construct- ing separate models for each modality and subse- quently combining their outputs to obtain a final re- sult (Cai and Xia 2015; Nojavanasghari et al. 2016).

---

data Xm E Rlmxdm.
he
h
Mono-modal representation
h Text Encoding (Xt)
(1)
h Visual_ Visual_Encoding (Xx,)
(2)
0at
-- 1
AxA
hg Audio Encoding (Xa)
(3)
where hm
RR m E v, a
Visual and Audio _Encoding LSTMs and Text _onc__in is BERT model.
bidirectional are
Figure 2: The framework of Supervised Angular Margin-based Contrastive Learning
for regression labels. Consequently, we put forward a novel training objective called SupAre, which ex- hibits enhanced discriminative capabilities when modeling continuous sentiment scores compared to conventional contrastive training objectives.
= FUSION ([ht, Nv. Ial)
(4)
where FUSION module multi mulli-llyyerre perceptron and [.....] is the concatenation of three modality representations.
3 Methodology
Inference Finally, take the output of fusion and feed it into predictor get predicted score y/
3.1 Problem Statement
We aim to discern sentiments in videos by utilizing multimodal signals. Each video in the dataset is broken down into utterances-an utterance is segment of speech separated by breaths or pauses. Every utterance, mini-video in itself, is treated as an input for the model.
PREDICTOR (h)
(5)
where á ranges from -3 to 3 and PREDICTOR module a multi-layee perceptron.
Training objective
For each utterance U, the input is comprised three sequences low-level features from the language (t), visual (v), and acoustic (a) modali- ties. These are denoted as U,, Uv, and Ua. each belonging to its own dimensional space, where Um E Rlmxdm, with in represents the sequence length and dm represents the dimension of the modality m's representation vector, m E t, u, a. Given these sequences Un (where m can be f, v. or a), the primary objective is to predict the emo- tional direction the utterance U. This can be from a predefined set of C' categories, represented y, or a continuous intensity variable, represented y.
In this section, we introduce the objective function of our model with two different optimization objec- tives: Supervised Angular Margin-based Con- trastive Learning and Triplet Modalities Triplet loss. The model's overall training is accomplished by minimizing this objective:
= Cmain + a Luppare + 3 Cti
(6)
Here, a, 3 is the interaction weights that deter- mine the contribution of each regularization compo- nent the overall loss L. Lmain is main task loss, which mean absolute error. Each of these com- ponent losses is responsible for achieving the de- sired subspace properties. In the next two sections, we introduce two different optimization objectives:
3.2 Model Architecture
As shown Figure 1, each sample has three raw inputs: text transcript of video, video, and audio. Before feeding into the model, our model firstly processes raw input into numerical sequential vec- tors. We transform plain texts into sequence of integers for the transcripts using tokenizer. We use feature extractor to pre-process raw format into numerical sequential vectors for visual and audio. Finally, we have the multimodal sequential
4.1 Supervised Angular Margin-based Contrastive Learning
This section explains the formulation and its deriva- tion from our Supervised Angular Margin-based Contrastive Learning (SupAre).
In order to represent the pairwise associations between samples as positive or negative, we first

---

create fusion representations and categorize them into positive and negative pairs. These pairs are then used as input for a training objective we aim to optimize. In classical NLP problems text-only tasks - this classification hinges on the specific rela- tionships we wish to highlight positive pairs may represent semantically similar or relevant sentences. In contrast, negative pairs typically consist of un- related contextually disparate sentences. In this multi-modal setting, we consider fusion represen- tation from equation 4. Previous works consider samples whose sentiment score is more significant than (or less than 0) to be the same class in their objective function. However, we get each posi- tive (and negative) sample in batch by looking at the difference between the sentiment scores of the two examples. A pair of fusion representations considering same class have its sentiment score difference under threshold TH:
Margin
Text
Visual
2
Audio
Masking
2
Get close
Push
Figure 3: Implementation of Triplet Modalities Triplet loss.
dataset how can model the volume of the difference of pair of samples. To overcome the problem, we propose new training objective Supervised Angular-based Contrastive Loss modified from 4.1 for fusion representation learning by adding be- tween opposing pair h4 and hj. Figure2 illustrates our implementation, à is the positive sample of a, while b, are two negative samples. Moreover, the difference between pair than a b. Thus, the margin (dashed line) of (pur- ple) further than (blue). The objective function called SupAre is formulated as follows:
Ub yز1 S threshold {
4J
other
After gathering the positive and negative pairs, we put them into a training objective:
is more significant
sim (hah))
C -lg
e h,17
where sim is the cosine similarity, F is tempera- ture hyper-parameter and n is the number of sam- ples within batch, i* is positive sample where t.,s- 1.
( 3/7
Ceupare -log
( / + E} colors mDdz)/T
(11)
Past research (Zhang et al., 2022) had proposed an AreCos objective to enhance the pairwise dis- criminative ability and represent the entailment re- lation of triplet sentences:
where Dij = 1y - My1 is the difference of senti- ment score between two samples and
4.2 Triplet Modalities Triplet loss
mm),
Previously the training objectives only considered the fusion representation between different samples with similarities or differences in features within a given space. reality, each type of input (visual, text, audio) plays a vital role predicting the sen- timent score. For example, examples 1a, 1b, 1c all have almost similar compliments (containing "good" in the sentence), thus if we consider only text in this task, the sentiment scores should not be too much different, however, with the difference between facial expressions (visual) and phonetic- prosodic properties (audio), the sentiment scores now are varying degrees. Thus, losing one or two modalities before integration can cause lacking in- formation. Existing methods cannot handle such
Larcens -log e (B, +m//T
Eif e000sa/T
where angular Big is denoted as follow:
f.j = arccos (1Z4)) Af hh [1 * [Asll)
(10)
and m is an extra margin, è is cos function However. there are differences between samples if considering examples 2c and 2c are both nega- tive samples of example 2a, moreover in the fusion space, for example, the distance between, we mod- ify the contrastive which can work on supervised

---

problems between modalities and can cause bias problem in one modality (Hazarika al., 2022). To distinguish the slight differences in semantics between different sentences, we propose a new self- supervised task that models the entailment relation between modalities of an input. For each triplet modality {1, U, a in the dataset, we will mask one and two in three modalities before feeding into the fusion module. We will have a for masking one modality x and haw for masking two x, y of three.
The CMU-MOSEI dataset ex- pands upon the MOSI dataset by including greater quantity of utterances, a wider range of samples, speakers, and topics. This dataset encompasses 23,453 annotated video segments (utterances), de- rived from 5,000 videos, 1,000 unique speakers, and 250 diverse topics.
CMU-MOSEI
5.2 Metries
In our study, we adhere to the same set of evalua- tion metries that have been persistently used and juxtaposed in past research: Mean Absolute Er- ror (MAE), Pearson Correlation (Corr), seven-class classification accuracy (Acc-7), binary classifica- tion accuracy (Acc-2), and Fl score.
1/ = FUSION ([O * has by: hel)
(12)
h,2. FUSION ([O # haz O+ hys hzl)
(13)
As masking only one modality will keep more information than masking two, thus, in the fusion space, it must be nearer to the complete. h is more similar to his than ha As illustrated in Fig- ure 3, three vectors that are losing one modality (one masking) must be closed to the completed one in the center, while others with only one modality left have be moved far away. We apply a triplet loss an objective function:
The Mean Absolute Error (MAE) is popularly used metric for quantifying predictive performance. calculates the average of the absolute differences between the predicted and actual values, thereby providing an estimate of the magnitude of predic- tion errors without considering their direction. It gives an intuition of how much, on average, our predictions deviate from the actual truth values.
Pearson Correlation (Corr) statistical mea- sure that evaluates the degree relationship or as- sociation between two variables. In our context, it indicates the extent of skewness in our predictions. This measure is particularly crucial as it helps un- derstand if our prediction model overestimates or underestimates the true values.
Ltri
ma sh, myyx)
z.yE(t.v.) - sh, hxaa) + Meri)
(14)
where s(a, b) is the similarity function between a and b we use cosine similarity in this work, mari is margin value.
The seven-class classification accuracy (Ace-7) is another significant metric. It denotes the ratio of correctly predicted instances that fall within the same range of seven defined intervals between 3 and +3, in comparison with the corresponding true values. This metric essential for understand- ing how accurately our model can classify into the correct sentiment intervals.
5 Experimental Setup
In this part, outline certain experimental aspects such the datasets, metrics and feature extraction.
5.1 Datasets
We perform experiments on two open-access datasets commonly used in MSA research: CMU-MOSI (Zadeh et al., 2016) and CMU- MOSEI (Bagher Zadeh et al., 2018).
Furthermore, we also employ binary classifica- tion accuracy (Acc-2), which provides insights into how well our model performs in predicting whether sentiment is positive negative. It is computed as the ratio of the number of correct predictions the total number of predictions.
CMU-MOSI This dataset comprises YouTube monologues where speakers share their views on various topics, primarily movies. It has total of 93 videos, covering 89 unique speakers, and includes 2.198 distinct utterance-video segments. These utterances are hand-annotated with a continuous opinion score, ranging from -3 3, where sig- nifics strong negative sentiment, and +3 indicates strong positive emotion.
Lastly, the Fl score is computed for posi- tive/negative and non-negativve/neeatiee classifica- tion results. This metric is harmonic mean of precision and recall, providing a balanced measure of a model's 's performance in binary or multi-class classification setting. It's particularly useful when the data classes are imbalanced, helping us assess

---

CMU-MOSI
CMU-MOSEI
Models
MAE
F1
Cor
Acc-7
Acc-2
MAE
FI
Corr
Acc-7
Acc-2
ICCN
0.862
183.0
0.714
39.0
183.0
0.565
-184.2
0.713
51.6
184.2
MulT
80.6/83.9
0.711
81.5/84.1
0.580
/82.3
0.703
-82.5
MISA
0.804
80.77/82.03
0.764
80.79/82.10
0.568
82.67/83.97
0.724
82.59/84.23
MAG-BERT
0.731
82.6/84.3
0.789
82.5/84.3
0.539
83.7/85.1
0.753
83.8/85.2
Self-MM
0,713
84,42/85,95
0.798
84,00/85.99 98
0.530
82.53/85.00 333
0.765
82.81/85. 177
MIMM
0,700
84.00/85.98
0.800
46.65
84.14/86.06
0.526
82.60/85.94
0.772
54.24
82.24/85.97
Ours
0.679
84.32/86.25
0.806
48.11
84.40/86.28
0.520
84.71/85.88
0.774
55.01
84.57/85.97
Table 3: Results on CMU-MOSI and CMU-MOSEI are as follows. present two sets evaluation results for Acc-2 and Fl: non-negativeheggaiv (non-neg) (left) and positive/negative (pos) (right). The best results are indicated by being marked in bold. experiments, which are underlined, have experienced paired t-tests with p < 0.05 and demonstrated overall significant improvement.
Models
MAE
Curr
Acc-7
Acc-2
5.3.3 Acoustic Features
84.71785.88
55.01
84,57785.97
Lappore
0,527
79,82/83.83
0.766
54.61
79.21/83.76
Multimodal sentiment analysis relies crucial audio properties like MFCC, spectral centroid, spectral flux, beat histogram, beat sum, most pro- nounced beat, pause duration, and pitch. Extracted using COVAREP, these low-level statistical audio functions contribute to the 74-dimensional acous- tic features (d,) in both MOSI and MOSEL. These features comprise 12 Mel-frequency cepstral coef- ficients, pitch, VUV segmenting attributes, glottal source parameters, and more related to emotions and speech tonality.
wlo Lest
0.523
83.53/85.41
0.771
54.37
83.24/85.63
w/o Crupores Eeri
0.532
81.87/85.28
0.758
54.21
81.40/85.32
Table 4: Ablation results on CMU-MOSEL.
the effectiveness of our model in handling such scenarios.
5.3
Feature Extraction
For fair comparisons, we utilize the standard low- level features that are provided by the respective benchmarks and utilized by the SOTA methods.
5.4 Settings
5.3.1 Language Features
We use BERT (Devlin al., 2018) (BERT-base- uncased) as the language model for the text extrac- tor, we load the BERT weight pre-trained on Book- Corpus and Wikipedia from Pytorch framework Huggingface, and we also use BERT tokenizer for feature extractor on text, on top of the BERT, we apply an MLP to project the hidden representation into smaller space. For acoustic and visual, we use 1-layer BiLSTM with feature sizes of input are 74, 36, respectively, and the hidden state's sizes are both 32. The fusion module is the MLP, with an input size of 32x3 (text, visual, acoustic) and for fusion vector size 32. We set a 0.1, 8 = 0.1 for both objectives. We trained our model is trained with a learning rate 4, = 1e in 12 epochs using Adam W as an optimizer; we set the batch size of 32 on V100 GPU and trained about 2-4 hours for CMU-MOSEL and about 30 minutes for CMU- MOSI.
Previous studies used GloVe embeddings for each utterance token, however, recent works have started to apply pre-trained transformers model (like BERT) on input transcripts. We also follow this setting which uses BERT tokenizer- WordPiece tokenizer. It works by splitting words either into full forms (e.g.. one word becomes one token) or into word pieces- - where one word can be broken into multiple tokens.
5.3.2 Visual Features
The MOSI and MOSEI datasets use Facet gather facial expression elements, including facial action units and pose based on FACS. Facial expressions are vital for understanding emotions and senti- ments, they serve a primary means of con- veying one's current mental state. Smiles are par- ticularly reliable in multimodal sentiment analysis, and OpenFace,
Experimental Results
open-source tool, can extract and interpret these visual features. This process is performed for each frame in the video sequence. The dimensions the visual features are dy, with 47 for MOSI and 35 for MOSEI.
6.1 Baseline
We conducted a comparison between our method and other baselines

---

(a) Base text-visual
(b) model text-visual
(c) text-audio
(d) model text-audio
(e) visual-audio
Our model visual-audio
Figure 4: Visualization of the masking fusion representation in the testing set of the MOSEI dataset using PCA
technique. We conduct on 3 pairs: text-visual, text-audio, visual-audio on models: base model and ours.
Self-supervised Multi-Task Learning SELF- MM (Yu et al., 2021) assigns specific uni- modal training task to each modality, utiliz- ing automatically generated labels. This work aims modify gradient back- mack-propagation
Multimodal-infomee MMIM (Han et al., 2021): utilizes two-level mutual information (MI) maximization approach to synthesize fu- sion results from multi-modality.
(a) Base model
(b) Original contrastive
6.2 Results
Following the methodology of previous studies, conducted five times of model using the same hyper-parameter settings. The average per- formance is presented Table 2. Our findings indicate that our model achieves better or compara- ble results compared to various baseline methods. To provide further details, our model demonstrates significant performance advantage over the state- of- th-thh-art (SOTA) in all metrics CMU-MOSL, as well in MAE, Corr, Acc-7, Ace-2 (non-neg), and F1 (non-neg) scores CMU-MOSEL Regard- ing others, our model exhibits very similar perfor- mance on Acc-2 (pos) and is slightly lower than MMIM on Fl (pos). These results offer initial evidence of the effectiveness of our approach in addressing this task.
(c) model
Figure 5: Visualization of the fusion representation in the testing set MOSEI dataset using PCA tech- nique. We conduct three settings: (a) base model, (b) original contrastive loss, (c) our model.
Multimodal Transformer MulT (Tsai al., 2019) develops an architecture that incorpo- rates separate transformer networks for uni- modal and crossmodal data. The fusion pro- cess is achieved through attention mecha- nisms, facilitating comprehensive integra- tion of the different modalities.
Interaction Canonical Correlation Network ICCN (Sun et al., 2019) minimizes canoni- cal loss between modality representation pairs to ameliorate fusion outcome.
Analysis
7.1 Regularization
-nnaiit and -Specific Representa- tions MISA (Hazarika al., 2020) involves projecting features into two distinct spaces with specific constraints, one for modality- invariant representations and the other for modality-specific representations.
To validate significance of our objectives, we select top-performing models for each dataset and gradually remove one loss at a time. Setting the cor- responding variables (a, ,3) to nullifies specific loss. Results are presented in Table 4, indicating that incorporating all losses leads to the best per- formance. While the triplet loss shows slight improvement, the contrastive loss significantly en- hances performance. These findings suggest that distinct representations of fusion spaces are truly beneficial. Combining the triplet and contrastive
Multimodal Adaptation Gate for BERT MAG- BERT (Rahman et al., 2020) designs and in- tegrates aligument gate into the BERT model to enhance and fine-tune the fusion process.

---

losses can further improve performance. The triplet loss helps avoid modality bias (c.g,, text), thus opti- mizing fusion space when used conjunction with contrastive loss.
egorized as multimodal sentiment analysis task, which is inherited from traditional text-based senti- ment analysis tasks. However, we are only working on annotation with sentiment in the range [-3, 3] while traditional text-based sentiment analysis can extend as an emotion recognition which is a multi- class classification task. We have not studied our methods on emotion and we acknowledge these ar- eas as opportunities for future research, aiming enhance our framework's 's optimization in various contexts and utilize wider range of applica- tions.
7.2 Visualizing Contrastive Representation
When applying contrastive objectives, it is impor- tant to analyze the generalization of these charac- teristics. To do so, we visualize fusion vector repre- sentations for test samples. Figure 5 shows that the angular-based contrastive loss performs better than the base model and original contrastive. Both con- trastive losses effectively map vectors for sentiment similarity, unlike the base model's representations, which lack clarity.
Acknowledgement
We thank all anonymous reviewers for their helpful comments. This research is supported by the Na- tional Research Foundation, Singapore under Al Singapore Programme, AISG Award No: AISG2- TC-2022-005 and by the Singapore Ministry of Ed- ucation (MOE) Academic Research Fund (AcRF) Tier (RS21/20).
7.3 Visualizing
modality fusion
We visualize masked modality fusion vectors to compare different representations when removing a modality. Following previous settings, we vi- sualize fusion vectors with one modality masked out for two models: the base model and our ap- proach. In Figures 4, the text-visual and text-audio patterns resemble the complete version. However, the base model's s representation has heavy over- lap between data points, while our method shows more generalization. For the audio-visual pair, both models poorly represent the features, but the base model mixes data points with different label scores. On the contrary, our model with dual training ob- jectives separates neutral points and divides posi- tive/negative nuances into two distinct groups.
References
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria. Frik Cambria, and Louis Shiis-Philippe Morency. 2018. Multimodal language amalysis in the wild: CMU- MOSEI dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2236-2246, Melbourne. Australia. Association for Computational Linguistics.
Guoyong Cai and Binbin Xia. 2015. Convolutional neu- ral networks for multimedia sentiment analysis. In Natural Language Processing and Chinese Comput- ing: 4th CCF Conference, NLPCC 2015, Nanchang, China, October 9-13, 2015, Proceedings 4, pages 159-167. Springer.
8 Conclusion
In conclusion, framework, Supervised Angular- based Contrastive Learning for Multimodal Senti- ment Analysis, addresses the limitations of existing methods by enhancing discrimination and gener- alization in the multimodal representation while mitigating biases in the fusion vector. Through ex- tensive experiments and visualizations, we have demonstrated the effectiveness of our approach in capturing the complexity of sentiment analy- sis. The publicly available implementation of our framework paves the way for further research and development in this field, benefiting applications such as opinion mining and social media analysis.
S. Chopra, R. Hadsell, and Y. LeCun. 2005. Learning similarity metric discriminatively. with application to face verification. 2005 IEEÉ Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR 005) volume 1, pages 539-546 vol. 1.
Jiankang Deng, Jia Guo, and Stefanos Zafeiriou. 2018. Areface: Additive angular margin loss for deep face recognition. CoRR, abs/1801 801.07698.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04800
9 Limitations
Wei Han, Hui Chen, and Soujanya Poria. 2021. Im- proving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on
Our model works on two popular datasets: CMU- Mosi and CMU-Mosei. These datasets are cat-

---

Empirical Methods in Natural Language Processing, pages Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Soujanya Poria, Iti Chaturvedi, E. Cambria, and Amir Hussain. 2016b. Convolutional mkl based multi- modal emotion recognition and sentiment analysis. 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 439-448.
Devamanyu Hazarika, Yingting Li. Bo Cheng, Shuai Zhao, Roger Zimmermann, and Soujanya Poria. 2022. Analyzing modality robustness in multimodal senti- ment analysis. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 685-696, Seattle, United States. Association for Computational Linguistics.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Ami- rAli Bagher Zadeh, Chengfeng Mao, Louis-Phillipe Morency, and Ehsan Hoque. 2020. Integrating mul- timodal information in large pretrained transform- ers. In Proceedings of the 58th Anmual Meeting of the A Association for Computational Linguistics, pages 2359 2369, Online. Association for Computational Linguisties.
Devamanyu Hazarika, Roger Zimmermann, and Sou- janya Poria. 2020. Misa: boddaliy-invariane and -specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM Interna- tional Conference on Multimedia, MM '20, page 1122-1131, New York, NY, USA. Association for Computing Machinery.
Verónica Pérez Rosas, Rada Mihalcea. and Louis- Philippe Moreney. 2013. Multimodal sentiment anal- ysis of spanish online videos. IEEE intelligent sys- tems, 28(3)38844.
Kihyuk Sohn. 2016. Improved deep metric learning with multi n-pair loss objective. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhik- sha Raj, and Le Song. 2017. Sphereface: Deep hy- persphere embedding for face recognition. In 2017 IEFE Conference on Computer Vision and Puttern Recognition (CVPR), pages 6738-6746.
Hyun Oh Song, Yu Xiang, Stefanic Jegelka, and Sil- vio Savarese. 2016. Deep metric learning via lifted structured feature embedding. In 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR). pages 4004 4012.
Mengmeng Ma. Jian Ren, Long Zhao, S. Tulyakov, Cathy Wu, and Xi Peng. 2021a. Smil: Multimodal learning with severely missing modality. ArXiv, 1556570..
Teng Sun, Wenjie Wang, Liqaing Jing, Yiran Cui, Xue- meng Song, and Liqiang Nie. 2022. Counterfactual reasoning for -ut-of-distribution multimodal senti- ment analysis. In Proceedings of the 30th ACM In- ternational Conference on Multimedia. ACM.
Xiaofei Ma. Cicero Nogueira dos Santos, and Andrew O. Arnold. b021b. Contrastive fine-tuning improves ro- bustness for neural rankers. In Findings of the Asso- ciation for Computational Linguistics: ACL-IJCNLP 2021, pages 570-582, Online. Association for Com- putational Linguisties.
Zhongkai Sun, Prathusha Kameswara Sarma, William A. Sethares, and Yingyu Liang. 2019. Learning relation- ships between text, audio, and video via deep canoni- cal correlation for multimodal language analysis. In AAAI Conference on Artificial Intelligence.
Thong Nguyen and Anh Tuan Luu, 2021. Contrastive learning for neural topic model.
Thong Nguyen, Xiaobao Wu, Xinshuai Dong, Anh Tuan Luu, Cong-Duy Nguyen, Zhen Hai, and Lidong Bing. 2023. ctradientbboostee decision tree for listwise context model in multimodal review helpfulness pre-
Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018a. Attentive gated lexicon reader with con- trastive contextual co-attention for sentiment clas- sification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3443-3453.
diction.
Thong Nguyen, Xiaohao WU, Anh-Tuan Luu, Cong- Duy Nguyen, Zhen Hai, and Lidong Bing. 2022. Adaptive contrastive learning multimodal trans- former for review helpfulness predictions.
Yi Tay. Luu Anh Tuan, and Siu Cheung Hui. 2017. Dyadic memory networks for aspect-based sentiment analysis. In Proceedings of the 2017 ACM on Con- ference on Information and Knowledge Management, pages 107-116.
Behnaz Nojavanasghari, Deepak Edakkattil Gopinath, Jayanth Koushik, Tadas Baltruaitis, and Louis- Philippe Morency. 2016. Deep multimodal fusion for persuasiveness prediction. Proceedings of the 18th ACM International Conference on Multimodal Interaction.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018b. Learning to attend via word-asped aspect associative fusion for aspect-based sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32.
Soujanya Poria, Erik Cambria, Newton Howard, Guang- Bin Huang, and Amir Hussain. 2016a. Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocompuing, 174:50-59.
Huo-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for

---

unaligned multimodal language sequences. In Pro- ceedings of the conference. Association for Computa- tional Linguistics. Meeting, volume 2019, page 6558. NIH Public Access.
Amir Zadeh. Rowan Zellers, Eli Pincus, and Louis- Philippe Morency. 2016. MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. CoRR, abs/ 606.06259.
Yao-Hung Hubert Tsai, Martin Ma. Muqiao Yang. Ruslan Salakhutdinov, and Louis-Phillipe Morency. 2020. Multimodal routing: Improving local and global interpretabilitty of multimodal language anal- ysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1823-1833, Online. Association for Computational Linguistics.
Jiandian Zeng, Tianyi Liu, and Jiantao Zhou. 2022. Tag- assisted multimodal sentiment analysis under uncer- tain missing modalities.
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Bingiang Zhao. 2022. contrastive framework for learning sentence representations from pairwise and triple-wisc perspective in angular space. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4892-4903, Dublin, Ireland. Association for Computational Linguisties.
Anran Wang, Anh Tuan Luu, Chuan-Sheng Foo, Hongyuan Zhu, Yi Tay, and Vijay Chandrasekhar. 2019. Holistic multi-modal memory network for movie question answering. IEEE Trunsactions on Image Processing, 29:489 499.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jingchao Zhou, and Wei Liu. 2018. Cosface: Large margin cosine loss for deep face recognition. CoRR, abs/1801.00 .09414.
Jie Wei, Guanyu Hu, Luu Anh Tuan, Xinyu Yang. and Wenjing Zhu. 2023. Multi-scale receptive field graph model for emotion recognition in conversa- tions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages! 1-5. IEEE.
Jie Wei, Guanyu Hu, Xinyu Yang. Anh Tuan Luu, and Yizhuo Dong. 2022. Audio-visual domain adaptation feature fusion for speech emotion recognition. In INTERSPEECH, pages 1988-1992.
Jie Wei, Guanyu Hu, Xinyu Yang, Anh Tuan Luu, and Yizhuo Dong. 2024. Learning facial expression and body gesture visual information for video emo- tion recognition. Expert Systems with Applications, 237:121419.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In Computer Vision ECCV 2016, pages 499-515, Cham. Springer International Publishing.
Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong, 2022. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2768-2660.
Martin Wöllmer, Felix Weninger, Tobias Knaup, Bjöm Schuller, Congkai Sun, Kenji Sagae, and Louis- Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in audio-visual context. JEEE Intelligent Systems, 2863):46-53
Wenmeng Yu. Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self- supervised multi-task learning for multimodal senti- ment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10790-1075


Reviews:
[{"review_id": "4YfKSlzb8f", "reviewer": "Reviewer_qpZv", "summary": "", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments!\nWe're glad that you appreciate our well-written paper and comprehensive experiments.\nWe sincerely hope our response can address your concerns and improve your ratings.\n\n===========================================================================================\n\n**_Q1: Do you mean previous MSA research mainly uses contrastive learning for representation learning?_**\n\nThank you for your question!\nWhat we mean is that previous works only focus on the difference between positive class and negative class, but not on how different they are within a class. For example, in MMIM, they build two normal distributions for positive (score > 0) and negative (score < 0)  samples to calculate the entropy of a multivariate normal distribution but do not consider the differences between samples in the same class.\n\n===========================================================================================\n\n**_Q2: I can't find equation 3.2 because of the format mismatch. Do you mean equation (4)?_**\n\nYes, it should be equation (4). Thank you for pointing them out, and we will correct them in the revision.\n\n===========================================================================================\n\n**_Q3: The benefit of the loss function defined in equation (11) should be detailed. Why it can model the volume of the difference of a pair of samples?_**\n\nIn ArcFace and ArcCSE, the authors want to push away the negative sample with m degree using m (extra margin for decision boundary in angular-based CL). However, m is fixed for all anchor-negative pairs in the batch. Our SupArc uses the volume between label scores to scale m to adaptively push away negative samples with different extra margins. For instance, if we get a pair of examples 2a and 2b (Table 2), we will have an extra margin of 2.4m (3.0 - 0.6) while taking a pair of 2a and 2c, the extra margin is now 6.0m (3.0 - -3.0).\n\n===========================================================================================\n\n**_Q4: The modality-losing problem has been solved by several previous works, but the authors don't discuss them._**\n\nThank you for your suggestion! We will discuss them in the related work section of the revision.\n\n===========================================================================================\n\n**_Q5: More recent baselines should be compared, such as Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis_**\n\nThank you for your suggestion. Among three related papers [1,2,3] that you suggested, only two papers [1,2] address the same tasks (CMU-MOSI, CMU-MOSEI) while the others solve the problem of Multimodal Emotion Recognition.\nAs in [1], the authors only reported two metrics (i.e., Acc and F1) on 7-class classification and 2-class classification, while [2] reported only the accuracy of 2-class classification. We provide the comparison as follows:\n\n|      |  |           | CMU-MOSI      |           |           |  |           | CMU-MOSEI       |           |           |\n|------|:--------:|:---------:|:-----:|:---------:|:---------:|:---------:|:---------:|:-----:|:---------:|:---------:|\n|      | MAE      | F1        | Corr  | Acc7      | Acc2      | MAE       | F1        | Corr  | Acc7      | Acc2      |\n| TATE [2] | -        | -         | -     | -         | 84.90     | -         | -         | -     | -         | -         |\n| CLUE [1] | -        | 84.38     | -     | 48.04     | 84.31     | -         | 84.46     | -     | 53.42     | 84.52     |\n| Ours | 0.679    | **86.25** | 0.806 | **48.11** | **86.28** | 0.520     | **85.82** | 0.774 | **55.01** | **85.97**\n\n\nIn general, our method achieves better performance compared to [1,2]. We will include this comparison in the revision.\n\n===========================================================================================\n\n**_Q6: The performance improvement in the experimental section is not significant_**\n\nWe conducted the significant test t-tests with $p<0.05$ (caption of Table 3). It proves that our proposed method achieves significant improvement compared to the baselines.\n\n===========================================================================================\n\n**References:**\n\n[1]. Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis \n\n[2]. Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities \n\n[3]. Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel framework is proposed to enhance the discriminative representation of samples.\n2. The authors introduce a self-supervised triplet loss that captures the generalized representation of each modality.\n3. This paper is well-organized.", "reasons_to_reject": "1. The authors say that ''Although previous methods have proposed multimodal representations and achieved promising results, most of them focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class'', do you mean previous MSA research mainly use contrastive learning for representation learning?\n2. I can't find equation 3.2 because of the format mismatch. Do you mean equation (4)?\n3. The benefit of the loss function defined in equation (11) should be detailed. Why it can model the volume of the difference of a pair of samples?\n4. The modality-losing problem has been solved by several previous works, but the authors don't discuss them. For example, Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities and Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities.\n5. More recent baselines should be compared, such as Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis.\n6. The performance improvement in the experimental section is not significant. ", "questions_to_authors": "See weakness.", "paper_topic_and_main_contributions": "This paper focuses on the Multimodal Sentiment Analysis task and proposes a framework called Supervised Angular-based Contrastive Learning for Multimodal Sentiment Analysis. This framework aims to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities.\nTag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities.\nCounterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis."}}, {"review_id": "TJVbQZytJw", "reviewer": "Reviewer_qMbV", "summary": "", "questions": "How do you compare your results on the effect of various modalities with some old results on the significance of each single modality in emotion detection tasks? you can refer to some results on the above-mentioned works.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments!\n\nWe're glad that you appreciate our well-written paper and comprehensive experiments.\n\nWe sincerely hope our response can address your concerns and improve your ratings.\n\n===========================================================================================\n\n**_Q1: missing some methods including some old methods_**\n\nThank you for your suggestion. We will add those pioneer studies in the related work section of the revision. \n\n===========================================================================================\n\n**_Q2: Related work section_**\n\nThank you for your suggestion! We'll add the important references and others to the related work section. In 2022, UniMSE is proposed and get SOTA on CMU-MOSI, CMU-MOSEI, and also other two emotion recognition tasks. We achieved SOTA on these two tasks; moreover, their SOTA on CMU-MOSI and CMU-MOSEI reported in the UniMSE paper are MMIM () too. However, UniMSE trained their model with large datasets - almost double the amount of training data compared to our method before evaluation:\n\n|      |           |                 |  CMU-MOSI |           |                 |           |                 | CMU-MOSEI |           |                 |\n|------|:---------:|:---------------:|:---------:|:---------:|:---------------:|:---------:|:---------------:|:---------:|:---------:|:---------------:|\n|      | MAE       | F1              | Corr      | Acc7      | Acc2            | MAE       | F1              | Corr      | Acc7      | Acc2            |\n| Uni* | 0.693     | 83.98/85.91     | 0.802     | 47.56     | 84.04/85.96     | 0.523     | 84.56/85.55     | 0.771     | 54.29     | 83.24/85.27     |\n| Ours | **0.679** | **84.32/86.25** | **0.806** | **48.11** | **84.40/86.28** | **0.520** | **84.71/85.82** | **0.774** | **55.01** | **84.57/85.97** |\n\n(*) the results are obtained based on the official published code at https://github.com/LeMei/UniMSE.\n\nIn general, our method achieves better performance compared to UniMSE[1]. We will include this comparison in the revision.\n\n===========================================================================================\n\n**_Q3: The main idea already exists._**\n\nWe proposed a novel contrastive objective, and it’s different from previous approaches. ArcFace [2] focuses loss function used in face recognition tasks, and ArcCse [3] is for sentence representation learning. Both two losses have fixed m which is a margin. Our supervised contrastive objective flexibly adapts m with the difference between anchor and negative samples. Furthermore, we are the first who apply angular-based contrastive learning to regressive tasks, from there, we can extend this technique to different problems not just MSA.\n\n===========================================================================================\n\n**_Q4: Figure 3 has not been referenced in the text._**\n\nThank you for pointing them out, we will add the Figure 3 reference in section 4.2 in the final revision.\n\n===========================================================================================\n\n**References:**\n\n[1]. UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition EMNLP(2022)\n\n[2]. ArcFace: Additive Angular Margin Loss for Deep Face Recognition\n\n[3]. A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Choosing an important problem to focus on and solve\n2. nice extension of an existing method to multimodal representation learning and fusion.\n", "reasons_to_reject": "1. missing some methods including some old methods: https://arxiv.org/abs/1707.07250 , and https://aclanthology.org/W19-4331/ where study effect of dimension and factorization on prediction tasks with applications on the datasets studied in this draft.\n\n2. related work section is very weak, and not covering SOTA. you have to add various integration methods focusing to solve multimodal integration issues. \n\n3. The main idea already exists, and this work is a simple extension of the previously proposed method, with some simple intuitions.", "questions_to_authors": "How do you compare your results on the effect of various modalities with some old results on the significance of each single modality in emotion detection tasks? you can refer to some results on the above-mentioned works.", "paper_topic_and_main_contributions": "The authors aim to capture the significance of unimodal representations in the fusion vector using sentiment scores.  This framework aims to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality using Margin-based Contrastive Learning.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. https://arxiv.org/abs/1707.07250 \n2. https://aclanthology.org/W19-4331/\n3. Many works published from 2021-present", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 3 has not been referenced in the text."}, "extra_text": {}}, {"review_id": "D19RBP9sQ2", "reviewer": "Reviewer_yH9j", "summary": "", "questions": "- What is the main motivation of the triplet loss? Is it mostly for cases when there is a missing modality you can still accurately predict the correct sentiment? Or to avoid over reliance/spurious correlations with one modality? Or both?\n", "limitations": "", "rating": 3, "confidence": 1, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments!\n\nWe're glad that you appreciate our well-written paper and comprehensive experiments.\n\nWe sincerely hope our response can address your concerns and improve your ratings.\n\n===========================================================================================\n\n**_Q1: What is the main motivation of the triplet loss? Is it mostly for cases when there is a missing modality you can still accurately predict the correct sentiment? Or to avoid over reliance/spurious correlations with one modality? Or both?_**\n\nThank you for your question!\n\nOur triplet loss, designed for triplet modalities, endeavors to enhance the representations of partial modalities within the fusion vector. This enhancement is crucial for preventing the model from exhibiting bias towards a single modality. Prior research [1] has demonstrated that numerous Multimodal Sentiment Analysis (MSA) models often exhibit a bias towards the text modality.\n\nOur visualization in Figure 4 effectively demonstrates the efficacy of our proposed method. Upon masking out the text modality, both the base model and our model exhibit poor feature representations. However, a notable disparity arises in the fusion space: the base model's representations appear to be ineffectual, whereas our model successfully generates two distinct positive and negative clusters.\n\nThis visualization underscores the superiority of our approach in mitigating modality bias and improving the overall quality of the learned representations.\n\n===========================================================================================\n\n**_Q2: Grammar Style And Presentation Improvements_**\n\nWe thank you for pointing them out, and we will definitely correct them in the revision.\n\n===========================================================================================\n\n**References:**\n\n[1]. Analyzing Modality Robustness in Multimodal Sentiment Analysis NAACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Considering the varying degree of difference between sentiment scores compared to just sampling points from the positive and negative classes\n- preventing over reliance on a given modality by using the triplet loss \n", "reasons_to_reject": "-", "questions_to_authors": "- What is the main motivation of the triplet loss? Is it mostly for cases when there is a missing modality you can still accurately predict the correct sentiment? Or to avoid over reliance/spurious correlations with one modality? Or both?\n", "paper_topic_and_main_contributions": "The main topic of the paper is in the area of fusing representations from multiple modalities to have better representations. The contributions are:\n- A novel approach called “Supervised Angular Margin Based Contrastive learning for multimodal sentiment analysis” which increases the discriminative representation of samples with different degrees of sentiments. This is introduced as a loss function where there are two additional regularizers. \n- The paper introduces a “self supervised triplet loss” in order to generalize representations of each modality (text, audio, video)\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 312 seems like there is a word missing =  \"to overcome the problem, we propose a new training objective - supervised angular-based contrastive loss modified from 4.1 for fusion representation learning by adding a <missing_term> between opposing par hi and hj\"?"}}, {"review_id": "HFqhgCgsAk", "reviewer": "Reviewer_kyoU", "summary": "", "questions": "A. Explain the analysis clearly especially from figure 4 and figure 5.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments!\n\nWe're glad that you appreciate our well-written paper and comprehensive experiments.\n\nWe sincerely hope our response can address your concerns and improve your ratings.\n\n===================================\n\n**_Q1: Concerns about the Related work_**\n\nThank you for your suggestion! We'll add more important references in the final revision.\n\n===================================\n\n**_Q2: Figure 4 explanation_**\n\nWe extract the fusion features and visualize these vectors for test samples using PCA. **_Base_** means the model with only task loss. The text-visual and text-audio representations resemble the complete version (like in Figure 5). However, when we mask out text modality - the most important modality, although, both models poorly represent the features, the base model shows useless representations. Our model can create two discrete positive/negative groups.\n\n===================================\n\n**_Q3: Figure 5 explanation_**\n\nWith the same visualization setting from experiments in Figure 4, Figure 5 shows that our supervised angular-based contrastive loss performs better than the base model. We can see that all 3 models show generalization. However, in the base model, there are samples with different sentiment points that overlap quite a lot, for example, the slightly red and gray parts or the bluish and gray parts. In original contrastive learning, the feature representation is better when the overlap is less, the examples form a continuous and regular color range. Lastly, our method shows better feature representations, and better generalizability, and puts forward each different sample pair when adding the difference between two label scores to the contrastive loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. Proposed angular margin-based contrastive learning.\nB. Fusion of Self-supervised triplet loss is good.\nB. The paper is written well.\nc. EXperimental setup and analyis are good.", "reasons_to_reject": "A. Related work can be improved. \nB. The insights are not clearly explained from the analysis", "questions_to_authors": "A. Explain the analysis clearly especially from figure 4 and figure 5.", "paper_topic_and_main_contributions": "The paper is about Supervised Angular-based Contrastive Learning for Multimodal Sentiment Analysis to improve discrimination and generalization of multimodal representation. The contributions of the paper include: proposed supervised Angular Margin-based Contrastive Learning for Multimodal Sentiment Analysis, introduced self-supervised triplet loss that captures the generalized representation of each modality, and extensive experimental setup.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Can add more references related to this work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a supervised angular-based contrastive learning for multimodal sentiment analysis and shows how the proposed framework can attain more discriminate and generalized representation from each modality and overcome any bias towards a single modality by enhancing partial modality representation after fusion. The authors show the efficacy of the proposed framework with widely used sentiment datasets: CMU-Mosi and CMU-Mosei and compare with some state-of-the-art model performances.

The challenges addressed in the paper are well-researched, and the proposed framework brings together the ideas and techniques to apply them in MSA. 

The reviews received for the paper are mixed. The majority of the reviewers point out the following concerns:
– The paper has a very weak related section, missing recent and old approaches proposed to solve these challenges. 
– The presented idea is a simple extension and lacks sufficient novelty.
– Some insights are not well-explained in the paper.


The authors addressed some of the concerns in the rebuttal and added a new baseline. The author committed to improve the related work section in future versions of the paper.

Overall the framework proposed for SA, is a simple solution that shows comparable performance compared to  UniMSE (2022) and MIMM (2021).  The framework and the results in the paper can of be of interest to the researchers. We recommend that the authors take the suggestions from the reviews to further strengthen their work.
Decision: Accept-Findings