Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
Authors: Haoyu Zhang
Yu Wang
Guanghao Yin
Kejun Liu
Yuanyuan Liu
Tianshu Yu
Venue: EMNLP 2023 Main (2023)
Keywords: multimodal sentiment analysis
multimodal representation learning
TL;DR: A novel Adaptive Language-guided Multimodal Transformer (ALMT) for multimodal sentiment analysis

Abstract: Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (*e.g.,* language, video, and audio), the potential sentiment-irrelevant and conflicting information across modalities may hinder the performance from being further improved. To alleviate this, we present Adaptive Language-guided Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper-modality representation, the model can obtain a complementary and joint representation through multimodal fusion for effective MSA. In practice, ALMT achieves state-of-the-art performance on several popular datasets (*e.g.,* MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the validity and necessity of our irrelevance/conflict suppression mechanism.

Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
Haoyu Zhang 2, Yu W Guanghao Yin Kejun Liu?, Yuanyuan Li ,Tianshu Yu
'The Chinese University of Hong Kong, Shenzhen
"China University of Geosciences, Wuhan
*Nanyang Technological University
4 Shenzhen Institute of Artificial Intelligence and Robotics for Society
1,4 [=hhagghayyu, yutianshuJ@cuhk. edu. cn
2 (zhangghoyy, vvy190701, ygh2, liukejun, liuyy)ecug. edu. cn cese-yylluêntu. edu. sg
Abstract
am fine.
Model
Positive
Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (e.g. language, video, and audio), the potential sentiment-irelevantt and conflicting infor- mation across modalities may hinder the performance from being further improved. alleviate this, we present Adaptive Language-guider Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn irrelevanceccooffiitsssppess representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper- modality representation, the model can obtain complementary and joint representation through multimodal fusion for effective MSA. practice, ALMT achieves state-of-the-art performance several popular datasets (e.g.. MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the valid- ity and necessity of our ircelvaace/connfiet suppression mechanism.
Model
Negative X
am
Model
-
Negative X
Figure 1: In multimodal sentiment analysis, language modality usually is dominant modality in modali- ties, while audio and visual modalities contributing as much to performance as language modality.
Yu et al., 2021; Han al., 2021; Guo al., 2022) and multimodal fusion-centered methods (Zadeh et al., 2017; Liu al., 2018; Tsai et al., 2019a; Huang et al., 2020). The representation learning- centered methods mainly focus on learning refined modality semantics that contains rich and varied human sentiment clues, which can further improve the efficiency of multimodal fusion for relation- ship modelling. On the other hand, the multi- modal fusion-centered methods mainly focus on directly designing sophisticated fusion mechanisms to obtain a joint representation of multimodal data. In addition, some works and corresponding abla- tion studies (Hazarika et al., 2020; Rahman et al., 2020; Guo et al., 2022) further imply that vari- ous modalities contribute differently to recognition, where language modality stands out as the domi- one. We note, however, information from dif- ferent modalities may be ambiguous and conflict- ing due to sentiment-irrlleancee especially from non-dominating modalities (c.g., lighting and head pose video and background noise in audio). Such
1 Introduction
Multimodal Sentiment Analysis (MSA) focuses on recognizing the sentiment attitude of humans from various types of data, such video, audio, and language. It plays central role in several ap- plications, such healtheare and human-computer interaction (Jiang et al., 2020; Qian et al., 2019). Compared with unimodal methods, MSA methods are generally more robust by exploiting and explor- ing the relationships between different modalities, showing significant advantages in improving the understanding of human sentiment.
Most recent MSA methods can be grouped into categories: representation learning-centered methods (Hazarika al., 2020; Yang et al., 2022;
Coorespoonding author

---

disruptive information can greatly limit the perfor- mance of MSA methods. We have observed this phenomenon in several datasets (see Section 1, and an illustration in Figure 1. To the best of our knowledge, there has never been prior work explic- itly and actively taking this factor into account. Motivated by the above observation, propose a novel Adaptive Language-guided Multimodal Transformer (ALMT) to improve the performance of MSA by addressing the adverse effects of dis- ruptive information in visual and audio modalities. ALMT, each modality is first transformed into a unified form by using a Transformer with initial- ized tokens. This operation not only suppresses the redundant information across modalities, but also compresses the length of long sequences to facilitate efficient model computation. Then, introduce an Adaptive Hyper-modality Learning (AHL) module that uses different scales of lan- guage features with dominance to guide the vi- sual and audio modalities to produce the interme- diate hyper-modality token, which contains less -ireeeent-rrelevant information. Finally, we apply a mooss-my fusion Transformer with language features serving query and hyper-modality fea- tures serving as key and value. In this sense, the complementary relations between language and vi- sual and audio modalities are implicitly reasoned, achieving robust and accurate sentiment predic- tions. In summary, the major contributions of our work can be summarized as:
2 Related Work
In this part, we briefly review previous work from two perspectives: multimodal sentiment anal- ysis and Transformers.
2.1 Multimodal Sentiment Analysis
As mentioned in the section above, most previ- ous MSA methods are mainly classified into two categories: representation learning-centere meth- ods and multimodal fusion-centered methods.
For representation learning-centered methods, Hazarika et al. (2020) and Yang et al. (2022) ar- gued representation learning of multiple modalities as domain adaptation task. They respectively used metric learning and adversarial learning to learn the modality- and mopelity-specific subspaces for multimodal fusion, achieving ad- vanced performance in several popular datasets. Han al. (2021) proposed a framework named MMIM that improves multimodal fusion with hi- crarchical mutual information maximization. Rah- man et al. (2020) and Guo et al. (2022) devised different architectures to enhance language rep- resentation by incorporating multimodal interac- tions between language and non-vert non-verbal behavior information. However, these methods do not pay enough attention sentiment-irrelevant redundant information that more likely to be present visual and audio modalities, which limits the per- formance of MSA.
For multimodal fusion-centered methods, Zadeh et al. (2017) proposed a fusion method (TFN) us- ing tensor fusion network model the relation- ships between different modalities by computing the cartesian product. Tsai al. (2019a) and Huang et al. (2020) introduced a multimodal Transformer to align the sequences and model long-range de- pendencies between elements across modalities. However, these methods directly fuse information from uni-modalities, which is more accessible to the introduction of sentiment- sentiment-irrelevanl informa- tion, thus obtaining sub-optimal results.
We present novel multimodal sentiment analysis method, namely Adaptive Language- guided Multimodal Transformer (ALMT), which for the first time explicitly tackles the adverse effects of redundant and conflicting information in auxiliary modalities (i.e., vi- sual and audio modalities), achieving a more robust sentiment understanding performance.
We devise a novel Adaptive Hyper-modality Learning (AHL) module for representation learning. The AHL uses different scales of language features to guide the visual and au- dio modalities to form a hyper modality that complements the language modality.
2.2 Transformer
Transformer is an attention based building block for machine translation introduced by Vaswani et al. (2017). It leams the relationships between tokens by aggregating data from the entire sequence, show- ing an excellent modeling ability in various tasks, such as natural language processing, speech pro- cessing, and computer vision, etc. (Kenton and
ALMT achieves tfate-ofthe--a art performance in several public and widely adopted datasets. We further provide in-depth analysis with rich empirical results to demonstrate the validity and necessity of the proposed approach.

---

Multimodal
Cross-modality Fusion Transformer
Fusion
Hi Query
H Magee
Key Value
drop
Hi MaDDnnΣE
b00000a1001100110001000011111111 Adaptive Hyper-modality Learning Layer
Transformer
Ef
Layer
Adaptive
Adaptive Hyper-modality Learning Layer
Hyper-modality Learning
Transformer
E
Layer
Adaptive Hyper-modality Learning Layer
H
H 00::: MA00 drop
Ii annntiitt drop
:: 1:00:000
drop
Transformer
Transformer
Transformer Layer
Modality Embedding
E
E
E
Layer
Layer
HC D:
H
BERT
Librosa
OpenFace
Multimadal Input
am fine.
Language
Audio
Video
Figure 2: Processing pipeline of the proposed ALMT for multimodal sentiment analysis (MSA). With the multimodal input, we first apply three Transformer layers to embed modality features with low redundancy. Then, we employ a Hyper-modality Learning (AHL) module to learn dyper-modalee representation from visual and audio modalities under the guidance of language features at different scales. Finally, Cross cross-modality Fusion Transformer is applied to incorporate hyper-modality features based on their relations to the language features, thus obtaining a complementary and joint representation for MSA.
Toutanova, 2019; Carion et al.. 2020; Chen et al., 2022; Liu et al., 2023a). In MSA, this technique has been widely used for feature extraction, repre- sentation learning, and multimodal fusion (Tsai et al., 2019a; Huang et al., 2020; Liu et al., 2023b; Yuan al., 2021).
3.2 Multimodal Input
Regarding the multimodal input, each sample consists of language (I), audio (a), and visual (v) sources. Referring to previous works, we first ob- tain pre-computed sequences calculated by BERT (Kenton and Toutanova, 2019), Librosa (McFee et al., 2015), and OpenFace (Baltrusaitis al., 2018), respectively. Then, we denote these se- quence inputs as Un E RTmxdm, where m E {1, u, a), Tm is the sequence length and dm is the vector dimension of each modality. In practice, Tm and dm are different on different datasets. For ex- ample, on the MOSI dataset, T., Ta. Ti, das d, and dj are 50, 50, 50, 5, 20, and 768, respectively.
3 Method
3.1
Overview
The overall processing pipeline of the proposed Adaptive Language-guin guided Multimodal Trans- former (ALMT) for robust multimodal sentiment analysis is in Figure! 2. As shown, ALMT first extracts unified modality features from the input. Then, Adaptive Hyper-Modality Learning (AHL) module is employed to learn the adaptive hyper- modality representation with the guidance of lan- guage features at different scales. Finally, apply mross-modality Fusion Transformer to synthesize the hyper-modality features with language features as anchors, thus obtaining a language-guid -uided hyper- modality network for MSA.
3.3 Modality Embedding
With multimodal input Un. we introduce threc Transformer layers unify features of each modal- ity, respectively. More specifically, we randomly initialize a low-dimensional token H E T'd'xd... for each modality and use the Transformer to em- bed the essential modality information to these to-

---

H Hoper MANDEE
Adaptive Hyper-modality Learning Layer
H Ma0Tia
+ SV.
aV.
3V.
Q,K² Vd
Q.K vd.
softmax
0 softmax
V.
k.
Q 14 amppor
to
k.1
V.
mangcom
Audio Modality
Language Modality
Visual Modality
Figure 3: An example of the Adaptive Hyper-modality Learning (AHL) Layer.
kens
language features. In practice, we found that the language features significantly impact the model- ing of hyper-modality (with more details in section 4.5.4).
H E (concat(HH, Um), 0pp) 7'xd (1)
where Hi is the unified feature of each modal- ity with size of T x d. Eg and Og respec- tively represent the modality feature extractor and corresponding parameters, concat(-) represent the concatenation operation.
3.4.1
Construction of Two-scale Language Features
We define the feature H| as low-scale language feature. With the feature, we introduce two Trans- former layers to Iearn language features at middle- scale and high-scale (i.e. Hf and H²²). Different from the Transformer layer in the modality embed- ding stage that transfers essential information to an initialized token, layers in this stage directly model the language features:
In practice, T and are set to and 128, respec- tively. The structure of the transformer layer is de- signed as the same as the Vision Transformer (VIT) (Dosovitskiy et al., 2021) with depth setting of 1. Moreover, it is worth noting that transferring the essential modality information to initialized low- dimensional tokens is beneficial to decrease the redundant information that is irrelevant to human sentiment, thus achieving higher efficiency with lesser parameters.
Hi E .0g;) Rxxd
(2)
where i (2, Hi is language features differ- ent scales with a size of T d, EG and 0gg repre- sents the i-th Transformer layer for language fea- tures learning and corresponding parameters. practice, we used 8-head attention to model the information of each modality.
3.4 Adaptive Hyper-modality Learning
After modality embedding, we further employ an Adaptive Hyper-modality Learning (AHL) mod- ule to learn refined hyper-modality representa- tion that contains redevanc/conficcssppresss/ information and highly complements language fea- tures. The AHL module consists of two Trans- former layers and three AHL layers, which aim learn language features different scales and adaptively learn a coder-todality feature from vi- sual and audio modalities under the guidance of
3.4.2 Adaptive Hyper-modality Learning Layer
With the language features of different scales HJ. we first initialize a hyper-modality feature H
hyper E Rxx then update H hyper by calcu- lating the relationship between obtained language

---

features and two remaining modalities using multi- head attention (Vaswani et al., 2017). As shown in Figure 3, using the extracted Hi as query and Ho as key, we can obtain the similarity matrix a between language features and audio features:
CrossTrans, so the fusion process can be written
as:
H1 = Concat H1²) E (TT+1)xd
(6)
Huyper = Concat Ho. H Hhyper) E IR (R(T (7)
Q1K vdL
H CrossTrans (H1, Hhyper) Rlxd
(8)
a softmax
(3)
After the multimodal fusion, obtain the final sentiment analysis output v by applying classifier on the outputs of Cross-modality Fusion Trans- former H. In practice, we also used 8-head atten- tion to model the relationships between language modality and sopertmodllit... For more details the Cross-modality Fusion Transformer, we refer readers to Tsai et al. (2019a).
HiW WT. HIT vd
softmax
JRR
where softmax represents weight normalization operation, Wo: & Rdxde and WK. Rdxd are learnable parameters, dy is the dimension of each attention head. In practice, used 8-head atten- tion and dg 16.
Similar to a, represents the similarity matrix between language modality and visual modality:
3.6 Overall Learning Objectives
To summarize, our method only involves one learning objective, i.e., the sentiment analysis learn- ing loss L, which is:
B softmax Q1K Vdk
H Q WT H K.
softmax
RTT
10"
0
(9)
d
N
==
where WK, Rddx, is learnable.
where No is the number of samples in the training set, y" is the sentiment label of the n-th sample. i is the prediction our ALMT.
Then the hyper-modality features H hyper can be updated by weighted audio features and weighted visual features as:
In addition, thanks to our simple optimization goal, compared with advanced methods (Hazarika et al., 2020; Yu et al., 2021) with multiple optimiza- tion goals, ALMT is much easier to train without tuning extra syper-parameters, More details are shown in section 4.5.10.
Hi Hhaper
Hi-1 + aVa + 3V,
(5)
Hi-1 o.HWW. BH
where j E [1,2,3] and H hyper C Rxxd respec- tively represent the j-th AHL layer and correspond- ing output hyper-modality features, Wy, E Rddd and Wv. E Rdxdx are learnable parameters.
Experiments
4.1 Datasets
We conducted extensive experiments three popular trimodal datasets (i.e., MOSI (Zadeh al., 2016). MOSEI (Zadeh et al., 2018), and CH- SIMS (Yu et al., 2020)).
3.5 Multimodal Fusion and Output
In the Multimodal Fusion, we first obtain new language feature H1 and Hayper and new hyper- modality feature by respectively concatenating ini- tialized a token Hg E 1xd with Hy hyper and H/ Then we apply Cross-modality Fusion Transformer to transfer the essential joint and complementary information to these tokens. In practice, the Cross- modality Fusion Transformer fuse the language fea- tures H2 (serving the query) and hyper-modality features Hhyper (serving as the key and value), thus obtaining joint multimodal representation E x for final sentiment analysis. We de- note the Cross-modality Fusion Transformer as
MOSI. The dataset comprises 2,199 multimodal samples encompassing visual, audio, and language modalities. Specifically, the training set consists of 1,284 samples, the validation set contains 229 samples, and the test set encompasses 686 samples. Each individual sample is assigned sentiment score ranging from -3 (indicating strongly negative) to 3 (indicating strongly positive).
MOSEL. The dataset comprises 22,856 video clips collected from YouTube with a diverse fac- tors (e.g., spontaneous expressions, head poses,

---

Table 1: Comparison on MOSI and MOSEL. Note: the best result highlighted in bold; * represents the result is from Hazarika al. (2020); represents the result is from Mao al. (2022) and its corresponding GitHub page'.
MOSI
MOSEI
Method
Acc-7
Acc-5
Acc-2
F1
MAE
Cor
Acc-7
Acc-5
Acc-2
F1
MAE
Corr
349
LMF'
31.2
-/82.5
-/2.44
0.917
0.695
48.0
-/82.0
-/82.1
0.623
0.677
MEM
35.4
-/81.
1844
-/84.3
MulT
40.0
83.0
-/82.8
0.871
0.698
51.8
-/ 82.5
-/82.3
0.580
0.703
MISA
42.3
81.8/ 83.4
83.6
0.783
0.761
52.2
83.6/85.55
83.8
0.555
0.756
PMR
40.6
83.6
-/ 183,4
52.5
/83.3
- 182.6
MAG-BERT
43.62
82.37 84.43
82.50/ 84.61
0.727
0.781
52.67
82.51 /84822
82.77 94,71
0.543
0.755
SeI--MM
45.79
82,
84,91 84.00
0.712
82.66 84.96
82.
84.93
MMIM
46.05
84.14 86.00
84.00/85.96 85.98
0.700
0.800
54.24
82.24 85.97
82, 66.66/ 85
0.526
0,772
FDMER
16.1
CHEN
48.6
84.3 86.4
84.2/ 86.2 -/-
0.689
0.809
54.3
83.7 /6..
83.9/ 86.1
0.525
0.778
MUIT
42.68
54.18
MISA
47.08
+/-
53.63
Self MM!
53.47
-/-
55.53
ALMT
49.42
56.41
84.55 86.43
84.57 / 86.47
0.683
0.808
54.28
85.96
84.78 86.79
85,19 86.86
0.526
0.779
Table 2: Comparison results on CH-SIMS. Note: best result is highlighted in bold; f represents the result is from Mao et al. (2022) and its corresponding GitHub page!.
(Acc-3), five classification accuracy (Ace-5), seven classification accuracy (Acc-7), mean absolute er- ror (MAB), and the correlation of the model's pre- diction with human (Corr). Moreover, on MOSI and MOSEI, agreeing with prior works (Hazarika et al., 2020), we calculated Acc-2 and Fl in two ways: regative/non-negative and negative/positive on MOSI and MOSEI datasets, respectively.
Method
Acc-5
Acc-3
Acc-2
MAE
Cor
TENT
39,30
65.12
78.38
78.62
0.432
0.591
LMFÍ
40.53
64.68
77.77
77.88
0.441
0.576
MFM
75.06
75.58
0.477
0.525
MuLTI
37.94
64.77
78.56
79.66
0.453
0.564
MISA
76.54
76.59
0.447
0.563
MAG-BERTT
74.44
71.75
0.492
0.399
4.3 Baselines
ScIf-MM!
41.53
65.47
80.04
80.44
0.425
0.595
To comprehensively validate the performance of our ALMT, we make a fair comparison with the several advanced and state-of- state-of-the-art the-art methods, iee, TFN (Zadeh et al., 2017). LMF (Liu et al., 2018), 8), MFM (Tsai et al., 2019b), MuLT (Tsai et al., 2019a), MISA (Hazarika et al., 2020), PMR (Lv et al., 2021), MAG-BERT (Rahman et al., 2020). Self-MM (Yu et al., 2021). MMIM (Han et al., 2021). FDMER (Yang al., 2022) and CHFN (Guo al., 2022).
ALMT
45.73
68.93
81.19
81.57
0.404
0.619
occlusions, illuminations). This dataset has been categorized into 16,326 training instances, 1,871 validation instances, and 4,659 test instances. Each instance is meticulously labeled with sentiment score ranging from -3 to 3. And the sentiment scores from -3 indicate most negative to most positive.
CH-SIMS. is a Chinese multimodal sentiment dataset that comprises 2,281 video clips collected from variuous sources, such as different movies and TV serials with spontaneous expressions, vari- ous head poses, etc. It is divided into 1,368 training samples, 456 validation samples, and 457 test sam- ples. Each sample is manually annotated with sentiment score from -1 (strongly negative) (strongly positive).
4,4 Performance Comparison
Table and Table 2 list the comparison results of our proposed method and state-of- -het---rr methods on the MOSI, MOSEI, and CH-SIMS, respectively. As shown in the Table 1, the proposed ALMT obtained tfae-offthe-un performance in almost all metrics. On the task more difficult and fine- grained sentiment classification (Acc-7), our model achieves remarkable improvements. For example, on the MOSI dataset, ALMT achieved a relative improvement of 1.69% compared to the second- best result obtained by CHFN. It demonstrates that eliminating the redundancy of auxiliary modalities essential for effective MSA.
4.2 Evaluation Criteria
Following prior works (Yu et al., 2020), used several evaluation metries, i.e., binary classification accuracy (Acc-2), F1, three classification accuracy
http://cchhiaar///////////aaaaa results/result-stat
Moreover, it is worth noting that the scenarios in

---

SIMS are more complex than MOSI and MOSEI, Therefore, it is more challenging to model the mul- timodal data. However, as shown in the Table 2, ALMT achieved state-of-the-ar performance in all metrics compared to the sub-optimal approach. For example, compared to Self-MM, it achieved rela- tive improvements with 1 1.44% on Acc-2 and 1.40% on the corresponding F1, respectively. Achieving such superior performance on SIMS with more complex scenarios demonstrates ALMT's ability to extract effective sentiment information from var- ious scenarios.
feature concatenation) greadl decreases the perfor- mance, demonstrating the language-guide hyper- modality representation learning strategy is effec- tive. Moreover, after the removal of the fusion Transformer and Modality Embedding, the perfor- mance drops again, also supporting that the fusion Transformer and Modality embedding can effec- tively improve the ALMT's ability explore the sentiment information in each modality.
Table 4: Effects of different components. Note: the best result is highlighted in bold.
4.5
Ablation Study and Analysis
Method
MOSI
CH-SIMS
Acc-7
MAE
Acc-5
MAE
4.5.1
Effects of Different Modalities
ALMT
49.42
0.683
45.73
0,404
To better understand the influence of each modal-
w/o AHL
34.40
0.952
38.29
0.444
w/o Fusion Transforner
48. 38.69
0.703
43.76
0.410
ity in the proposed ALMT, Table 3 reports the ab- lation results of the subtraction of each modality to the ALMT on the MOSI and CH-SIMS datasets, respectively. It is shown that, if the AHL is re- moved based on the subtraction of cach modality, the performance decreases significantly in all met- rics. This phenomenon demonstrates that AHL is beneficial in reducing the sentinent-irrelevantt redundancy of visual and audio modalities, thus improving the robustness of MSA.
w/ Modality Embedding
47.96
0.701
43.11
0.429
4.5.3 Effects of Different Query, Key, and Value Settings in Fusion Transformer
Table 5 presents the experimental results of dif- ferent query, key, and value settings in Transformer on the MOSI and MOSEI datasets, respectively. We observed that ALMT can obtain better perfor- mance when aligning hyper-modality features language features (i.e., using H& as query and us- ing hyper as key and value). We attribute this phenomenon to the fact that language information is relatively clean and can provide more sentiment- relevant information for effective MSA.
In addition, we note that after removing the video and audio inputs, the performance of ALMT remains relatively high. Therefore, in the MSA task, we argue that eliminating the sentiment- irrelevant information that appears in auxiliary modalities (i.e., visual and audio modalities) and improving the contribution of auxiliary modalities in performance should be paid more attention to.
Table 5: Effect of different Query, Key, and Value set- tings in Fusion Transformer.
MOSI
CH-SIMS
Table 3: Effects different modalities. Note: the best result highlighted in bold.
Q
K & V
Acc-7
MAE
Acc-5
MAE
H3
H3
48.10
0.707
44.64
0.410
hyper H&
Method
MOSI
CH-SIMS
Hayper
H
49.42
0.683
45.73
0.404
Acc-7
MAE
Acc-5
MAE
ALMT
49.42
0.683
45.73
0.404
w/o Audio
48.69
0.705
45.08
0.416
Video
47.96
0.704
44.64
0.403
4.5.4
Effects of the Guidance of Different Language Features in AHI
w/o Audio AHL
46.91
0.724
43.54
0.407
w/ Video AHL
47.08
0.726
43.76
0.406
To discuss the effect of the guidance of different language features in AHL, we show the ablation result of different guidance settings on MOSI and CH-SIMS in Table 6. In practice, replace the AHL layer that do not require language guidance with MLP layer. Obviously, we can see that the ALMT can obtain the best performance when all scals of language features (i.e., HH. HJ, H3) in- volve the guidance of hyper-modality learning.
w/o Video Audio
46.79
0.752
40.26
0.405
4.5.2 Effects of Different Components
To verify the effectiveness of each component of our ALMT, in Table 4, we present the ablation result of the subtraction of each component on the MOSI and CH-SIMS datasets, respectively. We observe that deactivating the AHL (replaced with

---

In addition, we found that the model is more difficult converge when AHL is removed. indicates that sentiment-irrlevant and conflicting information visual and audio modalities may limit the improvement of the model.
computational cost (2.50M). shows that ALMT achieves better trade-off between accuracy and computational burden.
Table 8: Analysis on model complexity. Note: the parameter of other Transformer-basee methods was cal- culated by authors from open source code with default hyper-parameters MOSI.
Table 6: Effects of different guidance of different lan- guage features AHI.. Note: the best result high-
lighted in bold.
Method
Parameter
Acc-7 on MOSI
MOSI
CH-SIMS
H²
MuLT
2.57M
40.00
Acc-7
MAE
Acc-5
MAE
MISA
3.10M
42.3
34.40
0.952
38.29
0.444
MAG-BERT
1.22M
43.62
47.38
0.704
43.54
0.412
48.10
0.709
43.11
0.415
ALMT
2.50M
49.42
48.54
0.711
43.98
0.412
46.36
0.736
45.51
0.417
48.10
0.707
44.20
0.409
47.81
0.729
43.76
0.416
4.5.7
Visualization of Attention in AHL
49.42
0.683
45.73
0.404
In Figure 4, present the average attention matrix (i.e., and 3) on CH-SIMS. As shown, ALMT pays more attention to the visual modal- ity, indicating that the visual modality provides more complementary information than the audio modality. In addition, from Table 3, compared to removing audio input, the performance of ALMT decreases more obviously when the video input is removed. It also demonstrates that visual modality may provide more complementary information.
4.5.5 Effects of Different Fusion Techniques To analyze the effects of different fusion tech- niques, conducted some experiments, whose results are shown in the table 7. Obviously, the MOSI dataset, the use of our Cross-modality Fusion Transformer to fuse language features and hyper-modality features the most effective. On the CH-SIMS dataset, although TFN achieves bet- ter performance on the MAE metric, its Acc-5 is lower. Overall, using Transformer for feature fu- sion is an effective way.
language features cash frame
languaga fastures of each frama
Table 7: Effects of different fusion techniques.
MOSI
CH-SIMS
Fusion Technique
Acc-7
MAE
Acc-5
MAE
low
audio features of each frame
visuel features of each frame
Concatenation
48.69
0.703
43.76
0.410
Addition
46.36
0.706
42.45
0.411
GRU
47.81
0.710
44.86
0.414
Figure 4: Visualization of average attention weights from the last AHI layer on CH-SIMS dataset. (a) Av- erage attention matrix between language and audio modalities; (b) average attention matrix 3 between lan- guage and visual modalities. Note: darker colors indi- cate higher attention weights for learning.
Tensor Fusion (TFN)
47.23
0.710
44.20
0.403
Low-rank Fusion (LMF)
46.65
0.715
45.08
0.408
Ours
49.42
0.683
45.73
0.404
4.5.6
Analysis on Model Complexity As shown in Table 8, we compare the pa- rameters of ALMT with other state-of-the-ar art Transformer-basee methods. Due to the different hyper-parameter configurations for each dataset may lead to slight difference in the number of parameters calculated. We calculated the model parameters under the hyper-paramedt settings on the MOSI. Obviously, our ALMT obtains the best performance (Acc-7 49.42 %) with a second
4.5.8 Visualization of Robustness of AHL To test the AHL's ability to perceive sentiment- irrelevant information, shown in Figure 5, we visualize the attention weights (3) of the last AHL layer between language features (IIf) and visual features (H.)) on CH-SIMS. More specifically, we first randomly selected a sample from the test set. Then we added random noise to a peak frame

---

(marked by the black dashed boxes) of Hy and finally observed the change of attention weights between H and H seen that when the ran- dom noise added to the peak frame, the atten- tion weights between language and the correspond- ing peak frame show remarkable decrease. This phenomenon demonstrates that AHL can suppress cettiment-irrelevane information, thus obtaining more robust hyper-modality representation for mul- timodal fusion.
H
H
noisy frame
30
20
10
-10
-20
language features of each frame
high
-30
larguage of eash
Figure 6: Visualization of different representations in 3D space by using t-SNE.
Self-MM converges the fastest, its MAE of conver- gence is larger than ALMT at the end of the epoch. On the validation set, ALMT seems compared other methods, while the curves of other methods show relatively more dramatic fluc- tuations. It demonstrates that the ALMT is easier to train and has better generalization capability.
stable
(a)
(b)
Figure 5: Visualization of the attention weights between language and visual modalities learned by the AHL for randomly selected sample with and without random noise on the CH-SIMS dataset. (a) The attention weights without a random noise: (b) the attention weights with random noise. Note: darker colors indicate higher attention weights for learning.
2.90
MAE
0.3
0.2
4.5.9 Visualization of Different Representations
0.1
12
15
25
25
30
70
15
20
75
epoch
epoch
In Figure 6, we visualized the hyper-modality representation Haper"" visual representation Hy and audio representation HJ in a 3D feature space by using t-SNE (Van der Maaten and Hinton, 2008) on CH-SIMS. Obviously, there is modality dis- tribution gap existing between audio and visual features, well as within their respective modali- ties. However, the hyper-modality representations learned from audio and visual features converge in the same distribution, indicating that the AHL can narrow the difference inter-/intra modality dis- tribution of audio and visual representations, thus reducing the difficulty of multimodal fusion.
(6) on Training
MAT on Vilidation Set
Figure 7: Visualization of convergence performance on train and validation sets CH-SIMS. (a) The comparison of MAE curves the training set; (b) the comparison of MAE curves on the validation set. Note: the results of other methods reproduced authors from open source code with default
Conclusion
In this paper, a novel Adaptive Language-guided Multimodal Transformer (ALMT) is proposed to better model sentiment cues for robust Multimodal Sentiment Analysis (MSA). Due to effectively sup- pressing the adverse effects of redundant informa- tion in visual and audio modalities, the proposed method achieved highly improved performance on several popular datasets. We further present rich in- depth studies investigating the reasons behind the effectiveness, which may potentially advise other researchers to better handle MSA-related tasks.
4.5.10 Visualization of Convergence Performance
In Figure 7, compared the convergence hehavior of ALMT with three state-of-the-ar meth- ods (i.e., MulT, MISA and Self-MM) CH-SIMS. We choose the MAE curve for comparison as MAE indicates the model's ability to predict fine-grained sentiment. Obviously, on the training set, although

---

Limitations
Devamanyu Hazarika, Roger Zimmermann, and Sou- janya Poria. 2020. MISA: modality-invariant invariant and -specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM interna- tional conference on multimedia, pages 112--1131. ACM.
Our AMLT which is Transformer- model usually has large number of parameters. It re- quires comprehensive training and thus can be sub- jected to the size of the training datasets. As current sentiment datasets are typically small in size, the performance of AMLT may be limited. For ex- ample, compared to classification metrics, such as Acc-7 and Acc-2, the more fine- fine-grained regression metrics (i.e., MAE and Corr) may need more data for training, resulting in relatively small improve- ments compared to other advanced methods.
Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, and Mingyue Niu. 2020. Multimodal transformer fu- sion for continuous emotion recogmition. 2020 IEEE International Conference on Acoustics, Speech and Signal Processin((CAASSP), pages 3507-3511. IEFF.
Yingying Jiang. Wei Li, M. Shamim Hossain, Min Chen, Abdulhameed Alelaiwi, and Muneer Al-Hammadi. 2020. A snapshot research and implementation of multimodal information fusion for data-driven emo- tion recognition. Information Fusion, 53:209- 221.
References
Jacob Devlin Ming-Woi Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLI, pages 1171-4186.
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Phillips' Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. 13th IEEE International Conference on Automatic Face & Gesture Recogni- tion, FG 2018, Xi' an, China, May 15-19, 2018, pages 59-66. IEEE Computer Society.
Yuanyuan Liu, Wenbin Wang, Chuanxu Feng, Haoyu Zhang, Zhe Chen, and Yibing Zhan. 2023a. Ex- pression snippet transformer for robust video- based facial expression recognition. Pattern Recognition, 188:609368.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Proceedings of the 16th Euro- pean Conference on Computer Vision, volume 12346. pages 213-229
Yuanyuan Liu, Haoyu Zhang, Yibing Zhan, Zijing Chen, Guanghao Yin, Lin Wei, and Zhe Chen. 2023b. roist-resistant multimodal transformer for emotion recognition, arXiv preprint arXiv:2305.02814.
Weidong Chen, Xiaofen Xing. Xiangmin Xu, Jianxin Pang, and Lan Du. 2022. Specehformer: A hicrarchi- cal efficient framework incorporating the character- istics of speech. In Proceedings of the 23rd Annual Conference of the International Speech Communica- tion Association, pages 346-350.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshmi- narasimhan, Paul Pu Liang, AmirAli Bagher Zadch, and Louis-Philippe Morency. 2018 Efficient low- rank multimodal fusion with modality-specii tors. In Proceedings of the 56th Anmual Meeting of the Association for Computtaional Linguistics ( (Vol- ume I: Long Pupers), pages 2247-2256 Melbourne, Australia. Association for Computational Linguistics.
fac-
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR.
Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. 2021. Progressive modality rein- forcement for human multimodal emotion recogni- tion from unaligned multimodal sequences. In 2021 IEEF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 2554-2562 Com- puter Vision Foundation IEEE.
Jiwei Guo, Jiajia Tang, Weichen Dai, Yu Ding, and Wanzeng Kong. 2022. Dynamically adjust word rep- resentations using unaligned multimodal information. In Proceedings of the 30th ACM International Con- ference on Multimedia, MM '22, page 3394-3402. Association for Computing Machinery.
Huisheng Mao, Ziqi Yuan, Hua Xu, Wenmeng Yu, Yihe Liu, and Kai Gao. 2022. M-sena: An integrated plat- form for multimodal sentiment analysis. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pages
Wei Han, Hui Chen, and Soujanya Poria. 2021. Im- proving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 91 80-9192, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th Python in Science Conference 2015 (SciPy 2015), Austin, Texas, July 6 - 12, 2015, pages 18-24, scipy.org.

---

Yongfeng Qian, Yin Zhang, Xiao Ma, Han Yu, and Limei Peng. 2019. EARS: emotion-aware recom- mender system based on hybrid information fusion. Information Fusion, 46: 141-1416.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cam- bria, and Louis-Phillipe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 100-11114, Copenhagen. Denmark. Association for Computational Linguistics.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. Integrating multi- modal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, page 2359. NIH Public Access.
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cam bria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2236-2246
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Moreney, and Rus- lan Salakhutdinov. 2019a. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6558- 6569, Florence, Italy. Association for Computational Linguistics.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis- Philippe Morency. 2016. Multimodal sentiment in- tensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82-44 88.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019b. Learning factorized multimodal representa- tions. ICLR.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances Neural Information Pro- cessing Systems, volume 30, pages 5998-6098 6008.
Dingkang Yang, Shuai Huang, Haopeng Kuang, Yang- tao Du, and Lihua Zhang. 2022. Disentangled repre- sentation learning for multimodal emotion recogni- tion. In Proceedings the 30th ACM International Conference on Multimedia, pages 1642-165) 16511 ACM.
Wenmeng Yu, Hua Xu, Fanyang Meng. Yilin Zhu, Yixiao Ma. Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. CH-SIMS: A chinese multimodal senti- ment analysis dataset with fine grained annotation of modality. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, 2020, Online, July 5-10, 2020, pages 3718-3727. Association for Computational Linguistics.
Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning moddalitysppecific representations with self- supervised multi-task learning for multimodal sen- timent analysis. In Proceedings of the AAA! Con- ference on Artificial Intelligence, volume 35, pages 10790-10797.
Ziqi Yuan. Wei Li, Hua Xu, and Wenmeng Yu. 2021. Transformer- hased feature reconstruction network for robust multimodal sentiment analysis. In Proceed- ings of the 29th ACM International Conference on Multimedia, pages 4400-4407. ACM.

---

A Hyper-parameters
0.05
0.38
In this section, we show the selection of some key hyyper-paameeee the validation set of CH- SIMS.
4000
4.45 43.9
3.000
C412)
2.433
0410
0.30
A.1 Overview
Toos lorth modaltt
[8]
We used PyTorch to implement our method. The experiments were conducted on PC with Intel(R) Xeon(R) 6240C CPU 2.6GHz and 128GB mem- ory and NVIDIA GeForce RTX 3090. The key parameters are shown in Table 9. We see that most hyper-parameters are the same across these datasets, demonstrating the ALMT does not require complex
Figure 8: Effects of Token Length Settings in Modal- ity Embedding. (a) Accuracy curve of different Token sequence lengths on the CH-SIMS; (b) MAE curve of different Token sequence lengths on the CH-SIMS
0435
2.022
adjustment.
21-42.01
(1 (40430
Table 9: Haper-parameters of ALMT we on the different datasets
2.42.13
(A 1.
0430
(LE
0.40
MOSI
MOSEI
CH-SIMS
Modality Feature Length T
8
8
8
Vector Dimension
128
128
128
Figure 9: Effects of Depth Settings of AHI. (a) Accu- racy curve of different AHL depth on the CH-SIMS; (b) MAE curve of different AHL depth on the CH-SIMS
Modality Embedding Depth
AllI Depth
3
Fusion Transformrr Depth
Batch Size
64
64
64
Initial Learning Rate
1c-4
Ic-4
Ic-4
Optimizer
Adam
Adam
Adam
Epochs
200
200
200
A.4 Effects of Depth Settings of Fusion Transformer
Warm Up
V
V
V
Cosine Annealing
In Figure 10, we presents the effects of depth settings of cross cross-modality fusion Transformer on CH-SIMS. We observed that the ALMT can obtain the best result Acc-5 and MAE when the depth 3 and 5, respectively. However, to balance performance and model computation, we set the depth to 4 on the CH-SIMS. Following the similar rule, we set the depth of the mross-modality fusion Transformer to 2 and 4 on the MOSI and MOSEI, respectively.
A.2 Effects of Length Settings of Modality Feature
In Figure 8, we show the effect of the sequence length T of the token Ho in modality embed- ding the CH-SIMS dataset. It is observed that there are significant performance changes when the hyper-parameter is changed. And a similar phenomenon occurred on the MOSI and MOSEI datasets. Although the MAE is not the best when the Ace-5 highest, e.g., T is set to 32. Consid- ering that the model computation rises when the T increases, we set T to 8, which is beneficial for ALMT to obtain the best performance with relatively lower computational cost.
6440
12.
24422
12.42.00
14.4
(4
(3.40.20 42121
2412
2.45
Tondsrmer
Depth coppaorrssoi faoo Tondomee
A.3 Effects of Depth Settings AHL
Figure 9 presents the effect of AHL depth set- tings for MSA. Obviously, the ALMT achieves the best performance the most difficult evalua- tion metrics, i.e., Acc-5 and MAE. Hence, in this study, we set the depth of AHL to Moreover, on the MOSI and MOSEI datasets, we set it to as the similar phenomenon is also observed.
Figure 10: Effects of Depth Settings Cross-modality Fusion Transformer. (a) Accuracy curve of different depth settings on the CH-SIMS: (b) MAE curve of dif- ferent depth settings the CH-SIMS


Reviews:
[{"review_id": "CXDSd1NENn", "reviewer": "Reviewer_JSNF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer JSNF\n\nWe sincerely appreciate your careful and thoughtful comments. We have made every effort to address all the concerns raised. Below, we will discuss the issues. \n\nDue to the ability to efficiently model long time sequences and its multi-head attention mechanism, Transformer has demonstrated more powerful modeling capabilities than other methods (*e.g.,* LSTM and GRU) in many tasks of multimodal learning. However, it is worth noting that most of the previous studies in the MSA (*e.g.,* MISA and MulT) have used an equal perspective to treat each modality, directly fusing the features of each modality as inputs to Transformer.  We observed that the contribution of each modality to the recognition accuracy in MSA datasets is significantly different, and we believe that the above approach will limit the modeling ability of Transformer in MSA tasks. Therefore, we sought to design a dominant modality (language modality) guided sentiment representation learning model (ALMT) based on Transformer. **Although Transformer has been already very common in the field of multimodal learning, our novelty is in designing a dominant modality-guided learning paradigm for the first time. In experiments, aside from remarkable performance, we also provide rich analysis and evidence supporting the significance of the proposed method.**\n\nIn addition, we found some interesting phenomena. We have tried to use different methods instead of AHL for representation learning. Experimental results are shown in the table below. We found that AHL based on Transformer design achieved the best performance, followed by MLP, while LSTM and GRU showed lower performance. If time and space permit, we will further analyze why LSTM and its variant networks perform so poorly in the final version.\n\n|Hyper-modality Learning  Method|Acc-7 on MOSI|MAE on MOSI|Acc-5 on CH-SIMS|MAE on CH-SIMS|\n|----------------------|:----------------------:|:----------------------:|:----------------------:|:----------------------:|\n|MLP (w/o language guidance)|34.40| 0.952       |38.29| 0.444          |\n|LSTM  (w/o language guidance)| 21.14         | 1.383       | 26.26            | 0.571          |\n|BiLSTM  (w/o language guidance)| 21.87 | 1.382       | 29.32            | 0.529          |\n|GRU  (w/o language guidance)| 22.16 | 1.377       | 28.67            | 0.529          |\n|**AHL**| **49.42**     | **0.683**   | **45.73**        | **0.404**      |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work presents a novel multimodal sentiment analysis method, namely Adaptive Language-guided Multimodal Transformer (ALMT), which for the first time explicitly tackles the adverse effects of redundant and conflicting information in auxiliary modalities (i.e., visual and audio modalities), achieving a more robust sentiment understanding performance.\n- This work devises a novel Adaptive Hyper-modality Learning (AHL) module for representation learning. The AHL uses different scales of language features to guide the visual and audio modalities to form a hyper modality that complements the language modality.\n- State-of-the-art performance and detailed analysis in several public and widely are provided.", "reasons_to_reject": "- Novelty limitations. This is an incremental job. It is common to use Transformers to fuse multiple modalities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes Adaptive Language-guided Multimodal Transformer (ALMT) to better model sentiment cues for robust Multimodal Sentiment Analysis (MSA). ALMT consists of three major components, i.e., modality embedding, adaptive hyper-modality learning, and multimodal fusion. Due to effectively suppressing the adverse effects of redundant information in visual and audio modalities, the proposed method achieved highly improved performance on several popular datasets. Detail experiments and analyses are provided to prove the effectiveness of the ALMT.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gc9gy0Xnp2", "reviewer": "Reviewer_WX81", "summary": "", "questions": "Can the authors provide more details on how the Adaptive Hyper-modality Learning (AHL) module works, specifically how it suppresses sentiment-irrelevant information? Is there any theoretical justification for the chosen architecture, especially the design of AHL in the proposed method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer WX81\nWe are thankful to you for your time and effort in reviewing the paper. We also appreciate all the comments and suggestions. \n\n**Regarding the reasons to reject, we will clarify it in 4 points below.**\n\n1. **Since CHFN and FDMER do not have open-source codes, we could not perform significance tests directly. If possible in the future, we will add the relevant results to the final version.** Moreover, we selected several SOTA methods (*e.g.,* MMIM and Self-MM) that have open-source code to conduct significance tests (Analysis of Variance, ANOVA) to validate the statistical significance of the observed improvements. The experimental result is shown in the following table. Obviously, our method is statistically more significant in terms of performance improvement because of the low p-value, which indicates a significant improvement in the prediction results of ALMT compared to other methods.\n||p-value|\n|----------------------|----------------------|\n|ALMT & MMIM|0.027|\n|ALMT & Self-MM|0.116|\n2. We emphasize that ALMT improves on almost all metrics in all datasets, while in some settings the improvement is quite significant. This serves to demonstrate the effectiveness and generalizability of our approach.\n3. It can be seen that ALMT improves significantly on the Acc-7 and Acc-5 of MOSI, which also shows the effectiveness of our method. Regarding the reason why there is not much improvement on some metrics such as Acc-2, we believe that it is because these metrics are less difficult. After years of research, existing methods may tend to approach the performance boundary on these simple metrics.\n4. The MOSI dataset is small in size (only 1284 samples in the training set), and we found that ALMT is always quickly overfitted during training. As well acknowledged, the Transformer-based model usually requires a large amount of data for training in order to realize its full performance. Therefore, we believe this is also one potential factor that limits the performance improvement of ALMT.\n\n**Regarding the question for the authors, the details about the AHL will be clarified in 4 points.**\n\n1. As shown in Figure 3 of the paper, AHL uses the dominant modality (language) as Query, and the audio and visual modalities as Key and Value. In this way, AHL is able to query the audio and visual modalities for potentially sentiment-relevant and complementary information to the language, respectively, where the Transformer's multi-attention plays a crucial role. For useful information, the attention mechanism will give higher weight scores to highlight this information, while for redundant information (*e.g.,* noise), the attention mechanism will give lower weight scores to suppress this information.\n2. In Section 4.5.6 of the paper, we visualize the AHL weights and it can be seen that on the CH-SIMS dataset, the model is guided by language with higher attention against the video information. It indicates that the visual modality provides more valid information than the audio modality on this dataset. This is further verified in the experiments in Section 4.5.1, where Acc-5 and MAE drop more drastically when removing the visual modality compared to removing the audio modality.\n3. In Section C of the Appendix, we first randomly selected a sample and added noise, and then observed the changes in the weights computed by AHL. We see that for frames with added noise, the weights given by AHL are smaller, implying that AHL is able to suppress redundant information that is not related to sentiment or does not complement the language.\n4. Regarding the choice of architecture, based on the phenomenon we observed and our motivations, we empirically believe that building ALMT/AHL based on Transformer is more appropriate. In addition, we have also done some experiments, and the experimental results show that Transformer-based architecture can better realize our idea and achieve better performance. For more details, please refer to Response to Reviewer JSNF and Reviewer PJLs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper presents a novel approach to tackle the issue of sentiment-irrelevant and conflicting information across different modalities in MSA, which is a significant contribution to the field.\n2) The proposed model, ALMT, incorporates an Adaptive Hyper-modality Learning (AHL) module, which is a novel concept that could potentially inspire future research in this area.\n3) The paper provides empirical evidence of the model's effectiveness by demonstrating state-of-the-art performance on several popular datasets.", "reasons_to_reject": "It appears that the improvement of ALMT over the CHFN method on the MOSI dataset is quite marginal. It would be beneficial if the authors could conduct significance tests to validate the statistical significance of the observed improvements.", "questions_to_authors": "Can the authors provide more details on how the Adaptive Hyper-modality Learning (AHL) module works, specifically how it suppresses sentiment-irrelevant information? Is there any theoretical justification for the chosen architecture, especially the design of AHL in the proposed method?", "paper_topic_and_main_contributions": "This paper presents a novel approach to Multimodal Sentiment Analysis (MSA) by introducing the Adaptive Language-guided Multimodal Transformer (ALMT). The primary problem addressed by this paper is the potential sentiment-irrelevant and conflicting information across different modalities (language, video, and audio) that may hinder the performance of MSA. The authors propose an Adaptive Hyper-modality Learning (AHL) module that learns an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. The paper claims that the proposed model achieves state-of-the-art performance on several popular datasets such as MOSI, MOSEI, and CH-SIMS.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sPeIAS5MuS", "reviewer": "Reviewer_PJLs", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer PJLs\n\nWe sincerely thank you for your kind and constructive comments. Regarding the questions, we have the following answers.  We will try our best to incorporate the suggestions in the revised version of the paper.\n\nRegarding the comparison of different fusion techniques, we conducted some supplementary experiments, whose results are shown in the following table. Note that on the MOSI dataset, the use of Cross-modality Fusion Transformer to fuse language features and hyper-modality features is the most effective. On the CH-SIMS dataset, although TFN achieves better performance on the MAE metric, its Acc-5 is lower. Overall, using Transformer for feature fusion is an effective way.\n\n|Fusion Method|Acc-7 on MOSI|MAE on MOSI|Acc-5 on CH-SIMS|MAE on CH-SIMS|\n|----------------------|:----------------------:|:----------------------:|:----------------------:|:----------------------:|\n|Feature Concatenation|48.69|0.703|43.76|0.410|\n|Feature Addition|46.36|0.706|42.45|0.411|\n|LSTM|47.67|0.700|23.63|0.554|\n|BiLSTM|47.52|0.707|26.26|0.540|\n|GRU|47.81|0.710|44.86|0.414|\n|Tensor Fusion (TFN)|47.23|0.710|44.20|**0.403**|\n|Low-rank Multimodal Fusion (LMF)|46.65|0.715|45.08|0.408|\n|**Cross-modality Fusion Transformer**|**49.42**|**0.683**|**45.73**|0.404|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is written well.\n2. Explored ALMT with AHL module which is quite interesting.\n3. Ablation study is good.", "reasons_to_reject": "1. The approach looks fine, but only one minor thing is it is good to share the results different fusion techniques as well.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about Adaptive Language-guided Multimodal Transformer, which incorporates an Adaptive Hyper-modality Learning module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales for multimodal sentiment analysis task. The main contributions of the paper are: Proposed a novel multimodal sentiment analysis method, namely Adaptive Language-guided Multimodal Transformer, and explored a novel Adaptive Hyper-modality Learning module for representation learning. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers rated the work as 4, across all categories. 

Reviewers largely agree that:
-The paper is well written
-Experiments were interesting and novel
-The work achieves state-of-the-art performance 

Some reviewers also noted that the gap between the prior state-of-the-art and the SotA that the authors achieve "seem[s] marginal".
Decision: Accept-Main