Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting
Authors: Haoyu Zhang
Yinan Zhang
Chaolong Ying
Xiaoying Tang
Tianshu Yu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Sentiment Analysis
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Multimodal Sentiment Analysis (MSA) aims to predict sentiment from diverse data types, such as video, audio, and language. Recent progress in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance across various tasks. However, in MSA, the increase in computational costs does not always correspond to a significant improvement in performance, raising concerns about the cost-effectiveness of applying MLLMs to MSA. This paper introduces the MLLM-Guided Multimodal Sentiment Learning Framework (MMSLF). It improves the performance of task-specific MSA models by leveraging the generalized knowledge of MLLMs through a teacher-student framework, rather than directly using MLLMs for sentiment prediction. First, the proposed teacher built upon a powerful MLLM (e.g., GPT-4o-mini), guides the student model to align multimodal representations through MLLM-generated context-aware prompts. Then, knowledge distillation enables the student to mimic the teacher’s predictions, thus allowing it to predict sentiment independently without relying on the context-aware prompts. Extensive experiments on the SIMS, MOSI, and MOSEI datasets demonstrate that our framework enables task-specific models to achieve state-of-the-art performance across most metrics. This also provides new insights into the application of general MLLMs for improving MSA.

Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting
Haoyu Zhang', Yinan Zhang', Chaolong Ying', Xiaoying Tang', Tianshu Yu**
SScooo of Data Science, The Chinese University of Hong Kong, Shenzhen School of Science and Engineering, The Chiese University of Hong Kong, Shenzhen (haoyuzhang3, y yinanzhangg,, chaolongying)@ll Hlikk ink........ (tangxiaoying yutianshu)@cuhi edu. cn
cuhk. edu.
Abstract
Multimodal Sentiment Analysis (MSA) aims predict sentiment from diverse data types, such as video, audio, and language. Recent progress in Multimodal Large Language Models (MLLMss) have demonstrated impressive performance across ious tasks. However, in MSA, the increase in computational costs does not always correspond to significant improvement in performance, raising concerns about the cost-effectivenees of applying MLLMs to MSA. This paper introduces the MLLM- Guided Multimodal Sentiment Learning Framework (MMSLF). It improves the performance of task-specific MSA models by leveraging the generalized knowl- edge MLLMs through a teacher-student framework, rather than directly using MLLMs for sentiment prediction. First, the proposed teacher built upon a powerful MILM (e.g., GPT-4o-mini), guides the student model align multimodal repre- sentations through M.L.M--enerated context-aware prompts. Then, knowledge distillation enables the student to mimic the teacher's predictions, thus allowing it to predict sentiment independently without relying on the context-aware prompts. Extensive experiments on the SIMS, MOSI, and MOSEI datasets demonstrate that our framework enables task-specific models to achieve fate--of-thearl performance across most metrics. This also provides new insights into the application general MLLMs for improving MSA.
1 Introduction
Multimodal Sentiment Analysis (MSA) aims to predict sentiment from various types of input, such as language, video, and audio. Accurate MSA crucial for several applications, such as Human- Computer Interaction and Hcalthcare [1, 2]. Compared to unimodal sentiment analysis, the mutually complementary nature of multiple modalities typically leads to better performance, thereby improving the applicability of MSA in real-world scenarios.
series of studies focused on improving MSA through well-designed representational learning and multimodal fusion networks. For example, Tsai et al. [3] introduces a novel model which employs multiple Transformers for pairwise alignment of modality information. Hazarika al. [4] propose method to disentangle each modality into modality-invarie invariant and modality-specifie features for multi-perspectives fusion. Additionally, Yu et al. [5] apply self-supervised learning to generate pseudo-labed for each modality to learn both modality consistency and inconsistency. Zhang et al. [6] make language modality as dominant modality to guide the learning of representations in other modalities, thus mitigating potential conflicts between different modalities. After years of exploration,
"Correspondence author 'Code: https://githut - CU0.0-CCKKZ/MMLLL
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

it has become increasingly challenging to achieve performance improvement in MSA. Fortunately, recent multimodal large language models (MLLMS) have demonstrated notable performance for various specific tasks [7-11, 2]. For example, Lian et al. [2] explores the application of GPT-4V [12] for MSA, showing that MI.LMs without finetuning can achieve performance comparable to many task-specific models through their general knowledge. However, the parameters of task-specific models mostly range from several million to tens of millions. Compared to these task-specific models, the increased parameter count and computational costs of general MLLMs does not always lead to a significant improvement in performance, raising concerns about the MLLMs' cost-effectiveness. This inspired us to explore whether it is possible to apply general MLLMs knowledge to assist in the training of sask-specific MSA models, thus achieving better MSA.
In this paper, we aim to bridge the gap between task-specific models and MLLMs in MSA by leveraging the generalized knowledge of MI.LMs to help with training task-speeific models. To this end, we introduce the MLLM- Guided Multimodal Sentiment Learning Framework (MMSLF), which embeds an MILMM within the teacher network to provide enhanced supervision for the task task-specific student model, thereby avoiding the direct use of the MLLMs for sentiment prediction. In the teacher network, we use a pre-trained MLLM (e.g., GPT-4o-mini [12]) to generate context-aware prompts that highlight key sentiment cues across different modalities. These prompts guide the model to learn conditional attention maps in specially designed alignment modules, helping it better capture sentiment information. The student network is a task- specifpe model that learns from the guidance of the teacher. It receives the same multimodal inputs but does not use prompts from MLLMs. Instead, it aligns the sentiment information from conditional attention and features Icarned by the teacher to improve performance of sentiment analysis. Extensive experiments on popular datasets, such as SIMS [13], MOSI [14], and MOSEI [15] demonstrate the effectiveness of MMSLF, showing its state-of-the-art performance. In summary, our work makes the following contributions, which introduce a novel solution to the challenges in MSA:
We explore using the general knowledge of MLLMs to guide the training of task-specific MSA models, offering new insight into applying general MLLMs improve MSA.
We design conditional alignment mechanism that enables the teacher model with MILM's knowledge to intuitively and efficiently guide the student model's multimodal alignment and representation learning.
Extensive comparisons and ablation studies on three popular datasets (e.g. SIMS, MOSI, and MOSEI) demonstrate that the proposed MMSLF can improve the training process of task-specific models, enabling them to achieve state-of-the-art performance across most metrics.
2 Related Work
2.1 Multimodal Sentiment Analysis
Multimodal Sentiment Analysis (MSA) aims to predict human sentiment by leveraging various types of data, such as video, audio, and text. Early methods, such as TFN [16] and LMF [17]. achieved state-of- of-tho---te-art performance by capturing relationships between modalities through Cartesian product- based tensor fusion. However, these methods face the challenge of rapidly increasing computational costs the feature dimensions and the number of modalities grow. With the advent of deep learning architectures, the attention mechanism has become popular in the design of MSA methods [3, 18, 4, 19-22, 6). For example, MulT [3] employs multi-head attention to align modalities, facilitating more effective multimodal fusion. ALMT [6] leverages language representations at different scales to guide the learning of other auxiliary modalities, mitigating the influence of noise that can negatively impact fusion. In addition, various other novel methods [23, 5, 24] have also made significant progress in the MSA. For example, Yu al. [5] proposed generating uni-modal sentiment labels to help the model capture both consistency and differentiation across modalities. Moreover, Yuan et al. [24] introduced an adversarial training strategy based on semantic reconstruction using original- -oisy instance pairs, achieving robust MSA in simulated noisy scenarios. Despite these progress, achieving further improvements in performance remains challenging. A recent study [2] explored the application of GPT-4V in MSA, demonstrating that MILMSs can achieve performance comparable to small-scale models. Different from this work, our work utilizes MI.MM to help the learning of task-specific models rather than directly using MLLMs for MSA.

---

2.2 Large Language Models
In recent years, large language models (LLMs) have made remarkable strides, with models such as GPT-3 [25], T5 [26], and LLaMA [27] demonstrating impressive capabilities by scaling both data and model sizes. However, despite these advances, uni-modal LLMs are limited to processing text-based information, restricting their applicability to broader range of tasks and scenarios. To overcome this limitation, researchers have explored the potential of multimodal large language models (MLLMs), building upon the foundation of uni-modal LLMs. Significant progress has been made in developing powerful MLLMs [28-31, 8-10, 32-35]. showcasing their surprising practical capabilities. For instance, GPT-4V [12] integrates natural language processing with visual understanding analyze images and provide textual responses to questions about them. Similarly, LLaVA [7] translates visual content into text by employing a linear layer to embed images, making the LLMs understand visual input. Video-LLaMA [8] achieving multimodal understanding by aggregating representations from different modalities after applying positional embedding through Q-formers [32]. Moreover, Zhao et al. [10] introduced MMICL, which leverages multimodal in-context learning to achieve state-of-te kate-offtheear performance on various visual language tasks, In this work, we utilize MLLMs to generate prompts for smaller task-specific models, enabling efficient multimodal learning.
2.3 Teacher-Student Models
The teacher-student framework has been widely applied in knowledge distillation, particularly for knowledge compression [36]. It focuses on transferring knowledge from a larger teacher model to smaller student model through carefully designed strategies, such as soft label matching [37-40] and feature matching [41-44]. For example, Hinton et al. [37] introduced the use of the teacher model's probability distribution as soft labels to guide the student model's learning process. By utilizing these soft labels, the student model is trained not only to predict the correct labels but also to closely align with the teacher model's 's soft predictions, thereby facilitating effective knowledge transfer. Additionally, Zagoruyko et al. [43] proposed an attention transfer method that improves the student model's performance by transferring activation-bast based and gradient-based attention maps from the teacher model. In the context of MSA, recent advancements include MC-Teacher [40], which introduced learnable pseudo-label selection and self- add-tivee exponential moving average strategies to achieve semi- supervipee MSA. In this work, we employ feature matching and attention transfer techniques to achieve our research objectives. To the best of our knowledge, this is the first attempt to transfer the general knowledge of MLLMs to smaller models for MSA.
3 Method
3.1
Overview
The overall pipeline of the MMSLF is illustrated in Figure 1. First, with the given preprocessed multimodal input sequences, each modality is processed through three embedding layers. Then, the extracted features are aligned using a designed Conditional Alignment module, where the condition provided by prompts from MILMs (e.g., GPT-4o-mini). Specifically, visual and audio features are aligned with language features via two alignment modules: Visual-to-Languaag (V L) Alignment and Audio-to- Languua (A + L) Alignment. These conditional alignment layers establish correspondences between modalities with the help of the MLLM's prompt, facilitating effective multimodal alignment. Finally, the multimodal fusion module combines the aligned features to produce unified representation, which is used to predict the final sentiment score via a regression Teacher (defined as Eq. 9).
loss L
Once the teacher is trained, the student is trained to mimic the behavior of the teacher. The key difference between the student and teacher is that the student align video and audio features with language features directly, without the conditional input (i.e., MLLM's Prompt) used in the teacher. Additionally, instead of using the regression loss of sentiment scores I SStudent (defined as Eq. 12), two regularization techniques are used to help the student learn from the teacher: (1) the student's 's attention maps are trained to match the teacher's conditional attention maps using an attention transfer loss Student (defined as Eq. 10), and (2) the fused unified representations of the student are encouraged to a
match those of the teacher through a unified representation matching loss Studett (defined as Eq. 11).
fusion These loss ensure that the model captures the same patterns as the teacher.
3

---

Prompt
gesture
Inpor
Visual Embedding
K.V
Qp
Xy
Conekonal Algnment (V sig
MMSLF Teacher
Language
Q
Multimodal Fusion
xentiment
Tonacher
Embedding
scure
Conditonal Aignment (bok)
XL.
Conditional Atvatton Mag
Audio
MyMatpa XA
Phutes
Embedding
K.V
Up
Weon
heywoord "Ue
Shunt
tasion
Juput
Prompt
W
Visual Embedding
K.V
Abontion
#
Xy
Alignment [V-S]]
Map
MMSLF Student
Language Embedding
Multimodal Fuston
sontiment
Suudent
Score
XL
Aligament [N-S]]
Audoo
Wwlhbellhoh XA
Audio Embeddlng
Figure 1: Overall pipeline of MMSLF. Note: L, A, and V refer to language, audio, and visual modalities, respectively.
3.2 Multimodal Input and Embedding
We utilize the preprocessed sequences in the datasets as inputs. Specifically, the language input processed using BERT [45]. while visual input is handled by OpenFace [46], and audio input is processed with Librosa [47]. We denote the multimodal input as Xm E R.T Xdm, where m E (I., A, V), Tm represents the length of the input sequence, and dm indicates the vector dimension.
Given the multimodal input Xm. we apply three embedding layers Em, each consisting lincar layer to extract features from each modality and map them into unified feature dimension d:
Sn = F. (x OF..) E TT Xd
where S. represents the embedded features of modality m, and OF.,, denotes the parameters associ- ated with each embedding layer.
3.3 Multimodal Alignment
Prompt Embedding, To extract features from the MLLMS prompt Xp and fix the feature dimension to d, we apply a pre-trained BERT along with an embedding layer (comprising Transformer encoder with a depth of two layers) to Xp. We denote the combined operation as Ep. The process can be described as:
Conditional Amsention
Wap
Sp = Ep(Xp: OE.)) E RIXxd
(2)
where Sp represents the embedded feature of the prompt, which has the same feature shape as SL, and Óg, denotes the parameters used in the MILMs, pre-trained BERT, and the embedding layer. In practice, for the V->L alignment, Xp con- tains the prompt information from both visual and language modalities. For the A->L alignment, Xp
121
Figure 2: An example of conditional attention used align modality 3 to modality a.

---

contains the prompt information from the audio and language modalities. However, since GPT-4o- mini does not support audio analysis, we only include language information with A->L alignment. We also experimented with generating prompts using Gemini-2.0-Flan which supports audio input, but its performance was not better than GPT-4o-mini. For detailed discussions, please refer to Section 4.7 and Section 4.8.
Conditional Attention. To introduce the general knowledge of MLLMs for assistance in model training, we added conditional inputs based on the multi-head attention mechanism. As illustrated in Figure 2, to align modality
modality a, the module first uses S. compute Query (Qa), while S3 is used to compute the Key (Kş) and Value (Vz). The relationshippaatention map Wa, between these two modalities is computed as follows:
Wa.3 Qak3 RTxTT
Vox
where dy denotes the dimension of each attention head, and Ta and T3 represent the sequence lengths of the corresponding modalities. Simultaneously, we apply the prompt Sp as conditional Query (Qp) to Ki and Va to compute a shifted attention map A E RR.x xTx Then, we obtained the conditional attention map W con by fusing W., S and A:
W con = softmax ( a. A) E RTTxT,
where the softmax represents weight normalization operation. Finally, the aligned feature H can be computed as follows:
Teacher 9-+a
Teacher GreddFForwadd(Me Vo: 0(wd) e R RTTxx
(5)
8-70
where Feed-Forward and Ored represent the MLPs and corresponding parameters. In practice, the conditional attention layer is used to replace the original attention layer in the Transformer decoder [48, 3] while keeping the other components unchanged.
Conditional Alignment in Teacher. The teacher aligned the obtained Sy and SA to Sp using the designed Conditional Aligmment module. Specifically, the MLLMs' prompts is used to specify which sentiment cues in each modality require more attention, thus helping the teacher better capture aligned sentiment information across these modalities. We denote the aligned outputs as H.Sachhrr and H Kacher which are then utilized for multimodal fusion. For example, the process that align visual modality to language modality can be described as:
Teacher H H
CondlAliggment(Xy.. (Xv, Xp; (lacher E Tixd
(6)
where CigdAligument represents the Conditional Alignment module, Xp denotes the prompt from MLLMs, Beacher is the parameters used to align the modalities.
Alignment in Student. The alignment module in student is designed learn the relationships between modalities independently (i.e., learning without the help of MLLMs' prompts). We denote the outputs of the module as Suddent and H For example, the H can be obtained by:
Student = Alignment, (v,, X1; (Sudentt H V-L
TR Tixd
where Alignment and Student represent the Alignment module and parameters, respectively.
3.4
Multimodal Fusion and Prediction
With these features extracted from the various modalities, we employ a Transformer encoder with self-attention blocks for multimodal fusion. In practice, we concatenate the obtained features with randomly initialized and learnable regression token / E 1 as input, then the Transformer encoder transfer and compress essential information to the Huusion thus making sentiment prediction through this token. For the final sentiment prediction, we apply a linear layer to Hussion
= Regression Husion, Oregr) IR.
(8)
where y denotes the predicted sentiment score, Regression represents the linear layer, and Oregr represents the parameters of the linear layer.
5

---

3.5 Learning Objectives
As outlined Section 3.1, the training MMSLF consists of two stages: (1) training the teacher and (2) training the student. In the first stage, the teacher learns to perform MSA under the guidance of prompts from MLLMs. The overall leamning objective defined as:
1 Σ
overall
rege
Teacher
1 -y/
where is the number of samples in the training set, 3/ is the sentiment label of the i-th sample, i' is the prediction of teacher. In the second stage, the student is trained under the supervision of the teacher, whose parameters remain frozen. The attention transfer loss (Sudentt formulated as:
N W
Studeett
W
Wonl
(10)
where W is the attention map from last layer of the alignment module the student, and W is the conditional attention map from the last layer of the conditional alignment module in teacher. The fused unified representation matching loss (Suueett is defined as: fusion
con
N
Student C
M
Miusion
(1I)
fusion
N
where Hi and iusion represent the fused features from the student and teacher, respectively. The sentiment prediction loss for the student is defined as:
Tegr
Stadent
1
Σ
1'b.
(12)
where j/ the prediction of student. Overall, the learning objective of student is:
[Stuent
Cieg + &LSiteen [tadent
(13)
where the a and 33 are empirically chosen hyperparameters. In practice, for the SIMS dataset, a and are set 60.0 and 8.0, respectively, while for the MOSI dataset, they set 100.0 and 4.0. For more discussion of the hyperparameters, please refer to Appendix C.1 and Appendix B.9.
4 Experiment and Analysis
4.1 Datasets
SIMS. SIMS [13] is Chinese MSA dataset, with data sourced from Chinese movies, TV series, and variety shows, featuring complex real-world scenarios. It consists of 1,368 training samples, 456 validation samples, and 457 test samples. Each sample is annotated with a continuous sentiment score ranging from -1 to 1, where -1 represents negative sentiment, and represents positive sentiment.
MOSL. MOSI [14] is an English MSA dataset, composed of data collected from YouTube. The dataset includes 1,284 training samples, 229 validation samples, and 686 test samples. Each instance is manually annotated with a continuous sentiment score ranging from -3 to 3, with -3 representing strongly negative and representing strongly positive.
MOSEL MOSEI [15] is an English MSA dataset with data collected from YouTube. contains 22,856 video clips, including 16,326 training samples, 1,871 validation samples, and 4,659 test samples. Similar to MOSI, each sample manually annotated with a score ranging from -3 to
4.2 Baselines
We compare our method with several advanced task-specific MSA methods, whose model parameters range from several million to tens of millions. These methods include: TFN [16), LMF [17], MuLT [3]. MISA [4], Sclf-MM [5], TETEN [21], ALMT [6], DLF [49], and concurrent work DeepMLF [50].

---

The performance of these models is all reproduced using a popular framework MMSA [51]. We also include some MLLMs, such as Video-LLaMA2 [9], GPT-4V [12]. GPT-4o-mini [12]. and Gemini- 2.0-Flash [52], for comparison. Additionaly, due to factors including differences in experimental settings, the lack of open-source implementation for certain methods, and space limitations, we have conducted additional comparisons in Appendix B.1 for more detailed comparison and discussion.
4.3 Evaluation Criteria
Consistent with previous studies [4, 6], we evaluate the regression tasks by reporting the mean absolute error (MAE) and the correlation between the model's 's predictions and human annotations (Corr). Since the predicted sentiment score can be used to compute classification accuracy, we also report Acc-2 and Fl scores for all datasets. For example, scores > O are treated as positíve while scores I are non-positive. This method widely used in the MSA studies. Additionally, in line with prior work [4. 6]. we report accuracy based on both negative/positive and negative/non--neattie classifications for the MOSI and MOSEI datasets, In the tables, performance metrics computed using these two classification methods are separated by "f", with the left side representing negative/non- negative performance and the right side representing negative/positive performance. All results are averaged over five runs, with standard deviations reported.
Since there are significant differences in the performance of many MLLMs between classification tasks and regression tasks, directly calculating classification accuracy based on regression metrics leads to poor performance for models like GPT-4o-mini. To more accurately demonstrate the true capabilities of MLLMs, we conducted two evaluations on the MLLMs. One of the tests involved using a classification prompt template to evaluate the Acc-2 and Fl metrics, while the other involved using a regression prompt template to assess the MAE and Corr metrics. For fair comparison, all task-specific models were evaluated in the same manner, i.e., using Acc-2 /SMMSS/negaivvenonnnngggn
Acc-2 (MOSI and MOSEI) to determine the model parameters for evaluation of classification performance and using MAE to determine the model parameters for evaluation of regression performance. More related discussions can be found in Limitation.
4.4 Performance Comparison
Table 1, Table and Table present the performance results on SIMS, MOSI and MOSEL, respectively, which is advanced MLLM at present, performs the best in most metrics. Notably, the performance of the teacher is close to the GPT-4o-mini in many metrics, and it outperforms both Vidco-LLaMA2 and GPT-4V in all metrics on all datasets. Furthermore, compared to Video-LLaMA2 and GPT-4V, both the teacher and student demonstrate improvements across most metrics. For example, on the SIMS, the student achieves an Acc-2 of 81.40 marking a relative improvement of 1.64% over Video-LLaMA2. When compared to the task-specific model ALMT, student achieves 2.10% relative improvement in F1 on the SIMS. A similar trend is observed on the MOSI and MOSEI dataset (Table 2), showing the general applicability of MMSLF across cultures, i.e., both Chinese and English datasets. Moreover, it is worth noting that the student can achieve advanced performance with fewer parameters compared to MLLMs, which underscores the potential of task-specific models in the MSA field. Furthermore, shown in the Table 3, the results on the larger dataset MOSEI show that teacher/-Student achieves advanced performance on many metrics. This demonstrates that MMSLF has good generalization ability datasets with different sizes. It is worth noting that the concurrent work DeepMLF demonstrates notable performance across the three datasets. By introducing a task-specific MLLM with learnable tokens for multimodal fusion, DeepMLF brings a promising research direction for MSA. Alongside MMSLF, it further highlights the potential of MLLMs in advancing the MSA field.
4.5 Effect of Each Component
In Table 4, we show the results by removing specific components. First, when we removed the MLLMs' prompt from the teacher, we observed significant drop in performance across both datasets. Specifically, on the SIMS dataset, the Fl score decreased from 84.06% to 80.84, and MAE increased from 0.370 to 0.436. A similar trend was observed on the MOSI dataset, where the Fl score dropped from 85. 15% to 79. 79.60% and MAE increased from 0.734 to 0.914. These phenomenoa show that the MLLMs plays crucial role in helping the model capture relevant multimodal information
7

---

Table 1: Performance comparison on SIMS dataset. a represents the results reproduced by the authors from open-source code with default hyperparameters. b represents the results are from [2]. represents the results are from [13].
Method
Acc-2
F1
MAE
Corr
Video-LLaMA2
80.09
79.94
0.584
0.476
GPT-4V
81.24
GPT-4o-min'
82.71
82.51
0.453
0.663
Gemini-2.0-flash
85.12
84.69
0.381
0.747
TFN
1..1
7..83-11.62
0.434.11.12
0.579
MISA
77.72=1.10
76.54+1.67
0.451
0.570=1.95
Self-MM
77.94#1.11
77.72:#0.68
0.418
0.589+1.54
TETEN"
80. 10.1890.49
79.34=0.52
0,422+11.0
0.58
ALMT
79.21
80.17=0.60
0.421110.69
0.583+10.70
DeepMLF
82.89
83.09-42.32
0.362:10.30
0.720+10.30
MMSLF
Teacher
83906=0.95
04.06:0043
0.370+
0.690-
Student
81.40=1158
81.85
0.382+1.39
0.662::1.26
Table 2: Performance comparison on MOSI dataset. a represents the results reproduced by the
authors from open-source code with default hyperparameters, while represents the results are from [2].
Method
Acc-2
Fl
MAE
Corr
Video-LLaMA2"
83.24/86.43
82.60/86.23
1.149
0.696
GPT-4V
80.43/-
GPT-4o-mint
87.32/89.48
87.17/89.42 .2
0.997
0.842
Gemini-2.0-flash
87.76/89.49
87.74/91.61
0.633
0.856
TFN
77.38:1..3778.11100
1..35 03/50.3..77700
0.949=3.13
0.662:11.95
MISA
$0.93=10.99/81105-000
10.90:1103//3 10.87
0.773=1.81
0.775110.63
Self-MM
1094--0.63/8.18--0.00
2..55+...33/3.099H0000
0.717+1.53
0.792++0.55
TETEN"
0377=0.52/80.82-111
0.726=1.68
0.791=+0.86
ALMT
83.00-00.2/855. 12:0020
$3.00++0.22/88 1.22
0.713=0.75
0.795-10.54
DLF
-833.69.0.299
-0.277
0.761
0.78110.48
DeepMLF
1..65_11.04/81.0-0000
112.11..7/81.0411100
0.795
0.75811..12
MMSLF
Teacher
85.05-1..666666110..669
85.1510.66766661100.00
0.734=1.46
0.797110600
Student
83.62-20.91/85337111000
83.6810.9685..0-110006
0.746=1.63
0.775
Table 3: Performance comparison on MOSEI dataset. a represents the results reproduced by the authors from open-source code with default hyperparameters.
Method
Acc-2
MAE
Corr
Video-LLaMA2 83.29/84.50
83.23/85.21
0.922
0.406
GPT-4o-min Somini-2.0-flasht
85. 00/48/969
85.25/87.04
1.015
0.744
38.07/58.86
88.08/72.08
0.583
0.743
TFN"
83.00-+0.45/22.000 9.082.4990443
1...8--0.40/82.888--000
0.566-+0.31
0.725--0.21
MISA
0344110.30055.090000
84.16610.00855.0211
0.553-0:46
0.759-10.25
Self-MM
$4.1550..508449000000
44.155:0.43844.991100
0.529=0.47
0.764:10.5
TETFNS
84.1810.62/85.42--000
64.06100.3/885.11110
0.543=0.51
0.76910.27
84.35=0.33884466--0455
84.10:40.32/44.25--
0.542=0.45
0.768-10.17
D
-184.76+0.32
-184.70+0.35
0.543+0.11
0.759+1..00
DecpMLF
83.49+0.5588..67-0..44
83.79:0:45786.7744..11
0.510=0.42
0.800+L0.24
MMSLF
Teacher
85.55102248667110777
0.53
0.773.11.51
Student
83.96--.338884.66-000
84.20.L0-4865444411..38
0.548-10.41
0.747-0.51

---

more effectively. Second, we removed the guidance of the teacher during the training of the student. This led decrease in the student model's performance, with the Fl score on SIMS dropping from 81.85% to 78,72%, and on MOSI from 83.68% to 83.00% The increase in MAE values on both datasets also reflects the student model's s reduced ability to align multimodal information without teacher guidance. It also shows that the importance of knowledge distillation, as the teacher's guidance can help the student learn the relationship between each modality effectively. Furthermore, the ablation results on the MOSEI dataset can be found in Appendix B.
Table 4: Effect of each component.
SIMS
MOSI
Method
F1
MAE
F1
MAE
MMSLF-
84.06=10.43
0.350:10.50
85.15
0.734
w/o prompt
80.8410.93
0.43611.57
79.60
0.914=0.68
MMSLF-Student
81.85=1141
0.39
83.688-0..9685555000006
0.746::1.63
w/o guid. of teacher
78.72++0.53
0.429++1002
83.00-0..59885077H00..55
0.743
4.6 Effect of Each Regularization
To evaluate the effect of each regularization in the student, we removed CSuudent Student and both Latt
CStdde and Latt (Stueent The results are presented in Table 5. We observe that both F1 and MAE decrease when each regularization is removed, indicating that every regularization contributes positively to the performance of student. Moreover, it is evident that the impact of each regularization more significant on the SIMS dataset than on the MOSI dataset. For example. when (Sudent is removed, Lit the Fl score drops by a relative 3.24% on SIMS, while it decreases by only 1.11% on MOSI. These differences could be attributed to the varying levels of difficulty between the MOSI datasets. Additionally, we tried different combinations of a and 3. please see Appendix B.9 for more details.
Table 5: Effect of each regularization.
SIMS
MOSI
Method
F1
MAE
F1
MAE
MMSLF-Studdnt
$1.85+1..41
0.382 1.39
0.68--0..6/8555--10..66
0.746:11.63
w/o Stulent
79.7
0.453
0.741
w/o
Civrion
79.23:10.69
0.428:10.87
83.16:0.51188544410..5
0.738:10.76
w/o
L Lusion
& Student
78.72+0..3
0.429+1.02
83.00=0.5998..7740..2
0.743+
4.7 Effect of Prompt from Different MILMss
To evaluate the impact of different prompts on task-specific model performance, we compared the results of using prompts generated by Gemini-2.0-Flass and GPT-4o-mini. As shown in Table 6, the teacher and student models guided by GPT-4o-mini outperform those guided by Gemini-2.0- O-Flash across most metrics. Despite Gemini-2.00Flash additional capability to process audio, its prompts are less effective, resulting in a significant performance decrease (further discussion on the quality of prompts can be found in Section 4.8). This indicates that the quality of prompts generated by large models is important to the performance of rask-specific models, and that optimizing prompt generation for specific tasks can significantly improve task-specific models' performance.
4.8 Analysis of Sentiment Cues in MLLMs' Prompt
To further analyze the impact of prompts generated by different large models, we show the results of sentiment classification using prompts from different modalities in the Table 7. The results indicate that Semini--20--Faas performs significantly worse than GPT-40-mini across multiple metrics. For example, in the SIMS dataset, Gemini-2.0 2.0FFash has an Acc-2 and Fl of 18.82% and 31.68% respectively, for the linguistic cue, while GPT-4o-mini achieves 78.99% and 79. 79.64%. similar trend is observed in the MOSI dataset. These results indicate that, despite Gemini-2 2.0 O-Flash having the

---

Table 6: Effect of prompt from different MLLMs.
SIMS
Method (MLLM)
Acc-2
MAE
Corr
MMSLF (Gemini-2.0-Flash)
Teacher
81.09:10.23 80.00:10.41
81.0910.29
0.377:10.73 0.68
Student
80.11
0.422.10.96 0.627
MMSLF (GPT-4o-mini)
Teacher
83.06=00.95
84.06=+0,43
0.57 0.382
0.690=0.80
Student
11.40
81. 1..4
0.66
MOSI
Method (MLLM)
Acc-2
F1
MAE
Cоrт
MMSLF (Gemini-2.0-Flashh)
Teacher
00.58 $0.51+0069982255
0.865+2.55 +2.57
Student
83.56-0.338/5.371...0 $3.4910.33/85.261100 0.722:10.93 0.78
MMSLF (GPT-40-mini)
Teacher
85.05110.6/86661110...6 85.15 0.734 0.797
Student
83.62:0.91/85.37--111 (...8++0.66/8550---00 0.746 1.63 0.775 1.10
added capability of analyzing the audio modality compared GPT-4o-mini, the accuracy of the prompts it generates is still not as good as that of GPT--4-mini. They may contain more misleading information, which may result in poorer performance when using prompts generated by Gemini-2.0r 0 Flash for model training as discussed in Section 4.7. This also indicates that the accuracy of the sentiment information in the prompts has a significant impact on the task-specific model.
Table Analysis of sentiment cues MLLMs' prompt. The left side and right side of "/" are Acc-2 and FI, respectively.
SIMS
MOSI
Prompts Source
Gemini-2.0-Flan GPT-4o-mini
18.82/31.68 78.99/79.64
14.22/24.90 78.56/75.82
4.60/8.79
83.39/83.37 86.88/86.87
57.29/50.35
56.85/49.91
76.68/75.84
Conclusion and Future Work
In this paper, we explored the application of MLLMs MSA tasks and assumed that MLLMs can assist task-specific models during training, thus achieving better performance. To validate this idea, we introduced novel MLLM-guided Multimodal Sentiment Learning Framework (MMSLF), This framework leverages general-purpose MLLMs to generate prompts that guide the learning process of teacher model. The teacher subsequently transfers the acquired knowledge to a student model, enabling it to perform inference independently without further reliance on MLLM MLL.M-eeneraee generated prompts. Extensive experiments on the SIMS, MOSI, and MOSEI datasets demonstrate that MMSLF achieves competitive performance across most metrics, thereby confirming our initial assumption and offering new insights into the integration of MILMSs for MSA tasks.
Moreover, current MSA research faces two key challenges. First, achieving significant performance improvements has become increasingly difficult. One of the reasons that many current research over-relies on conventional feature extraction toolkits such as Librosa and OpenFace. While effective, these tools may not provide sufficiently rich representations for advanced architectures. Exploring more powerful pretrained multimodal feature extractors could offer new directions for progress. Second, reproducibility remains a critical issue. Many existing MSA datasets are relatively small, leading to high training variance and sensitivity to initialization. This data scarcity not only hinders model generalization but also complicates fair comparisons across studies. Therefore, we argue that there is need to establish larger, more diverse, and well-annotated MSA benchmarks. In our future work, we aim to address these limitations and extend the proposed framework to broader affective computing tasks.
10

---

Acknowledgement
This work was supported in part by the National Science and Technology Major Project of China (2022ZD01 16408) and by the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001)
References
[1] Yingying Jiang, Wei Li, M. Shamim Hossain, Min Chen, Abdulhameed Alelaiwi, and Muneer H. Al-Hammadi. A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Inf. Fusion, 53:209-221, 2020.
[2] Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. GPT-4V with emotion: A zero-shot benchmark for generalized emotion recognition. Inf. Fusion, 108:102367, 2024. doi: (0.1016)..NFFUS. 2022.002262.
[3] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang. J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 6558-6569 Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1/P) 11MP19116566
[4] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. MISA: modality- invariant and -specific representations for multimodal sentiment analysis. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16,
2020, pages 1122- 1131. ACM, 2020. doi: 10. 10.1144//994777331136758
[5] Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-specifie representations with self-supervised multi-task learning for multimodal sentiment analysis. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 10790-10797. AAAI Press, 2021.
doi:
[6] Haoyu Zhang. Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, and Tianshu Yu. Learning language-guinn guided adaptive hyper-modality representation for multimodal sentiment analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 756-767, -67. Association for Computational Linguistics, 2023. doi: 10.18653/V 12253//1/2023... EMNLP-
[7] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[8] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunet audio-visual language model for video understanding. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. EMNLP 2023 System Demonstrations, Singapore, December 6-10, 2023, pages 543 553-553.44 Association for Computational Linguistics, 2023.
[9] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. CoRR, abs/2406.07476, 2024.
[10] Haozhe Zhao. Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang. Wenjuan Han, and Baobao Chang. MMICL: empowering vision-language model with multi-modal in-context learning. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Openkeview.net, 2024.
[11] Wenbin Wang, Liang Ding, Li Shen, Yong Luo, Han Hu, and Dacheng Tao. Wisdom: Improving multimodal sentiment analysis by fusing contextual world knowledge. In ACM Multimedia 2024, 2024.

---

[12] OpenAl. GPT-4 technical report. CoRR, abs/2303.087744 2023.
[13] Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma. Jiele Wu, Jiyun Zou, and Kaicheng Yang. CH-SIMS: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7718-3727. Association for Computational Linguistics, 2020.
[14] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intell. Syst., 31(6): 82-88, 2016.
[15] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2236-22360 2236..2.....22 Association for Computational Linguistics, 2018.
[16] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis- Phillip Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1103-1114. Association for Computational Linguistics, 2017.
[17] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhann, Paul Pu Liang, Amir Zadeh, and Louis-Phillipe Morency. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2247-2256. Association for Computational Linguistics. 2018.
[18] Wasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Douis-Phillippe Morency, and Mohammed E. Hoque Integrating multimodal information in large pretrained transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2359 2369. Association for Computational Linguistics, 2020.
[19] Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu. Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 24, 2021, pages 4400 4407. ACM, 2021.
[20] Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 2554-2562. Computer Vision Foundation IEEE, 2021.
[21] Di Wang, Xutong Guo, Yumin Tian, Jinhui Liu, Lihuo He, and Xuemei Luo. TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis. Pattern Recognit. 136: 109259, 2023.
[22] Di Wang, Shuai Liu, Quan Wang, Yumin Tian, Lihuo Hc, and Xinbo Gao. Cross-modal enhancement network for multimodal sentiment analysis. IEEE Trans. Multim., 25:4909-4921, 2023.
[23] Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 202), Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 9180-9192. Association for Computational Linguistics, 2021.
[24] Ziqi Yuan, Yihe Liu, Hua Xu, and Kai Gao. Noise imitation based adversarial training for robust multimodal sentiment analysis. IEEE Trans. Multim., 26:529-539, 2024.
[25] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vorr Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
12

---

Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.
[27] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
[28]
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Tean-Baptine Alayrae, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.1 1805, 2023.
[29] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. CoRR, abs/231 1.03079, 2023.
[30] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OeenReview.net, 2024.
[31] Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 12585-12602. Association for Computational Linguistics, 2024.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 1CML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 2023.
[33] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans. LA, USA, December 10 16, 2023, 2023.
[34] Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. arXiv preprint, 2024.
[35] Xuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini, and Pontus Stenetorp. Us- ing natural language explanations to improve robustness of in-context learning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2024, Bangkok, Thailand, August 1 1116, 2024, pages 13477-13499, Association for Computational Linguistics, 2024.
13

---

[36] Chengming Hu, Xuan Li, Dan Liu, Haolun Wu, Xi Chen, Ju Wang, and Xue Liu. Teacher- student architecture for knowledge distillation: A survey. CoRR, 0as/2308.04260 2023. doi: 10.48550/ARXIV22000.
[37] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, fess/500302553 2015.
[38] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consisteney targets improve semi-superviscd deep learning results. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1195-1204, 2017.
[39] Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang. and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 1002-3910. Computer Vision Foundation/ / IEEE, 2020.
[40] Ziqi Yuan, Jingliang Fang, Hua Xu, and Kai Gao. Multimodal consistency-based teacher for semi-supervised multimodal sentiment analysis. IEEE ACM Trans. Audio Speech Lang. Process., 32:3669-3683, 2024. doi: 10. 00.1109TAASLPP222444333333
[41] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[42] Jangho Kim, Sconguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2765-27744 2018.
[43] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[44] Mingyang Li, Ying Gao, Wentian Cai, Weixian Yang, Zihao Huang, Xiping Hu, and Victor C. M. Leung. Enhanced attention guided teacher-student network for weakly supervised object detection. Neurocomputing. 597:127918 10, 2024. doi:
[45] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HLLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-41888 Association for Computational Linguistics, 2019.
[46] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. Openface 2.0: Facial behavior analysis toolkit. In 1 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, Xi an, China, May 15-19, 2018, pages 59-66. IEEE Computer Society, 2018.
[47] Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt Vicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th Python in Science Conference 2015 (SciPy 2015), Austin, Texas, July 6- 12, 2015, pages 18-24. scipy.org, 2015.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 0998-6008, 2017.
14

---

[49] Pan Wang, Qiang Zhou, Yawen Wu, Tianlong Chen, and Jingtong Hu. DLF: disentangled- language-focused multimodal sentiment analysis. In AAAI-25, pages 21180-21188 AAAI Press, 2025.
[50] Efthymios Georgiou, Vassilis Katsouros, Yannis Avrithis, and Alexandros Potamianos, Deepmlf: Multimodal language model with learnable tokens for deep fusion in sentiment analysis. CoRR, abs/2504. 1082, 2025.
[51] Huisheng Mao, Ziqi Yuan, Hua Xu, Wenmeng Yu, Yihe Liu, and Kai Gao. M-SENA: integrated platform for multimodal sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 System Demonstrations, pages 204-213. Association for Computational Linguisties, 2022.
[52] Sundar Pichai, D Hassabis, and K Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era, 2024.
[53] Xinyu Feng, Yuming Lin, Lihua He, You Li, Liang Chang, and Ya Zhou. Knowledge-guin guided dynamic modality attention fusion framework for multimodal sentiment analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, 2024.
[54] Yong Li, Yuanzhi Wang, and Zhen Cui. Decoupled multimodal distilling for emotion recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, pages 6631-6640, IEEE, 2023.
[55] Zehui Wu, Ziwei Gong, Jaywon Koo, and Julia Hirschberg. Multimodal multi-loss fusion network for sentiment analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 5588-3607 Association for Computational Linguistics, 2024.
(56) Ming Zhou, Weize Quan, Ziqi Zhou, Kai Wang, Tong Wang, and Dong-Ming Yan. TCAN: text-oriented cross attention network for multimodal sentiment analysis. CoRR, abs/2404.04545, 2024.
15

---

A Limitations
In this paper, we explored the possibility of using prompls from general MLLMs to help model alignment, thus improving model's performance. To effectively verify the feasibility of the ideas, we use simple and intuitive model designs. However, this also leads to some limitations of the model. (1) Due to the of traditional feature extractors, the performance of MMLSF has not been fully realized. For example, in Appendix B.3, the model's performance improved after replacing with more powerful feature extractor. (2) Due to the large number of parameters of MMSLF-Teadhe when training it on small-scale datasets such as MOSI and SIMS, the phenomenon of overfitting is extremely severe. (3) Due the small parameters of the MMSLFF-ttudent, the model training is unstable (as shown in Appendix B.1 During the training process, we found that there were significant differences in performance under different random seeds.
B
Additional Experiments and Analysis
B.1 Additional Comparison
Despite the differences of the settings, such as five runs, in the MSA method, we still to use the original results to ensure comprehensive comparison. This methods includes TEN [16], LMF [17], MuLT [3], MAG-BERT [18), MISA [4), Self-MM [5], TETFN [21], CENET [22], ALMT [6], KuDA [53], DMD [54], MMML [55] (See Appendix B.3), DLF [49], TCAN [56] and concurrent work DeepMLF [50].
As illustrated in Table 8, Table and Table 10, we use MAE as the key metric determine the model checkpoints which is consistent with the comparative methods. It is obvious MMSLF demonstrates competitive performance across all datasets, with particularly strong results on the student model. We also observed an interesting phenomenon: the student slightly outperforms the teacher on some metrics. For example, the teacher achieves an Acc-2 of 76.71 while the student achieves an Acc-2 of 80. We believe that this occurs because we retained direct supervision from the ground truth labels rather than fully aligning with the teacher during student training, so some of the teacher's subtle biases were not transferred. These phenomenon demonstrate the effectiveness of our proposed framework. In addition, we can see that DeepMLF achieves excellent performance on multiple datasets, which is inseparably linked to its architecture design that directly utilizes LLMs such as GPT-2.
Table 8: Performance comparison on SIMS dataset. a represents the results are from MMSA [51]. b represents the results reproduced by the authors from open-source code.
Method
Acc-2
F1
MAE
Corr
KuDA
80.74
80.71
0.408
0.613
TFN"
78.38
78.62
0.432
0.591
LMF"
77.77
77.88
0.441
0.576
M
78.56
79.66
0.453
0.564
MISA
76.54
76.59
0.447
0.563
Se
80.04
80.44
0.425
0.595
TETFN"
81.18
80.24
0.420
0.577
CENET
77.90
77.53
0.470
0,540
ALMT*
10.68
78.24.10.58
0.421
0.583
DeepMLF*
82.19
82.50.10,74
0,362
0.720-L0.30
MMSLF
Teacher
76.7 1 3.305
77.3112.72
0.55
0.690-L0.80
Student
80.18+1.007
80.18+1.006
0.382+1.39
0.662+1.26
B.2 Generality of the Proposed Framework
To evaluate the generality of the MMSLF, we applied the Teacher-Student framework to ALMT [6]. As shown Table 11, AL.M--Teache outperformed MMSL.F-Teacher across all metries on both the MOSI datasets, demonstrating the effectiveness of utilizing MLLMs to improve the learning of
16

---

Table 9: Performance comparison MOSI dataset. represents the results from MMSA [51]. b represents the results reproduced by the authors from open-source code.
Method
Acc-2
F1
MAE
Corr
MAG-BERT
-/86.10 10
-/86.00
0.712
0.796
DMD
-/83.23
-/83.29
0.752
TCAN
./86.28
-/86.15
0.714
0.797
KuDA
84.40/86.43
84.48/86.46
0.705
0.795
TFN"
77.99/79.08
77.95/79)
0.947
0.673
LMF"
77. 97.9999.18
77 77.8/79.15
0.950
0.651
MuLT
79. 19.71/80,
79.63/80.95
0.880
0.702
MISA
81. $1.84/83.55
81 $1.8283.58
0.777
0.778
Self-MM
83 36.
85.46/85.44 433
0.708
0.796
TETEN"
83. 83.24/85.37
83.13/85.33
0.708
0.798
CENET
83.53/85.21
83,49/85.22
0.725
0.795
ALMT*
$2.22.0.83//8 12.50.55
12.1551..87784111 0.555
0.713-10.75
0.795.10.54
DLF"
1833.66
-183.70=0.43
0.761
0.781
DeepMLF
$1.55=#0.88833.8-0000
0.795+22441
0.758
MMSLF
Teacher
84.32.10.92085.9911.001
0.960.-0.9
0.734.1..66
0.797
Student
82.52.1042284114107.33
83.49110.51144188.0.777
0.746.1163
0.775111110
Table 10: Performance comparison on MOSEI dataset. a represents the results are from MMSA [51]. b represents the results reproduced by the authors from open-source code.
Method
Acc-2
F1
MAE
Corr
DMD
-/84.62
-/84.62
0.543
TCAN
-/86.27
-/86.
0.532
0.774
KuDA
83 83.26/86.46
82.97/86.59
0,529
0,776
TEN"
78.50/81.88
78. 78,96/81 98.96668.74 1.74
0.573
0.714
LMF
80.54/80.94
83 83.48/83.36
0.576
0.717
M
81.15/84.63
81.56/84.52
0.559
0.733
MISA
80.67/84.67
81.12/84.66
0.558
0.752
Se
83.76/85.15
83.82/84.90
0,531
0.765
TETFN"
84.12/86.21
84.35/86.11
0.537
0.770
CENET
83.52/86.38
83.85/86.32
0.526
0.778
ALMT
13.28:10.40085.16--1
83.20=10.78/85 7..77 1.76
0.542=10.5
0.76811011
DLF
-184.53-0.52
-184.49-0.47
0.54
0.7591:0300
DeepMLF*
$1.57:60.7118..2344..33
12.16..0060//6..77 0.27
0.510-10422
0.800
MMSLF
Teacher
83.55=11.61885.5550773
83.74 1.238..11.333
0.539
0.773
Student
81.442.21/85.13104444
$2.00:2.0208..9110.2.9
0.52010.33
0.74110.60
task-specific small models. However, ALMT-Student did not exhibit the same level of improvement SMMSLF-Studenl We attribute this isbecause that ALMT was not originally designed with the Maucher-Student framework. Its reliance on multiple specialized attention maps complicates the optimization of the student model during the knowledge distillation process. Additionally, it is worth noting that MMSLF-Studde achieved better results than ALMTT-Studen with a significantly smaller number of parameters, further demonstrating the effectiveness and efficiency of the MMSLE.
B.3 Modality Extractor Analysis
Some recent methods like DeepMLF [50] and MMML [55] use advanced feature extractors to help the model achieve better performance. We also investigated the impact of the feature extractor on the MMSLF's performance and compared it with MMML. Consistent with MMML, we ues RoBERTa and Data2Vec as the text and audio modality extractor, respectively. Moreover, we used MAE determine the model parameters and report the results with three runs. The results are shown in the Table 12 below. We can see that MMSLF achieve performance improvement using more powerful feature extractors. Compare with MMML, although MMML achieves higher performance
17

---

Table 11: Generality of the proposed framework.
Method
Acc-2
FI
MAE
Corr
SIMS
ALMT
Teacher
84.20-0.577
84.45-0.81 80,58.1 1.05
0.36
0.711+11.50
Student
79.87
0.418
0.587:13.97
MMSLF
Teacher
83.06.10.95
84.06.10.43
0.370
0.690.10.80
Student
11.40=1.58
81.85=1.44 1.41
0.382+1.338
0.662:#1.26
MOSI
ALMT
Teacher
86.56:+0.660888.22:00000
86.63
0.677=00.57
0.834
Student
83.26-:0::41855.33H00.1
83.38:+0.311885522HH.
0.720=#0.54
0.784-10.28
MMSLF
Teacher
85.05110.6/8666110000
85.15110.6686.6999
1.734=1.46
0.797:10.00
Student
83.62:10.1166537:11.00 83.68_0.99/855501.00020
0.746=1163
0.775
metrics. For example, the teacher achieves better results on Corr (0.79244..5 15). The student achieves better results on Acc2-NonO 37.009+.225 F1-NonO (87.18#0.. and MAE (0.513+1.27) These results indicate that it is feasible to use updated and more powerful extractors to achieve better MSA performance.
on Acc2-Hass (86.32%) and Fl-HasO (86.23%), method outperforms
Table 12: Comparison with MMML on MOSEI dataset.
Method
Acc-2
F1
MAE
Corr
MMML
86.32/86.73
86.23/86.49
0.517
0.791
MMSLF- 85.47:10.25 87.06.10.46
85.53110.27 87.16.0.38
0.522 1.31
0.792.10.15
MMSLF-Student 85.93.10.58 87.0
86.08 / 87.18.10224
0.513:1..27
0.785
B.4
Trade-off' Analysis
trade-off analysis in As shown in Table 13, we quantitatively presented the datasets. Note that the reported parameter terms of parameters, GFLOPs, and inference time SIMS counts do not include those the feature extractors used for serialization, such as OpenFace [46],
Librosa [47], and BERT [45]. This is because tools like Librosa lack well-defined parameter counts. More importantly, excluding them allows for a fairer comparison focused on the core modeling components across methods. The overall computational complexity of the models can be assessed using the GFLOPs metric.
Obviously, our student model achieves competitive performance (Fl of 81.85+1..4 1,41 SIMS) with only 0.82M parameters, 8.6 GFLOPs, and 6.39s test-time inference. This demonstrate that our methods can achieve better 2.0-Flash and GPT-4o require larger parameters and >22min inference time. These results demonstrate that method offers trade-off between performance and computational cost. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs relatively higher. This is because do not compress the input sequences' length as the prior methods [4, 6]. With further optimization (e.g., sequence dimension reduction), the computational cost of MMSLF can be further reduced.
trade-off than other methods. In contrast, Gemini-
B.5 Significance Analysis
As shown Table below, we show two-tailed t-tests between the Student model and the task-specific baseline ALMT. The resulting p-values are MAE=0.0029, Com=0.00022, Acc-2=0.095, and F1=0.066. Under the conventional 0.05 threshold, the Student's improvements on MAE and Corr statistically significant, while the gains ACC-2 and Fl show the same positive trend.
18

---

Table 13: Comparison of efficiency and performance on the SIMS dataset.
Method
Parameters
GFLOPs
Inference Time
F1
MAE
GPT-4V
30min
81.24
GPT-4o-mini
> 7B
27min
82.51
0,453
Gemini2.0 F-aaai
> 7B
> 22min
84.69
0.381
TFN
35.63M
0.101
3.46s
1..83+1.62
0,434+1.12
MISA
21.66M
7.33
12.32s
76.54+1.66
0.451+1.83
Self-MM
0.38M
6.66
11.40s
77.72++0.68
0.418+1.05
TETFN
1.53M
6.72
26.57s
79.34.10.52
0.422.1
ALMT
2.60M
7.00
16.08s
80.17110.60
0,42111.69
MMSLF-Teacher
2.54M
96.16
12.31s + 27min
84.06:10.43
0.370
MMSLF-
0.82M
8.61
6.39s
81.85=+1.41
0.382
Table 14: Two-tailed t-tests between the Student model and the ALMT on SIMS dataset.
Method
p-value (MAE)
p-value (Corr)
p-value(Ace.2)
p-value (FI)
MMSLF-Student & ALMT
0.0029
0.00022
0.095
0.066
B.6 Prompt Sampling Analysis
Considering both cost and performance, we obtain all prompts once before training the teacher model. We also experimented with generating three prompts per sample and randomly sampling one during training on SIMS dataset, but found that this way lead to higher cost without significant performance improvement. As shown in Table below, the "No Sampling" setting achieves comparable or even better results. Moreover, we observe that the "Sampling" strategy introduces noticeably higher variance MMSLF-Teacher suggesting that sampling different prompts may introduce inconsistent guidance and lead to unstable training. However, for MMSLF-Student, the variance across runs is relatively small in both settings (e.g., Fl std: 1.41 0.66), indicating that the student model is less sensitive to prompt sampling. This is likely because the student learns from the teacher's distilled representations and attention patterns, rather than directly using the prompts.
especially in MAE and Corr (e.g., 2.71 vs. 0.50 for MAE std),
Table 15: Effect of prompt sampling methods on performance.
Method
Acc-2
F1
MAE
Corr
MMSLF-Teacher (Sampling)
1.75
1332
0.370++2.71
0.682:4.66
MMSLF-Student (Sampling)
81.05-L0.66
81.18:00.66
0.385+L0.92
0.667+1.87
-eehheee (No Sampling)
83.06:10.95
84.06:10.43
0.370+10.50
0.690+10.80
MMSI.FFSSudeet (No Sampling)
81.40+11.58
81.85+1.411
0.382++1.39
0.662++1.26
B.7 Prompt Sensitivity Analysis
As shown in Table 16 below, we evaluated our method using different prompt templates to assess robustness. The experimental results in the table show small performance variance across different prompt formulations within the same MILMM This indicate that advanced large-scale MLLMs may generate stable outputs regardless of reasonable prompt variations. This stability validates our framework's robustness to prompt engineering choices. In addition, the performance difference between GPT-4o-mint and Gemini-22 Camini--2.0FFaae 2.0--lash may stem from their varying abilities to generate accurate and informative prompts for sentiment-centere multimodal tasks, rather than sensitivity to prompt variations. Therefore, compared to changing the prompt input, the more important is to select appropriate mLLMM with strong domain-specific capabilities.
B.8 Performance Impact of Varying Student Parameters
Table 17 presents the performance impact of different parameter settings on the student model. We control the model parameters by modifying the depth of the alignment module. Notably, the student
19

---

Table 16: Comparison of Prompt Sensitivity and Robustness on SIMS datasets.
Method
Prompt
Acc-2
F1
MAE
Corr
Teacher
Prompt (Default, GPT-4o-mini)
83.06:10.95
84.06:10443
0.370=0.50
0.690=0.80
Teacher
Prompt (GPT-4o-mini)
81.23:15.94
83.93=1.116
0.346=1.12
0.716=1147
Teacher
Prompt 3 (GPT-4o-mini)
80.04:15.37
82.51
0.372=1.20
0.68712.72
Teacher
Prompt S-emini-22.0-Faast
81 81.09:-0.23
81.09+0.29
0.377=0.73
0.686+1.53
Teacher
Prompt (Gemini-2.0 (.0mmi--.0-
77.90=7.003
82.87+1.21
0.355=2.225
0.704=3.08
Teacher
Prompt (Gemini-2.0-FBil
71.42:+:.11
81.95+0.077
0.384=1.47
0.658
Student
Prompt (Default, GPT-4o-mini)
81.40:11.58
81.85+1411
0.382=1.39
0.662=1.226
Student
Prompt (GPT-4o-mini)
81.40:11099
81.65+0.96
0.394=1.06
0.667=0.45
Student
Prompt (GPT-4o-mini)
80.74=0.37
80.92=0.774
0.393=1.80
0.664#1.34
Student
Prompt Deemini-2.0-Flame
80.00:10.41
80.11
0.422-0.96
0.62
Student
Prompt (Gemini-2.0-Flass,
81.18:10.97
81.15=0..99
0.387=1.92
0.670=2.699
Student
Prompt (Gemini-2.0-Flaam
81.05++0.45
81.14+0.37
0.381 0.96
0.676=1.59
achieves optimal performance with 0.82M parameters, corresponding to a configuration (as shown in Table 20) of embedding layers, 2 alignment layers, and 2 multimodal fusion layers. beyond this point, increasing the model size does not significantly improve the performance, showing that the model has likely already fully utilized its learning capacity.
Table 17: Performance Comparison of Varying Student Model Parameters on SIMS dataset. The parameters from BERT used for input preprocessing excluded from the reported parameter count.
Method
Parm.
Ace-2
Fl
MAE
Corr
ALMT
2.60M
79.91-10.29
80.17-10.60
0,421 :00699
0.583110.70 0.690:10.80
Teacher
2.54M
83.06:10.95
84.06:10.43
0.370:10.50
0.49M
80.74-E1.16
81.44-11.03
0.408
0.638:12..15
0.82M
81.40-H:1.58
11.8
0.382++1.39
0.662:t1.26
Student
1,46M
80.66-L0.55
81.47-L0.54
0.400=11.64
0.631+11.22
2.11M
81.36:11.29
82.32-10.75
0.394=1.33
0.646:11.43
4.05M
81.40-10.71
81.79:10.00
0.39411.65
0.636::1.93
B.9 Effect of Regularization Weight on Model Performance
To investigate the impact of regularization weights, we experimented with various combinations of and B on the SIMS dataset. The results are presented in Table 18. It is evident that both a and B influence the performance of the student.
Table 18: Effect of regularization weight on model performance
Acc-2
F1
MAE
Corr
60.0
81.40 1158
81.85
0.382-11.39
0.662.1 1.26
80.0
8.0
81.01 = 1.51
81.27+1.34
0.39441..40
0.650-12.36
40.0
8.0
81.18-1.66
8..4
0.388+11.15
0.662-11.58
20.0
8.0
80.79=1.29
81.46=1.17
0.387+1.33
0.661
77.94=1.12
79.2810075
0.453+10.88
0.524+1.87
60.0
10.0
81.01=1877
81.27+1.67
0.389+1.20
0.656:11.30
60.0
6.0
80.88=1.20
81.37+0.92
0.392+1.53
0.653+1.82
60.0
4.0
80.74=1.01
81.23+1.16
0.393+1.63
0.650+2.26
60.0
2.0
80.53=0.97
81.05+00.99
0.396+1.006
0.645+2.299
60.0
78.29-0.42
79.2310.69
0.428
0.56413.11
0
7.44
78.72.0..3
0.429
0.567.11.39
20

---

B.10 Supplement of Each Component Effect
As shown in Table 19, we present the ablation experiments of each component on the MOSEI dataset as supplement to Section 4.5. We can see that each module is still useful on the MOSEI dataset, demonstrating the effectiveness of the MMSLF.
Table 19: Effect of each component on MOSEI dataset.
MOSEI
Method
MAB
MMSLF-Teacherr
85.551102248667110007
0.539 1 0.71
w/o prompt
84. 84.11
0.55810.78
MMSLF-Student w/o guid. of teacher
84.200=0-4488444000.2
0.548=0.41
83.3410.55844:30022
0.553=0.34
B.11 Convergence Performance Analysis
In Figure 3, we visualize the loss curves of student on the SIMS and MOSI datasets. While the overall trend shows decrease, the variance of Student across different seeds relatively high. We believe this is due to the difficulty student faces in aligning with the teacher's 's learning outcomes without the help MLLMs' prompts, resulting in fluctuations during the optimization process. Despite this, student still achieves competitive performance on both the SIMS and MOSI datasets, demonstrating effectiveness of the proposed MMSLF framework.
SIMS
Epoch
Epoch (b)
Epoch
(8)
(c)
MOSI
Frog
Epoch
Epoch
Epoch
(d)
(f)
Figure 3: Visualization of convergence eurves. The values the curve represent the averages corresponding to five runs.
B.12 Case Study of Conditional Attention Map
visualized the attention difference maps by subtracting the attention map without MILMs' prompts from the conditional attention map Teacher In combination with the key cue from large model, although these frames look similar, the first frame shows more pronounced mouth movements, while the third frame displays more obvious muscle movements in both the eyes and mouth. It is also important to note that the yellow frames merely indicate that model assigns them lower attention under the guidance of the MI.LM's prompts, not that they lack sentiment information. They are just relatively less important compared to the frames highlighted in blue. The observations indicate that the MLLM's prompts is useful for task-specific models' learning.
As shown in Figure
21

---

Key Cue: If the eyes wander or look down, it may convey foelings of loss or sadness.
Langing Mosblliy
[SUP]
Time
Vieual Modality
Figure 4: An example of attention difference maps on the SIMS. This difference map obtained by subtracting the attention map without MLLMs' prompts from the conditional attention map H Tencher Note: The blue areas indicate regions where the model focuses more when guided by the prompts, while the orange areas indicate regions where the model focuses less under the same prompts.
B.13 Examples of MLLMs' Prompts
As shown in Figure 5 and Figure 6, we provide more examples of MLLMs' (i.e., GPT-4o-mini and Gemini-22.0-Flase prompts, both in Chinese and English. For efficiency and cost-effectiveness, uniformly sample three frames from the video input as the input to the MILMs, consistent with previous works [2]. Compared with Gemini-2.00Flan analysis, its output did not include audio cues.
EPT--o-mini does not support speech data
22

---

Video
STUFF
STUFF
STUFF
BUT I THINK THERE WAS ALSO SOME SORT OF LIKE UM TONE WITHIN THE ACTUAL UM POWERS OF THE GREEN LANTERN AND HIMSELF AND THAT YOU CAN BASICALLY THINK UP ANY THING AND YOU CAN PROJECT THAT INTO THE REAL LIFE
Language
Visuisual Cues
1. * Facial Expressions""
any smiles positive expressions
enthusiasm exciterment topic.
Notice if the speaker's eyes widen or there brightening expressions that suggest engagement with the subject matter
2. BBody anguage"*:
Generated Prompts (GPT-4o-mini)
Observe gestures such hand movements may emphasize points being made, indicating passion or interest.
Pay sileution posture; open and relaxed posture suggest confidence and positivity, while closed posture might indicate uncertainty.
#08 Language Cues 6##
1. Emotionnlly Charged Words**
The phrase "thnk up of any mything" suggests creativity and imagination, which are often associated with positive feelings. The of "project that into real life" implies eepoweement and possibility, ruinforcing a posi sentiment.
2. *><ooee and Delivery=* listee for an upbest tone or inflection the speakker's voice that conveys excitement or enthusiasm about the topic.
(a) Example 1
ViVD
ViVO
VIVD
Video
Language
HEEE AM6. HL Ae (He has tied you up tightly, fully restrained)
38 S2E (Visual Cues)
1. 40- BI MAA (Pactal Expressions) 40:
* 38 (Smi) (ing) AU BE
BN (Obeuvve whethers smile appears, usually
couves eys positive amotions) ('rosziing) Indicate confusion or dissalisfaction.
18 Me (Pay attertion to any frowning. which may
(Eyye such as staring or oravoiding eye contact, can reflect the intensity and stlitude of emotions)
46 (AUHE BB)) mf Hil di fi (Changes in eye contact,
Generated Prompts (GPT-4o-mini)
2. seefli ($ d (Bocy Larguage) #4; (Gestures) of gestures, such as pointing or clenching fists, can convey the inteusity of emotonn). * 25 (Postures) we; 2: duriii 803 of denf of MX ae (The posture the body, such leaning forward or reclining, may indicate interest in or resistancce the topic being discussec).
12 50 hi (Inf9i0 36 tuit
ex H frequency type
Ii th #2 (Laggagge Cues) =me
1. *ee 1 Siii (Emotionally Charped Words) en;
RE fe de EE id fith (The phrase "tied up tightly" suggests heing bound, potentially conveying feelings of helplessesss or ogpression)..
"i A:3 88 se
" # digaaa se" fittt &k i8st Kittii " J2 sttkstttiiii
expression "fully restrained"
might carry playful or exaggerated tone, reflecting a seesse of bumor or sarcasm).
(b) Example 2
Figure 5: Examples of Prompts from GPT-4o-mini.
23

---

Video
STUFF
STUFF
STUFF
BUT I THINK THERE WAS ALSO SOME SORT OF LIKE UM TONE WITHIN THE ACTUAL UM POWERS OF THE ANYTHING
Language
PROJECT THAT INTO THE REAL LIFE
Visuual cues:**
The speaker maintains consistent eye contect, which conveys sincerity and engagementt
The speaker's facial expression appears neutral, suggesting that this segment the recording low-intensity.
Generated Prompts (Gemini)
Audio
The speaker has conversational tone that comes off conversational.
* Language cues.
The speaker deseribes how the powers of the Green Lantem work: he can think up anything and project it into life.
The tone descriptive and explanatory.
The speaker uses neutral language, loctsing on deseribing the powers Green Lanker.
(a) Example
Video
Language
TEX IL M has tied you tightly, fully restrainod)
ML (issual Cues)
Wor A TF 8o t o M0 MOAH Rut (The person's facial expression the video appears relarively IR A it. expressing thenselves seriously.
displays noticrable frown.)
of 68 RU A (EA MUB ol R.S. (The gaze focused, which may indicate that the person is speaking or
BEt dif RAA. Sh
TTRE it (200 (otnsiietttt Mot
(The body movemenis minimull and restrained, possibll
reflecting resereed emotional state.)
Auudio Cues):
Generated Prompts (Gemini)
18 34-4- 11 HUTHAAA
(The tone is steady, with no significant pitch varistions, indicating a
relalively stable
(The speech rale is moderate, without notiocable acvekeration
ur stowing down, which may reflect a calm emotional state.
(-00 at. "Li it. of He 2ogH 2 RU Mi. #i Ttt. (Pauses are not prominent, and the speech
is Ruent, suggesting that the speaker's thoughts are clear and their emocions are not noticeably disturbed.)
**ill Lanngaag Cues):
4J7 1f ri IH SUAKEE MiE 5 AT nf (LE (The
sentence
you tightly, fully discomfoot or (ear.)
being lied up
which may foclings
"I A.55" 3 4 -2:0 di
18 A.A.E te list (8 (The up tightly"
itself carries ocrtain nogative connotation, casily associated with situatioss of control or threat.)
(b) Example
Figure 6: Examples of Prompts from Gemini-2.00Flan
24

---

C Supplement of Implementation Details
C.1 Implementation Details and Hyperparameters
We implemented our proposed method using PyTorch 2.1.1 with CUDA 12.1. The experiments were conducted on PC equipped with an AMD EPYC 7513 processor (2.6GHz) and an NVIDIA Tesla A40 GPU. The key parameters are listed in Table 20.
In the training of the teacher, perform random mask the multimodal input to improve the data diversity. The ratio of random masks between 0 and 70% on the SIMS dataset and between 0 and 50% the MOSI and MOSEI datasets. Additionally, since GPT-4o-mini does not support speech analysis, we prompted it to infer possible speech cues based on the available language information. The prompt template used for this task is shown in Listing C.I.
Table 20: The parameters used on the SIMS, MOSI and MOSEI datasets
Parameter
SIMS
MOSI
MOSEI
Common
Batch Size
64
Optimizer Epochs Seeds
AdamW
Adam W
Adam
200
200 1111-1115
200
1111-1115
1111-1115
Warm Up
Cosine Annealing
d
64
64
64
Ti.. Tv. Tx. Ti
50, 55, 400, 50
50, 500, 375, 50
50, 500, 500, 50 1
The Depth of Language Embedding The Depth Visual Embedding The Depth of Audio Embedding The Depth of Prompt Embedding
1
2
2
MLLMs (GPT-4o-mini)
Temperature Version
0 2024-07-18
2024-07-18
2024-07-18
Teacher
Initial Learning Rate
lc4
lc-4
2c-4
The Depth of Conditional Alignment The Depth of Multimodal Fusion
6
6
Student
C₁ Initial Learning Rate
60.0, 8.0
100.0,
100.0, 4.0
2e-4
le-4
2e-4
The Depth of Conditional Alignment The Depth of Multimodal Fusion
1
2
C.2 Prompting Template to Generate Prompts for Teacher
Listing C. provides the prompting template used to generate prompts for the teacher on the MOSI dataset. Since SIMS is Chinese dataset, we directly translated this template into Chinese generate prompts for the teacher on the SIMS dataset. We can see that there is a strong guidance for prediction in the hints given by the MLLMs. Based these prompts, the teacher is more easily learn the alignment between modalities and in turn transfer this knowledge to the student which does not rely on MILMS prompts. More examples be seen in Appendix B.13.
25

---

Listing C.1: Prompting Template of MILMs to Generate Prompts for teacher
User
### Background ###
You multimodal sentiment analysis expert. We provide you with video-text pair and corresponding sentiment label. Please provide detailed hints to help task-specific small models identify sentiment cues.
### Video-text Pair and Corresponding Label ###lh
Note: The label is None the data is from the test set.
Video: Please refer to video input.
Text: THAT HUH BEFORE WE GO ON WITH THAT MUST SAY THAT THIS HAD A SURPRISINGLY DECENT CAST
Label: Positive
#*# Response Requirements ####
1. Focus on facial expressions in the vidco, especially smiles, frowns, and eye movements.
2. Focus the language in the video, especially emotionally charged words and phrases.
### Output Format ###
Your output consists of the following parts:
1. Visual cues: Observations related to facial expressions in the video.
2. Language cues: Important sentiment found the provided text.
GPT-4o-min mini
### Visual Cues
1. Facial Expressions: - Look for any smiles or positive expressions that indicate enjoyment or enthusiasm. genuine smile can enhance the positive sentiment. Observe the eyes: wide eyes convey excitement surprise, while squinting might indicate skepticism.
2. Body Language: Notice the gestures; open hand movements can suggest openness and positivity. If the individual is using their hands expressively, it may indicate engagement with the topic.
Pay attention posture; an upright and relaxed posture can signal confidence and positivity, while slouching might suggest disinterest.
### Language Cues ####
1. Emotionally Charged Words: The phrase "surprisingly decent cast" carries positive connotation. The word "surprisingly" suggests an unexpected positive outcome, which can amplify the sentiment.
The use of "decent" indicates favorable opinion, suggesting that the speaker is plcasantly surprised by the quality.
Impact Statement
This paper presents work whose goal is advance the field of Multimodal Sentiment Analysis and Multimodal Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations can be found in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper reports the model structure required for reproduction as well as the hyperparameter settings. In addition, the code will be available GitHub ASAP.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: The datasets used in the paper are open datasets and can accessed by anyone upon request. The code will be available GitHub ASAP.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/guides/Coees Stteess/CooSSubbisiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: //wips.cocc httppscccppuull///geeee wet.guudde/eeeaaeeieee
ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Optimizer selection, data split, hyperparameters, and other details are de- scribed.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the tables, we report the results with mta
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report this information Section Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: ttttp:://neuris... http:://ourrss..cpppbb.b ctthiiissceellleeeece ines?
Question:
Answer: [Yes]
Justification:
This study meets the requirements of ethical norms.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe this in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used data has been cited in the paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The dataset used is a public dataset, and the code will be available on GitHub after the paper accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve above problems.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve the above issues and the datasets used are all publicly available datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
32

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The framework proposed in the paper includes generating prompts with MILMs to guide the training of task-specific models. The relevant details have all been described in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httpp///eerripssccoooffeerreaaee /c//oonfeeecccc//225///.. for what should or should not be described.
33


Reviews:
[{"review_id": "QtvqrPti8w", "reviewer": "Reviewer_RVt6", "summary": "This paper proposes a distillation-based method to introduce multimodal LLM (MLLM) to the multimodal sentiment analysis (MSA) task. The teacher model is trained with the context prompt from MLLM, called conditional alignment in this paper. Finally, experiments are conducted based on the three popular MSA datasets.", "questions": "1. For the MSA task, the THUAIR repo. is a good reference, but it is not all. It is necessary to discuss more papers published after 2023 so that this paper claims the right thing.\n\n2. How to obtain all the prompts when training the teacher model? Is it obtained once, or sample and generate the prompt for each data point?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. The motivation and insight of this paper are interesting. MLLM is a popular technique; how to introduce MLLM to the MSA task is a good motivation. \n\n2. The experiments are conducted based on popular datasets. CMU-MOSEI, CMU-MOSI, and SIMS are the three popular MSA datasets.\n\n3. The paper is well-organized and well-written.\n\nCons:\n\n1. This paper misses many related works. MAG-BERT[1], DMD[2], DLF[3], TCAN[4], MMML[5]. The performance of this paper is not SOTA. Although the SOTA result is not necessary for publication at the top-tier conference, it is not appropriate to claim that this paper achieves the SOTA performance. All the following papers should be included when discussing MSA tasks. If not, please show reasonable reasons for that. The MMML used another encoder for text; for fair comparison, it is better to use the same encoder for text.\n\n[1]  \"Integrating multimodal information in large pretrained transformers.\" ACL 2020.\n\n[2] \"Decoupled multimodal distilling for emotion recognition.\" CVPR 2023.\n\n[3]  \"DLF: Disentangled-language-focused multimodal sentiment analysis.\" AAAI 2025.\n\n[4] \"TCAN: Text-oriented cross attention network for multimodal sentiment analysis.\" Arxiv 2025\n\n[5] \"Multimodal multi-loss fusion network for sentiment analysis.\" arXiv 2023\n\n2. For teacher-student models, or distillation-based models, efficient metrics are generally needed. Such as the model size, inference time or FLOPs.\n\n3. There is no visualization analysis to show the unique contribution of this paper.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in the rebuttal. However, I still believe the paper somewhat overstates its contributions. In particular, the comparison with MMML is not reported. If the paper intends to claim a state-of-the-art (SOTA) result, it should demonstrate superior performance compared to MMML; without such evidence, the SOTA claim is not accurate.\n\nThat said, achieving SOTA performance is not a prerequisite for publication. In my view, the most important aspects are the uniqueness of the insights and the strength of the experimental support. Therefore, I recommend that the authors revise or qualify the SOTA claim if the results do not surpass those of MMML.\n\nSince the reported results are not better than those of MMML, even when considering the best run, I still regard the paper as borderline acceptable at most, despite the fact that most of my earlier concerns have been addressed.", "rebuttal": "# Response to Reviewer RVt6\n\n**Response to Cons 1**\n\nThank you for the suggestion. **It should be noticed that five runs results** **are** **usually lower than searched best-run results.** We have realized that including more recent methods can strengthen our claims. **As shown in the Table 1 below**, we compare methods without open-sourced code (i.e., TCAN, DeepMLF) using reported best-run results. For those with open-sourced code (*i.e.,* MAG-BERT, DMD, DLF), since the limited time of rebuttal, we will reproduce five-run averages of these methods under the same setting on all datasets in the revision. Obviously, the results show our MMSLF achieves competitive/better performance in many metrics when compared with task-specific models on MOSI, MOSEI and SIMS datasets. For example, both teacher and student outperform the work DLF on MOSI, while DLF performs better on MOSEI. Compared to DeepMLF (task-specific MLLM), the student underperforms on several metrics. We believe this is primarily due to the fact that part of the knowledge transfer from the MLLM and the teacher may be lost due to the student’s limited capacity. Despite this, our method achieves competitive performance, demonstrating a favorable trade-off. We will also discuss the recent/contemporaneous works (*i.e.,* DLF and TCAN) in the revised version in section relevant work. \n\nIn addition, regarding MMML, since it employs stronger encoders for text and audio feature extraction, we are working toward a fair comparison under a consistent setting. Specifically, we adopt RoBERTa for text encoding and HuBERT for audio encoding, and we will report the average performance over three runs. **These experiments are currently in progress, and we will update the results on OpenReview once they are available.**\n\nTable 1. Performance comparison. **Bold** results indicates the optimal result.\n| | | | | | \n| ----------------- | -------------- | -------------- | -------------- | -------------- |\n|**SIMS**|||||\n|Method|Acc-2|F1|MAE|Corr|\n| DeepMLF (2025-04, Task-specific MLLM)| 82.75| 83.15| **0.353**|**0.729**|\n| MMSLF-Teacher (Best Run)|**84.25** | **84.16**|0.369|0.693|\n| MMSLF-Student (Best Run)|83.37| 83.51| 0.367|0.681|\n| MMSLF-Teacher (Five Runs)|83.06±0.95| 84.06±0.43|0.370±0.50|0.690±0.80|\n| MMSLF-Student (Five Runs|81.40±1.58| 81.85±1.41|0.382±1.39|0.662±1.26|\n|**MOSI**|||||\n|Method|Acc-2|F1|MAE|Corr|\n| DeepMLF (2025-04, Task-specific MLLM)| -/85.60 | -/85.58| **0.692**  | 0.811|\n| MAG-BERT (2020, Task-specific) | -/86.10| -/86.00| 0.712 | 0.796|\n| DMD (2023, Task-specific) | -/83.23| -/83.29| 0.752| - |\n| DLF (2025-02, Task-specific)   | -/85.06| -/85.04 | 0.731| 0.781|\n| TCAN (2025-04, Task-specific)  | -/86.28 | -/86.15|0.714| 0.797|\n| MMSLF-Teacher (Best Run)| **86.15/87.80** | **86.17/87.89**| 0.700| **0.812**  |\n| MMSLF-Student (Best Run)| 84.55/87.04| 84.73/87.14| 0.701| 0.795|\n| MMSLF-Teacher (Five Runs)| 85.05±0.66/86.61±0.69 | 85.15±0.66/86.69±0.69 | 0.734±1.46 | 0.797±0.60 |\n| MMSLF-Student (Five Runs)| 83.62±0.91/85.37±1.00 | 83.68±0.96/85.50±0.96 | 0.746±1.63 | 0.775±1.10 |\n|**MOSEI**|||||\n|Method|Acc-2|F1|MAE|Corr|\n| DeepMLF (2025-04, Task-specific MLLM) | -/**87.15** | -/87.10| **0.499**  | **0.804**  |\n| MAG-BERT(2020, Task-specific) |-| -| -| -|\n| DMD (2023, Task-specific)| -/84.62| -/84.62| 0.543| - |\n| DLF (2025-02, Task-specific)| -/85.42| -/85.27| 0.536| 0.764 |\n| TCAN (2025-04, Task-specific)| -/86.27| -/86.17| 0.532| 0.774 |\n| MMSLF-Teacher (Best Run)| **85.47/87.15** | **85.70/87.26**| 0.530      | 0.784      |\n| MMSLF-Student (Best Run)| 84.48/85.36  | 84.85/85.43 | 0.542      | 0.755      |\n| MMSLF-Teacher (Five Runs)| 85.08±0.36/86.62±0.75 | 85.55±0.24/86.71±0.71 | 0.539±1.06 | 0.773±1.51 |\n| MMSLF-Student (Five Runs)| 83.96±0.38/84.67±0.27 | 84.20±0.48/84.74±0.28 | 0.548±0.41 | 0.747±0.51 |\n| | | | | \n\n**Response to Cons 2**\n\nThank you for the suggestion. We provide a detailed comparison in **Table 2**. Notably, our student model achieves competitive performance (F1: 81.85±1.41 on SIMS) with only **0.82M parameters**, **8.61 GFLOPs**, and **6.39s** test-time inference. In contrast, Gemini-2.0-Flash and GPT-4o-mini require larger parameters and >22min inference time. These results demonstrate that our method offers a trade-off between performance and computational cost. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs are relatively higher. This is because we do not compress the input sequences' length as the prior methods [1, 2]. With further optimization (*e.g.,* sequence dimension reduction), the computational cost of MMSLF can be further reduced.\n\nTable 2. Comparison of Efficiency (Evaluate on SIMS dataset). **Bold** results indicates the optimal result. The experiments were conducted on a Computer with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40 GPU.\n\n|||||||\n| --------------- | -------------- | --------------- | ------------------------------------ | ---------------------- | ----------------------- |\n| **Models**      | **Parameters** | **GFLOPs**      | **Inference Time on SIMS Test Set**  | **F1** **on** **SIMS** | **MAE** **on** **SIMS** |\n| GPT-4V          | >7B            | -               | >30min (affected by network quality) | 81.24                  | -                       |\n| GPT-4o-mini     | >7B            | -               | >27min (affected by network quality) | 82.51                  | 0.453                   |\n| Gemini2.0-Flash | >7B            | -               | >22min (affected by network quality) |  **84.69**| 0.381                   |\n| TFN             | 35.63M         | **0.10** | **3.46s**                                | 77.83±1.62             | 0.434±1.12              |\n| MISA            | 21.66M         | 7.32     | 12.32s                               | 76.54±1.67             | 0.451±1.83              |\n| Self-MM         | **0.38M**      | 6.66     | 11.40s                               | 77.72±0.68             | 0.418±1.05              |\n| TETFN           | 1.53M          | 6.72     | 26.57s                               | 79.34±0.52             | 0.422±1.30              |\n| ALMT            | 2.60M          | 7.00     | 16.08s                               | 80.17±0.60             | 0.421±0.69              |\n| MMSLF-Teacher   | 2.54M          | 96.16    | 12.31s + 27min                       | 84.06±0.43| **0.370±0.50**          |\n| MMSLF-Student   | 0.82M          | 8.61     | 6.39s                            | 81.85±1.41             | 0.382±1.39              |\n|||||||\n\n**Response to Cons 3**\n\nThank you for the suggestion. We include a visualization in **Appendix Section D.6** and **Appendix Figure 4** that illustrates the alignment learned by the student model under the guidance of MLLMs. The results show that prompts from MLLMs can improve the student’s multimodal alignment. As images and hyperlinks are not supported in the rebuttal, we will include more qualitative cases in the revised version to further highlight the unique contributions of our method.\n\n**Response to Q1**\n\nPlease see the **Response to Cons 1**.\n\n**Response to Q2**\n\nThank you for the question. Considering both cost and performance, we obtain all prompts once before training the teacher model. We also experimented with generating three prompts per sample and randomly sampling one during training on SIMS dataset, but found that this way lead to higher cost without significant performance improvement. **As shown in Table 3 below**, the \"obtain once prompt\" setting achieves comparable or even better results. Moreover, we observe that the “sampling from three prompts” strategy introduces noticeably higher variance for MMSLF-Teacher, especially in MAE and Corr (e.g., 2.71 vs. 0.50 for MAE std), suggesting that sampling different prompts may introduce inconsistent guidance and lead to unstable training. However, for MMSLF-Student, the variance across runs is relatively small in both settings (e.g., F1 std: 1.41 vs. 0.66), indicating that the student model is less sensitive to prompt sampling. This is likely because the student learns from the teacher’s distilled representations and attention patterns, rather than directly using the prompts. \n\nTable 3. Effect of prompt sampling methods on performance. **Bold** results indicates the optimal result.\n\n| | | | | |\n| ------------------------------------------- | -------------- | -------------- | -------------- | -------------- |\n| Method | Acc-2          | F1             | MAE            | Corr           |\n| MMSLF-Teacher (Sampling from Three Prompts) | **83.57±1.73** | 83.23±1.30     | **0.370±2.71** | 0.682±4.69     |\n| MMSLF-Student (Sampling from Three Prompts) | 81.05±0.66     | 81.18±0.66     | 0.385±0.92     | 0.667±0.87     |\n| MMSLF-Teacher (Obtain Once)                 | 83.06±0.95     | **84.06±0.43** | **0.370±0.50** | **0.690±0.80** |\n| MMSLF-Student (Obtain Once)                 | 81.40±1.58     | 81.85±1.41     | 0.382±1.39     | 0.662±1.26     |\n| | | | | |\n\n**Reference**\n\n[1] MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In ACM MM 2020.\n\n[2] Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP 2023."}, {"review_id": "TIQIP5NSYu", "reviewer": "Reviewer_EAd9", "summary": "The authors introduces MMSLF (mLLM-guided sentiment analysis) where they feed mLLM output guidance as additional information into a (pretty much) standard sentiment analysis model to improve performance. Further, this model is used as a teacher to distill information into a sentiment analysis model that does not require mLLM input, thus significantly saving computational resources at inference. Performance for the mLLM+sentiment analysis model is close to SOTA for SIMS (especially), MOSI, and MOSEI datasets, while the performance for the teacher is still pretty good. The authors run a comparison with most of the popular MM fusion models/algorithms and perform a comprehensive ablation study.", "questions": "1. Could you use this on top a very strong baseline model - like the one mentioned above, how dependent is your mLLM prompt mixing approach on your AVT fusion architecture \n2. Prompt Sensitivity and Robustness: Given the significant performance difference between GPT-4o-mini and Gemini-2.0-Flash prompts, how sensitive is the method to prompt variations? \n3. Computational Cost Analysis: While you mention cost-effectiveness, can you provide concrete numbers comparing the total computational cost of your approach - also vs other SOTA methods like the ones i mentioned above\n4. How relevant is your approach for other multimodal tasks?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novel and Practical Approach: the idea of leveraging  mLLM knowledge via text output and then distilling this information in a teacher -student framework is novel.\n2. Comprehensive Evaluation: The paper includes thorough experiments across three datasets, multiple baselines (both traditional and mLLM-based), and extensive ablation studies examining each component's contribution.\n3. Technical Correctness: The authors do all the reasonable things without being especially novel in their MM architecture. The distillation is the most interesting part. \n4. Clear Motivation and Results: The cost-effectiveness argument is compelling and practical - the contribution of mLLM output is clear via the ablation study. Overall, persuasive results\n\nWeaknesses:\n1. The abstract and intro do little service to the paper. The main idea is not clearly exposed, i.e., explain what is the information that you are adding and in what format, explain the novelty of the paper in the distillation etc. Conclusions are a bit better. Still thorough rewrite is needed.\n2. You should at least explain where you sit in terms of mLLM SOTA, e.g., see this recent paper https://arxiv.org/pdf/2504.11082  -  in general your results are good but you are giving the impression that you are SOTA which is not true. \n3. Your literature review is ok and the methods you describe are solid - but i would like to see more focus on papers that introduce new ideas rather than having a ton of top performers in your studies, it would be nice to mabe also review ideas from https://lab-msp.com/MSP-Podcast_Competition/IS2025/\n4. The fusion and in general the architecture is of little novelty; however, the teacher-model part is interesting\n5. mLLM dependency is a concern - how you engineer the mLLM output is super-important and it is hard to tune as one can see from results from different mLLLMs\n6. Without the mLLM prompts results are a low - it is unclear to me if this a  weakness of the architecture or lack of tuning - please explain\n7. The method is not modular enough to be applied on top of any existing sentiment analysis model at its current form", "was_revised": true, "final_justification": "The additional experiments provided by the authors help resolve many of my concerns.", "rebuttal": "# Response to Reviewer EAd9\n\n**Response to W1**\n\nWe sincerely thank your feedback. Since the submission was only 9 pages long, some details may not be included. In the revised paper, we will: **(1)** Explain the information and format contained in the prompts. **(2)** Emphasize the innovation of the framework. **(3)** Clarify our ideas more clearly, which refer to using general knowledge from large-scale language models to guide the training of MSA models for specific tasks. Please let us know if you have any more specific suggestions. Thank you very much!\n\n**Response to W2**\n\nThank you for bringing this important recent work to our attention. **(1)** We missed this concurrent work (arXiv:2504.11082), which was released just one month before the NeurIPS submission deadline. We will add comparisons in our revision. **(2) As shown in Table 1 in Response to Reviewer RVt6**, we include the methods for comparison. We compare methods without open-sourced code (i.e., TCAN, DeepMLF) using reported best-run results. For those with open-sourced code (i.e., MAG-BERT, DMD, DLF, MMML), since the limited time of rebuttal, we will reproduce five-run averages under the same setting on all datasets in the revision. Obviously, the results show our MMSLF achieves competitive/better performance in many metrics compared with task-specific models. For example, both teacher and student outperform the work DLF on MOSI, while DLF performs better on MOSEI. Compared to DeepMLF (MLLM), the student underperforms on several metrics. We believe this is due to the fact that part of the knowledge transfer from the MLLM and the teacher may be lost due to the student’s limited capacity. Despite this, our method achieves competitive performance, demonstrating a favorable trade-off. We will discuss these works in the revised version. \n\n**Response to W3**\n\nWe sincerely thank you for your sharing and suggestions! In our revision, we will: **(1)** Expand the literature review to discuss novel conceptual contributions and emerging ideas in the field. **(2)** Include deeper discussion of the new insights we discovered, incorporating the key points raised in our responses to all three reviewers. \n\n**Response to W4**\n\nWe sincerely thank you for your recognition of our teacher model design. Regarding the student architecture, we would like to clarify: **(1)** While the pipeline is relatively straightforward, this design choice was made based on empirical findings that this structure can achieve distillation efficiency for our specific task. The simplicity of the architecture provides a clearer demonstration of our method's effectiveness by minimizing confounding factors from complex architectural innovations. **(2) As shown in Table 1 below**, we conducted preliminary experiments attempting to directly integrate MLLM prompts into the existing ALMT model. Obviously, ALMT-Teacher achieved the best performance, but ALMT-Student performed worse than MMSLF-Student. We believe that architectural simplicity is not merely a limitation but rather an informed design choice that balances effectiveness with training stability. **(3) As mentioned in lines 56-57 in the paper**, we believe the contribution lies in our whole framework that explores using the general knowledge of MLLMs to guide the training of task-specific MSA models, offering a new insight into applying general MLLMs to improve MSA.\n\nTable 1. Generality of the proposed framework. **Bold** results indicates the optimal result.\n\n||||||\n|-|-|-|-|-|\n|**SIMS**|||||\n|Method| Acc-2|F1|MAE|Corr|\n|ALMT-Teacher|**84.20±0.57**|**84.45±0.81**|**0.363±0.76**|**0.711±1.50**|\n|ALMT-Student|79.87±1.81|80.58±1.05|0.418±2.15|0.587±3.97|\n|MMSLF-Teacher|83.06±0.95|84.06±0.43|0.370±0.50|0.690±0.80|\n|MMSLF-Student|81.40±1.58|81.85±1.41|0.382±1.39|0.662±1.26|\n|**MOSI**|||||\n|Method| Acc-2|F1|MAE|Corr|\n|ALMT-Teacher|**86.56±0.68**/**88.02±0.67**|**86.63±0.69**/**88.06±0.68**|**0.677±0.57**|**0.834±0.46**|\n|ALMT-Student|83.26±0.41/85.43±0.14|83.38±0.31/85.52±0.15|0.720±0.54|0.784±0.28|\n|MMSLF-Teacher|85.05±0.66/86.61±0.69|85.15±0.66/86.69±0.69|0.734±1.46|0.797±0.60|\n|MMSLF-Student|83.62±0.91/85.37±1.00|83.68±0.96/85.50±0.96|0.746±1.63|0.775±1.10|\n||||||\n\n**Response to W5**\n\nThank you for raising this concern. The choice of MLLM significantly impacts performance through the quality of generated guidance signals. **As discussed in Section 4.8**, our analysis reveals that Gemini-2.0-Flash produces less accurate and informative prompts compared to GPT-4o-mini, reflecting its weaker capabilities in sentiment-aware multimodal understanding. However, **(1)** We provide our detailed prompt templates in our appendix. **(2)** As general MLLMs continue to advance rapidly, we believe that this dependency will become less problematic. We will discuss this in our revision.\n\n**Response to W6**\n\nThis phenomenon is caused by the architectural constraints. **(1)** To effectively learn from MLLM prompts, our teacher model requires sufficient capacity. **As shown in Table 8 in the Appendix,** we set it to 12 layers. When mLLM prompts are removed, this relatively large teacher model suffers from severe overfitting on the limited MSA training data, leading to poor generalization on test sets. **(2)** While the student model is parameter-efficient, it lacks the representational capacity to achieve competitive performance without teacher guidance. Therefore, our current design represents a carefully calibrated trade-off between performance and parameters. We will clarify this in the revised version.\n\n**Response to W7**\n\nWhile our current implementation has constraints that limit direct plug-and-play application, we believe this represents an important first step toward a broader research direction. **(1)** Our idea can be adapted to existing MSA models. For example, we tested this on a strong baseline ALMT. Specifically, we concatenate prompts with datasets' text inputs followed by dimensionality reduction. **As shown in Table 1 above**, although the final performance was also good, we observed slower convergence and suboptimal performance compared to our proposed MMSLF-Student model. **(2)** Despite these constraints of direct plug-and-play application, our work demonstrates the potential of utilizing MLLMs to improve task-specific model learning. We aim to validate this research direction and offer new insight into applying general MLLMs to improve MSA. We will discuss this in the revised paper.\n\n**Response to Q1**\n\nWe attempted to apply our framework to the recent work (arXiv:2504.11082). However, no open-source implementation was available for this method. Therefore, we present experimental results using ALMT, a strong baseline in the MSA domain. More detailed discussion can be found in **our response to W4, W7 and Table 1 above.**\n\n**Response to Q2**\n\n**As shown in Table 2 below**, we evaluated our method using different prompt templates to assess robustness. We selected GPT-4o-mini and Gemini-2.0-Flash because these larger MLLMs typically exhibit stable outputs across prompt variations, which is essential for reliable knowledge distillation. Our experimental results show minimal performance variance across different prompt formulations within the same MLLM, confirming that advanced large-scale MLLMs generate consistent outputs regardless of reasonable prompt variations. This stability validates our framework's robustness to prompt engineering choices. However, the more significant factor is the selection of an appropriate teacher mLLM with strong domain-specific capabilities. The performance difference between GPT-4o-mini and Gemini-2.0-Flash may stem from their varying abilities to generate accurate and informative prompts for sentiment-centered multimodal tasks, rather than sensitivity to prompt variations.\n\nTable 2. Comparison of Prompt Sensitivity and Robustness. **Bold** results indicates the optimal result.\n|||||||\n|-|-|-|-|-|-|\n|**SIMS**|||||\n|Method|Prompt |Acc-2|F1|MAE|Corr|\n|MMSLF-Teacher|Prompt 1 (Default, GPT-4o-mini)|**83.06±0.95**|**84.06±0.43**|0.370±0.50|0.690±0.80|\n|MMSLF-Teacher|Prompt 2 (GPT-4o-mini)|81.23±5.94|83.93±1.16|**0.346±1.12**|**0.716±1.47**|\n|MMSLF-Teacher|Prompt 3 (GPT-4o-mini)|80.04±5.37|82.51±0.71|0.372±1.20|0.687±2.72|\n|MMSLF-Teacher|Prompt 4 (Gemini-2.0-Flash)|81.09±0.23|81.09±0.29|0.377±0.73|0.686±1.53|\n|MMSLF-Teacher|Prompt 5 (Gemini-2.0-Flash)|77.90±7.03|82.87±1.21|0.355±2.25|0.704±3.08|\n|MMSLF-Teacher|Prompt 6 (Gemini-2.0-Flash)|71.42±4.11|81.95±0.07|0.384±1.47|0.658±1.82|\n|**MOSI**|||||\n|Method|Prompt |Acc-2|F1|MAE|Corr|\n|MMSLF-Student|Prompt 1 (Default, GPT-4o-mini) |**81.40±1.58**|**81.85±1.41**|0.382±1.39|0.662±1.26|\n|MMSLF-Student|Prompt 2 (GPT-4o-mini)|**81.40±1.09**|81.65±0.96|0.394±1.06|0.667±0.45|\n|MMSLF-Student|Prompt 3 (GPT-4o-mini)|80.74±0.37|80.92±0.74|0.393±1.80|0.664±1.34|\n|MMSLF-Student|Prompt 4 (Gemini-2.0-Flash)|80.00±0.41|80.11±0.54|0.422±0.96|0.627±1.70|\n|MMSLF-Student|Prompt 5 (Gemini-2.0-Flash)|81.18±0.97|81.15±0.89|0.387±1.92|0.670±2.69|\n|MMSLF-Student|Prompt 6 (Gemini-2.0-Flash)|81.05±0.45|81.14±0.37|**0.381±0.96**|**0.676±1.59**|\n|||||||\n\n**Response to Q3**\n\n**As mentioned in Section Introduction**, cost-effectiveness is mainly compared with MLLMs. Because directly applying MLLMs to MSA is costly. To make a more comprehensive comparison, we have also provided the parameter comparison with the task-specific model. As for the work (arXiv:2504.11082) you mentioned, since there is no open-source code available, we are unable to provide a comparison. Due to the limitation on the number of rebuttal words, the detailed results and discussion can be found in **Response to Cons2 and Table 1 in Response to Reviewer RVt6**. \n\n**Response to Q4**\n\nWe have extended our framework to Emotion Recognition and Sentiment Analysis in subsequent work, with preliminary results demonstrating that appropriately selected MLLMs can effectively guide task-specific model learning across different domains."}, {"review_id": "yOOtuxnxTz", "reviewer": "Reviewer_ctoX", "summary": "The paper introduces MLLM-Guided Multimodal Sentiment Learning Framework (MMSLF), a framework that leverages multi-modal large language models (MLLM) to improve the performance of task-specific multimodal sentiment analysis (MSA) models in a teacher-student setup.  The teacher uses information from a powerful MLLM model (specifically, GPT-4o-mini in this study) to generate context-aware prompts that help the student focus on better alignments between different modalities of the data. Experimental results on 3 benchmark datasets show that the proposed framework leads to competitive task-specific models.", "questions": "The authors show that the results obtained with  GPT-4o-mini  are still better than those obtained with Gemini-2.0-Flash. Some insights into why this happens would be useful.", "limitations": "The results of  Gemini-2.0-Flash are the best overall. A discussion of the tradeoff between computational resources and performance is lacking.\n \nIt is not clear what the assumptions of different baselines might be and what data modalities they work with. That might explain some of the results.\n\nThere is no statistical significance analysis.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The proposed framework leverages powerful MLLMs to improve the results of smaller task-specific fine-tuned models. To do that, it makes sense of a teacher-student framework in an interesting way, where the teacher is equipped with an MLLM and generates prompts that help the student align multimodal data and focus on what may be more representative features for the multimodal sentiment analysis task. \n\nThe authors run experiments on three datasets and compare their results with the results of a large set of baselines, including MLLMs and also task-specific fine-tuned models. \n\nExperimental results show that the proposed approach has the ability to improve the results of prior fine-tuned models. \n\nThe authors motivate the proposed approach by saying that the use of MLLMs for the task of multimodal sentiment analysis may not be justified, given the increase in computational costs required for good performance. However, an MLLM is still used for the teacher. While the student does not directly use output from an MLLM, its performance lags significantly behind that of the teacher and also the performance of other MLLMs (specifically, Gemini-2.0-Flash). It is not clear that the drop in performance is acceptable. The authors do not do any analysis in terms of computational resources used by different models, despite this being part of the motivation for the work. \n\nThe authors perform 5 runs for each experiment and report average metrics together with standard deviation. However, there is no statistical significance analysis and it's not clear that the student model is indeed better than most of the other fine-tuned models. \n\nAs a side note, there is no discussion of the specific multimodal information that each of the baselines are using. Gemini-2.0-Flash is able to use all the modalities of the data and gives the best results overall, while the MLLM used in the teacher doesn't seem to support audio analysis. The authors show that the results obtained with  GPT-4o-mini  are still better than those obtained with Gemini-2.0-Flash but it's not clear why. \n\nThe captions of the tables are not self-explanatory.", "was_revised": true, "final_justification": "The authors addressed most of my comments and suggestions, and they plan to update the paper accordingly, to the extent the page limit allows. I increased my score.  However, the significance of the contribution is still somewhat weak in my view.", "rebuttal": "# Response to Reviewer ctoX\n\n**Response to W1**\n\nThank you for your valuable comments. **(1) As shown in Table 1**, in the revised manuscript, we will add the following which quantitatively compares advanced MLLMs and our MMSLF in terms of parameter count, GFLOPs, and inference time. Obviously, our student model achieves competitive performance (F1: 81.85±1.41 on SIMS) with only **0.82M parameters**, **8.6 GFLOPs**, and **6.39s** test-time inference. In contrast, Gemini-2.0-Flash and GPT-4o-mini require larger parameters and >22min inference time. These results demonstrate that our method offers a trade-off between performance and computational cost. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs are relatively higher. This is because we do not compress the input sequences' length as the prior methods [1, 2]. With further optimization (*e.g.,* sequence dimension reduction), the computational cost of MMSLF can be further reduced.  **(2)** We can see that the Student’s performance is lower than that of the Teacher, GPT‑4o‑mini, and Gemini‑2 Flash because it omits MLLM‑generated prompts and inevitably loses some information during distillation. However, this design itself confirms the feasibility of using general MLLMs to improve a much smaller model’s performance. **(3)** Although there remains a gap between the Student and GPT‑4o‑mini or Gemini‑2.0-Flash, the student still surpasses other large models such as GPT‑4V. In summary, we believe our motivation is sound and the corresponding exploration meaningful, and we will further clarify this in the revised manuscript. In addition, due to the limitation on the number of rebuttal words. Overall, these results demonstrate that our method offers a trade-off between performance and computational cost. \n\nTable 1. Comparison of Efficiency (Evaluate on SIMS dataset). **Bold** results indicates the optimal result. The experiments were conducted on a Computer with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40 GPU.\n\n|||||||\n|-|-|-|-|-|-|\n| Models |Parameters|GFLOPs| Inference Time on SIMS Test Set |F1 on SIMS | MAE on SIMS|\n| GPT-4V          | >7B            | -               | >30min (affected by network quality) | 81.24                  | -                       |\n| GPT-4o-mini     | >7B            | -               | >27min (affected by network quality) | 82.51                  | 0.453                   |\n| Gemini2.0-Flash | >7B            | -               | >22min (affected by network quality) |**84.69**| 0.381                   |\n| TFN             | 35.63M         | **0.106288368** | **3.46s**                                | 77.83±1.62             | 0.434±1.12              |\n| MISA            | 21.66M         | 7.327169664     | 12.32s                               | 76.54±1.67             | 0.451±1.83              |\n| Self-MM         | **0.38M**      | 6.657025504     | 11.40s                               | 77.72±0.68             | 0.418±1.05              |\n| TETFN           | 1.53M          | 6.720118224     | 26.57s                               | 79.34±0.52             | 0.422±1.30              |\n| ALMT            | 2.60M          | 7.004771456     | 16.08s                               | 80.17±0.60             | 0.421±0.69              |\n| MMSLF-Teacher   | 2.54M          | 96.159077376    | 12.31s + 27min |84.06±0.43| **0.370±0.50**          |\n| MMSLF-Student   | 0.82M          | 8.610916288     | 6.39s| 81.85±1.41             | 0.382±1.39              |\n|||||||\n\n**Response to W2**\n\nThank you for pointing this out. **As shown in Table 2 below,** we have added an experiment: on the SIMS dataset we conducted two‑tailed *t*-tests between the Student model and the best task‑specific baseline. The resulting *p*-values are MAE = 0.0029, Corr = 0.00022, Acc‑2 = 0.095, and F1 = 0.066. Under the conventional 0.05 threshold, the Student’s improvements on MAE and Corr are statistically significant, while the gains on ACC‑2 and F1 show the same positive trend. Because of time and rebuttal‑length constraints, we will include a more detailed comparison on all datasets in the revised manuscript.\n\nTable 2. Two‑tailed *t*-tests between the Student model and the ALMT on SIMS dataset.\n\n||||||\n|-|-|-|-|-|\n| Models           | p-value (MAE) | p-value (Corr) | p-value(Acc-2) | p-value (F1) |\n| MMSLF-Student & ALMT | 0.0029                        | 0.00022                | 0.095                   | 0.066                        |\n||||||\n\n**Response to W3**\n\nThank you for raising this point. (1) In Section 4.8, we mentioned that \"the prompts may contain more misleading information, which may result in poorer performance when using prompts generated by Gemini-2.0-Flash for model training as discussed in Section 4.7. (2) We have provided one case (you can also refer to Case 1 in Appendix Figure 6), and the descriptions generated by Gemini sometimes contain errors. (3) **As shown in Table 3 below**, we use different templates to generate prompts and then use them to train the model. The results show that minimal performance variance across different prompt formulations within the same MLLM, confirming that advanced large-scale MLLMs generate consistent outputs regardless of reasonable prompt variations. Therefore, we believe this might be due to the fact that Gemini‑2.0- Flash was trained on comparatively less fine‑grained affective data, limiting its ability to craft sentiment‑focused multimodal descriptions. We will include this discussion in the revised manuscript.\n\n> Case 1 in Appendix Figure 6:\n>\n> Sentiment Label: Positive\n>\n> Audio Prompts of Gemini-2.0-Flash: The speaker has a conversational tone that comes off as conversational.\n\nTable 3. Comparison of Prompt Sensitivity and Robustness. **Bold** results indicates the optimal result.\n\n|||||||\n|-|-|-|-|-|-|\n| **SIMS**      |                                 |                |                |                |                |\n| Method        | Prompt                          | Acc-2          | F1             | MAE            | Corr           |\n| MMSLF-Teacher | Prompt 1 (Default, GPT-4o-mini) | **83.06±0.95** | **84.06±0.43** | 0.370±0.50     | 0.690±0.80     |\n| MMSLF-Teacher | Prompt 2 (GPT-4o-mini)          | 81.23±5.94     | 83.93±1.16     | **0.346±1.12** | **0.716±1.47** |\n| MMSLF-Teacher | Prompt 3 (GPT-4o-mini)          | 80.04±5.37     | 82.51±0.71     | 0.372±1.20     | 0.687±2.72     |\n| MMSLF-Teacher | Prompt 4 (Gemini-2.0-Flash)     | 81.09±0.23     | 81.09±0.29     | 0.377±0.73     | 0.686±1.53     |\n| MMSLF-Teacher | Prompt 5 (Gemini-2.0-Flash)     | 77.90±7.03     | 82.87±1.21     | 0.355±2.25     | 0.704±3.08     |\n| MMSLF-Teacher | Prompt 6 (Gemini-2.0-Flash)     | 71.42±4.11     | 81.95±0.07     | 0.384±1.47     | 0.658±1.82     |\n| **MOSI**      |                                 |                |                |                |                |\n| Method        | Prompt                          | Acc-2          | F1             | MAE            | Corr           |\n| MMSLF-Student | Prompt 1 (Default, GPT-4o-mini) | **81.40±1.58** | **81.85±1.41** | 0.382±1.39     | 0.662±1.26     |\n| MMSLF-Student | Prompt 2 (GPT-4o-mini)          | **81.40±1.09** | 81.65±0.96     | 0.394±1.06     | 0.667±0.45     |\n| MMSLF-Student | Prompt 3 (GPT-4o-mini)          | 80.74±0.37     | 80.92±0.74     | 0.393±1.80     | 0.664±1.34     |\n| MMSLF-Student | Prompt 4 (Gemini-2.0-Flash)     | 80.00±0.41     | 80.11±0.54     | 0.422±0.96     | 0.627±1.70     |\n| MMSLF-Student | Prompt 5 (Gemini-2.0-Flash)     | 81.18±0.97     | 81.15±0.89     | 0.387±1.92     | 0.670±2.69     |\n| MMSLF-Student | Prompt 6 (Gemini-2.0-Flash)     | 81.05±0.45     | 81.14±0.37     | **0.381±0.96** | **0.676±1.59** |\n|||||||\n\n**Response to W4**\n\nThank you for pointing this out. In the revised manuscript we will improve the table caption. For example, stating the brief information and explanations of any symbols or abbreviations.\n\n**Response to Q1**\n\nPlease see the **Response to W3**. We will include a more detailed comparison in the revised manuscript.\n\n**Response to L1**\n\nThank you for this question. **As mentioned in Response to W1 and the corresponding table**, although Gemini‑2 .0-Flash delivers the advanced performance, it relies on an external 7 B‑parameter API whose end‑to‑end inference exceeds 22 min and is bounded by network latency. By contrast, with the guidance of the teacher during the training stage, our MMSLF‑Student achieve competitive performance with fewer parameters, fewer GFLOPs, and fewer inference time. Similar phenomena can also be observed in GPT-4o-mini. These results illustrate that our framework offers a new insight into real-world applications of MSA. \n\n**Response to L2**\n\nThanks for your suggestions! We will specifically mention this in the revised paper. Actually, all task‑specific models, as well as Gemini‑2.0‑Flash and Video‑LLaMA, take vision, audio, and language as inputs. GPT‑4o‑mini and GPT‑4V take vision and text as inputs because they do not support audio analysis. \n\n**Response to L3**\n\n**As shown in Response to W2**, we have added the significance test to compare with the suboptimal model ALMT. We will include a more detailed comparison in the revised manuscript.\n\n**Reference**\n\n[1] MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In ACM MM 2020.\n\n[2] Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP 2023."}, {"review_id": "bNRqh0sPqh", "reviewer": "Reviewer_Wv59", "summary": "This paper focuses on improving multimodal sentiment analysis over language, video & audio inputs and proposes a teacher-student framework that leverages the use of a Multimodal LLM in the teacher model.  The pretrained MLLM is used to generate context-aware prompts focusing on facets in the multimodal inputs that are sentiment-related. The prompts are utilized to conditionally align the different modalities of input (e.g. between video & language), with the aim of capturing aligned sentiment information across the modalities. The outputs of the conditional alignment modules are then fused together and used to predict a sentiment score.\nThe student model aims to mimic the teacher in terms of the conditional attention map, the outputs of the fusion block, and the regression (sentiment) score. \nComparisons with baselines on 3 datasets shows improved performance across several metrics. Experiments are further conducted to demonstrate the effect of the choice of MLLM in the teacher, as well as the different components of the objective function for the student module.", "questions": "1. Are the human annotations referred to in Line 217 directly obtained from the datasets, or was an additional study conducted? If so, could you please provide details on this?\n2. Could you please clarify the choice of modeling this as a regression task vs classification? Also, how were the regression scores converted to sentiment labels to compute F1 & Accuracy?\n3. Have any experiments been conducted on out of domain datasets? \n4. In Table 6, Gemini 2.0 Flash is shown to *underperform* w.r.t GPT 4o mini as the MLLM in the proposed approach. However, in Tables 1 & 2, on the same datasets, by directly using these MLLMs for the task, we find them to be at par for Accuracy & F1, with Gemini  *outperforming* GPT 4o for MAE & Correlation. Any clarifications/thoughts on this occurrence would be helpful to know.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written with clearly explained methodology & experimentation. The in-depth discussion in Section 3 is helpful to understand the details & motivations behind the architecture.\n2. Utilizing the MLLM for conditional alignment in the teacher module is a promising and useful strategy to attend to the sentiment cues across various modalities. Leaving it out of the student module enables the usage of the student at inference time to be less resource-intensive which is beneficial.\n3. The datasets chosen for experimentation were multilingual (1 Chinese & 2 English) which demonstrates the generalizability of the framework\n\nWeaknesses:\n1. The choice of modeling the problem as a regression task over classification over sentiment labels for the teacher-student framework needs some elaboration. Further, it was unclear how sentiment labels were obtained from the regression scores in order to compute Accuracy & F1 scores reported in Tables 1-6. \n2. The student module's performance needs improvement if the intention is to only use this module at inference time for computational ease. For all 3 datasets (Tables 1, 2, 3), the teacher module outperforms the student, as would be expected given the use of the MLLM in the teacher. In fact, in Tables 2 & 3, the student's performance is at par or below the baselines compared with. \n3. In Table 2, the teacher's performance (presumably using GPT 4o mini as the underlying MLLM given that this is found to be a more effective MLLM in Table 6) is below that of using GPT 4o mini directly for the problem, for all 4 metrics. Similarly, for Table 3, the teacher's performance is at par with the MLLM itself for 3 out of the 4 metrics. This raises questions on the requirement of the added parameters of the teacher module vs using the MLLM directly for the problem.", "was_revised": true, "final_justification": "The authors have provided additional details during the rebuttal which will improve the quality of the paper, if included. However, as mentioned during the rebuttal, I would like to remain at my original score at this time.", "rebuttal": "# Response to Reviewer Wv59\n\n**Response to W1**\n\nThank you for the question. We follow standard practice in MSA and formulate the task as regression because all three benchmark datasets (SIMS, MOSI, MOSEI) provide continuous sentiment scores (SIMS: [-1, 1], MOSI/MOSEI: [-3, 3]). As mentioned in Appendix B, these fine-grained labels support regression-based training. To compute classification metrics (Acc-2, F1), we adopted a threshold-based way which is consistent with previous studies [1, 2]. For example, scores >0 are treated as positive, ≤0 as non-positive (for binary classification). This method is widely used in the MSA studies.\n\n**Response to W2**\n\nThank you for the insightful comment. **As shown in Table 1 below**, although it is expected that the student may underperform the teacher (which uses MLLM prompts as input), the student achieves competitive results with much fewer parameters and significantly lower inference cost. For example, our student model achieves competitive performance (F1: 81.85±1.41 on SIMS) with only **0.82M parameters**, **8.61 GFLOPs**, and **6.39s** test-time inference. These results demonstrate the potential of our framework that make general knowledge of MLLMs to guide the training of task-specific MSA models, offering new insight into applying general MLLMs to improve MSA. In addition, it is also worth noting that although MMSLF has a small number of parameters, its GFLOPs are relatively higher. This is because we do not compress the input sequences' length as the prior methods [1, 2]. With further optimization (*e.g.,* sequence dimension reduction), the computational cost of MMSLF can be further reduced.\n\nTable 1. Comparison of Efficiency (Evaluate on SIMS dataset). **Bold** results indicates the optimal result. The experiments were conducted on a Computer with an AMD EPYC 7513 CPU and an NVIDIA Tesla A40 GPU.\n\n|||||||\n| --------------- | -------------- | --------------- | ------------------------------------ | ---------------------- | ----------------------- |\n| **Models**      | **Parameters** | **GFLOPs**      | **Inference Time on SIMS Test Set**  | **F1** **on** **SIMS** | **MAE** **on** **SIMS** |\n| GPT-4V          | >7B            | -               | >30min (affected by network quality) | 81.24                  | -                       |\n| GPT-4o-mini     | >7B            | -               | >27min (affected by network quality) | 82.51                  | 0.453                   |\n| Gemini2.0-Flash | >7B            | -               | >22min (affected by network quality) | **84.69**| 0.381                   |\n| TFN             | 35.63M         | **0.10** | **3.46s**                                | 77.83±1.62             | 0.434±1.12              |\n| MISA            | 21.66M         | 7.32     | 12.32s                               | 76.54±1.67             | 0.451±1.83              |\n| Self-MM         | **0.38M**      | 6.66     | 11.40s                               | 77.72±0.68             | 0.418±1.05              |\n| TETFN           | 1.53M          | 6.72     | 26.57s                               | 79.34±0.52             | 0.422±1.30              |\n| ALMT            | 2.60M          | 7.00     | 16.08s                               | 80.17±0.60             | 0.421±0.69              |\n| MMSLF-Teacher   | 2.54M          | 96.16    | 12.31s + 27min                       | 84.06±0.43 | **0.370±0.50**          |\n| MMSLF-Student   | 0.82M          | 8.61     | 6.39s                            | 81.85±1.41             | 0.382±1.39              |\n|||||||\n\n**Response to W3**\n\nThank you for the observation. The teacher model in our framework serves as a medium to transfer the general knowledge of MLLMs into a smaller student model. While the teacher leverages prompts from a powerful MLLM (e.g., GPT-4o-mini), some performance gap is expected due to information loss during knowledge transfer, which is common in teacher-student frameworks. The purpose of introducing the teacher is not to surpass the MLLM, but to bridge its capabilities into a more efficient, task-specific model. Ultimately, our student model achieves trade-off performance, validating the effectiveness of our idea.\n\n**Response to Q1**\n\nIt is directly obtained from the datasets. We will emphasize these points in the revised dataset introduction. Thank you for your question.\n\n**Response to Q2**\n\nThank you for the question. We model the task as regression because all three benchmark datasets (SIMS, MOSI, MOSEI) provide continuous sentiment annotations (SIMS: [-1, 1], MOSI/MOSEI: [-3, 3]), and regression enables better supervision for fine-grained sentiment intensity. As mentioned in Appendix B, these fine-grained labels support regression-based training. To compute classification metrics (Acc-2, F1), we adopted a threshold-based way which is consistent with previous studies [1, 2]. For example, scores >0 are treated as positive, ≤0 as non-positive (for binary classification). This method is widely used in the MSA studies.\n\n**Response to Q3**\n\nThank you for the question. In this work, we use preprocessed modality sequence provided by each dataset, which are commonly adopted in prior MSA methods. These feature representations vary across datasets in both dimension and format, making direct cross-dataset evaluation infeasible. We follow this setting to ensure fair comparison with existing baselines. \n\n**Response to Q4**\n\n**As shown in Table 2 below**, we evaluated our method using different prompt templates to assess robustness. We selected GPT-4o-mini and Gemini-2.0-Flash because these larger MLLMs typically exhibit stable outputs across prompt variations, which is essential for reliable knowledge distillation. Our experimental results show minimal performance variance across different prompt formulations within the same MLLM, confirming that advanced large-scale MLLMs generate consistent outputs regardless of reasonable prompt variations. This stability validates our framework's robustness to prompt engineering choices. However, the more significant factor is the selection of an appropriate teacher mLLM with strong domain-specific capabilities. The performance difference between GPT-4o-mini and Gemini-2.0-Flash may stem from their varying abilities to generate accurate and informative prompts for sentiment-centered multimodal tasks, rather than sensitivity to prompt variations. Therefore, we believe this might be due to that Gemini‑2.0- Flash was trained on comparatively less fine‑grained affective data, limiting its ability to craft sentiment‑focused multimodal descriptions. We will include this discussion in the revised manuscript.\n\nTable 2. Comparison of Prompt Sensitivity and Robustness. **Bold** results indicates the optimal result.\n|||||||\n| --------------- | -------------- | --------------- | ------------------------------------ | ---------------------- | ----------------------- |\n| **SIMS**      |                                 |                |                |                |                |\n| Method        | Prompt                          | Acc-2          | F1             | MAE            | Corr           |\n| MMSLF-Teacher | Prompt 1 (Default, GPT-4o-mini) | **83.06±0.95** | **84.06±0.43** | 0.370±0.50     | 0.690±0.80     |\n| MMSLF-Teacher | Prompt 2 (GPT-4o-mini)          | 81.23±5.94     | 83.93±1.16     | **0.346±1.12** | **0.716±1.47** |\n| MMSLF-Teacher | Prompt 3 (GPT-4o-mini)          | 80.04±5.37     | 82.51±0.71     | 0.372±1.20     | 0.687±2.72     |\n| MMSLF-Teacher | Prompt 4 (Gemini-2.0-Flash)     | 81.09±0.23     | 81.09±0.29     | 0.377±0.73     | 0.686±1.53     |\n| MMSLF-Teacher | Prompt 5 (Gemini-2.0-Flash)     | 77.90±7.03     | 82.87±1.21     | 0.355±2.25     | 0.704±3.08     |\n| MMSLF-Teacher | Prompt 6 (Gemini-2.0-Flash)     | 71.42±4.11     | 81.95±0.07     | 0.384±1.47     | 0.658±1.82     |\n| **MOSI**      |                                 |                |                |                |                |\n| Method        | Prompt                          | Acc-2          | F1             | MAE            | Corr           |\n| MMSLF-Student | Prompt 1 (Default, GPT-4o-mini) | **81.40±1.58** | **81.85±1.41** | 0.382±1.39     | 0.662±1.26     |\n| MMSLF-Student | Prompt 2 (GPT-4o-mini)          | **81.40±1.09** | 81.65±0.96     | 0.394±1.06     | 0.667±0.45     |\n| MMSLF-Student | Prompt 3 (GPT-4o-mini)          | 80.74±0.37     | 80.92±0.74     | 0.393±1.80     | 0.664±1.34     |\n| MMSLF-Student | Prompt 4 (Gemini-2.0-Flash)     | 80.00±0.41     | 80.11±0.54     | 0.422±0.96     | 0.627±1.70     |\n| MMSLF-Student | Prompt 5 (Gemini-2.0-Flash)     | 81.18±0.97     | 81.15±0.89     | 0.387±1.92     | 0.670±2.69     |\n| MMSLF-Student | Prompt 6 (Gemini-2.0-Flash)     | 81.05±0.45     | 81.14±0.37     | **0.381±0.96** | **0.676±1.59** |\n|||||||\n\n**Reference**\n\n[1] MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In ACM MM 2020.\n\n[2] Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP 2023."}]

Author rebuttal:

Meta-review: The paper introduces a teacher-student framework for multimodal sentiment analysis (MSA) that leverages multimodal large language models (MLLMs) to guide the training of efficient, task-specific models. Reviewers generally agree that the motivation is timely and the approach is novel, with comprehensive experiments and clear methodology. 

The framework demonstrates competitive performance and significant efficiency gains, especially in the student model, which is much less resource-intensive than direct MLLM inference. However, reviewers caution against overstating state-of-the-art (SOTA) claims, noting that the method does not consistently outperform all recent baselines across every metric. 

They also highlight the need for clearer discussion of the trade-off between performance and efficiency, and suggest improvements in presentation and statistical analysis. 

The authors responded thoroughly to these concerns, providing additional experiments, fairer comparisons, and statistical significance testing, and committed to further clarifying claims and improving the manuscript. Overall, the work is seen as a solid and practical contribution to efficient multimodal sentiment analysis, with most reviewer concerns satisfactorily addressed.
Decision: Accept (poster)