Title: Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Sentiment Analysis
Abstract: Multimodal Sentiment Analysis (MSA) aims to predict sentiment from diverse data types, such as video, audio, and language. Recent progress in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance across various tasks. However, in MSA, the increase in computational costs does not always correspond to a significant improvement in performance, raising concerns about the cost-effectiveness of applying MLLMs to MSA. This paper introduces the MLLM-Guided Multimodal Sentiment Learning Framework (MMSLF). It improves the performance of task-specific MSA models by leveraging the generalized knowledge of MLLMs through a teacher-student framework, rather than directly using MLLMs for sentiment prediction. First, the proposed teacher built upon a powerful MLLM (e.g., GPT-4o-mini), guides the student model to align multimodal representations through MLLM-generated context-aware prompts. Then, knowledge distillation enables the student to mimic the teacher’s predictions, thus allowing it to predict sentiment independently without relying on the context-aware prompts. Extensive experiments on the SIMS, MOSI, and MOSEI datasets demonstrate that our framework enables task-specific models to achieve state-of-the-art performance across most metrics. This also provides new insights into the application of general MLLMs for improving MSA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving multimodal sentiment analysis over language, video & audio inputs and proposes a teacher-student framework that leverages the use of a Multimodal LLM in the teacher model.  The pretrained MLLM is used to generate context-aware prompts focusing on facets in the multimodal inputs that are sentiment-related. The prompts are utilized to conditionally align the different modalities of input (e.g. between video & language), with the aim of capturing aligned sentiment information across the modalities. The outputs of the conditional alignment modules are then fused together and used to predict a sentiment score.\nThe student model aims to mimic the teacher in terms of the conditional attention map, the outputs of the fusion block, and the regression (sentiment) score. \nComparisons with baselines on 3 datasets shows improved performance across several metrics. Experiments are further conducted to demonstrate the effect of the choice of MLLM in the teacher, as well as the different components of the objective function for the student module.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written with clearly explained methodology & experimentation. The in-depth discussion in Section 3 is helpful to understand the details & motivations behind the architecture.\n2. Utilizing the MLLM for conditional alignment in the teacher module is a promising and useful strategy to attend to the sentiment cues across various modalities. Leaving it out of the student module enables the usage of the student at inference time to be less resource-intensive which is beneficial.\n3. The datasets chosen for experimentation were multilingual (1 Chinese & 2 English) which demonstrates the generalizability of the framework\n\nWeaknesses:\n1. The choice of modeling the problem as a regression task over classification over sentiment labels for the teacher-student framework needs some elaboration. Further, it was unclear how sentiment labels were obtained from the regression scores in order to compute Accuracy & F1 scores reported in Tables 1-6. \n2. The student module's performance needs improvement if the intention is to only use this module at inference time for computational ease. For all 3 datasets (Tables 1, 2, 3), the teacher module outperforms the student, as would be expected given the use of the MLLM in the teacher. In fact, in Tables 2 & 3, the student's performance is at par or below the baselines compared with. \n3. In Table 2, the teacher's performance (presumably using GPT 4o mini as the underlying MLLM given that this is found to be a more effective MLLM in Table 6) is below that of using GPT 4o mini directly for the problem, for all 4 metrics. Similarly, for Table 3, the teacher's performance is at par with the MLLM itself for 3 out of the 4 metrics. This raises questions on the requirement of the added parameters of the teacher module vs using the MLLM directly for the problem.\n\nQUESTIONS:\n1. Are the human annotations referred to in Line 217 directly obtained from the datasets, or was an additional study conducted? If so, could you please provide details on this?\n2. Could you please clarify the choice of modeling this as a regression task vs classification? Also, how were the regression scores converted to sentiment labels to compute F1 & Accuracy?\n3. Have any experiments been conducted on out of domain datasets? \n4. In Table 6, Gemini 2.0 Flash is shown to *underperform* w.r.t GPT 4o mini as the MLLM in the proposed approach. However, in Tables 1 & 2, on the same datasets, by directly using these MLLMs for the task, we find them to be at par for Accuracy & F1, with Gemini  *outperforming* GPT 4o for MAE & Correlation. Any clarifications/thoughts on this occurrence would be helpful to know.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MLLM-Guided Multimodal Sentiment Learning Framework (MMSLF), a framework that leverages multi-modal large language models (MLLM) to improve the performance of task-specific multimodal sentiment analysis (MSA) models in a teacher-student setup.  The teacher uses information from a powerful MLLM model (specifically, GPT-4o-mini in this study) to generate context-aware prompts that help the student focus on better alignments between different modalities of the data. Experimental results on 3 benchmark datasets show that the proposed framework leads to competitive task-specific models.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed framework leverages powerful MLLMs to improve the results of smaller task-specific fine-tuned models. To do that, it makes sense of a teacher-student framework in an interesting way, where the teacher is equipped with an MLLM and generates prompts that help the student align multimodal data and focus on what may be more representative features for the multimodal sentiment analysis task. \n\nThe authors run experiments on three datasets and compare their results with the results of a large set of baselines, including MLLMs and also task-specific fine-tuned models. \n\nExperimental results show that the proposed approach has the ability to improve the results of prior fine-tuned models. \n\nThe authors motivate the proposed approach by saying that the use of MLLMs for the task of multimodal sentiment analysis may not be justified, given the increase in computational costs required for good performance. However, an MLLM is still used for the teacher. While the student does not directly use output from an MLLM, its performance lags significantly behind that of the teacher and also the performance of other MLLMs (specifically, Gemini-2.0-Flash). It is not clear that the drop in performance is acceptable. The authors do not do any analysis in terms of computational resources used by different models, despite this being part of the motivation for the work. \n\nThe authors perform 5 runs for each experiment and report average metrics together with standard deviation. However, there is no statistical significance analysis and it's not clear that the student model is indeed better than most of the other fine-tuned models. \n\nAs a side note, there is no discussion of the specific multimodal information that each of the baselines are using. Gemini-2.0-Flash is able to use all the modalities of the data and gives the best results overall, while the MLLM used in the teacher doesn't seem to support audio analysis. The authors show that the results obtained with  GPT-4o-mini  are still better than those obtained with Gemini-2.0-Flash but it's not clear why. \n\nThe captions of the tables are not self-explanatory.\n\nQUESTIONS:\nThe authors show that the results obtained with  GPT-4o-mini  are still better than those obtained with Gemini-2.0-Flash. Some insights into why this happens would be useful.\n\nLIMITATIONS:\nThe results of  Gemini-2.0-Flash are the best overall. A discussion of the tradeoff between computational resources and performance is lacking.\n \nIt is not clear what the assumptions of different baselines might be and what data modalities they work with. That might explain some of the results.\n\nThere is no statistical significance analysis.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduces MMSLF (mLLM-guided sentiment analysis) where they feed mLLM output guidance as additional information into a (pretty much) standard sentiment analysis model to improve performance. Further, this model is used as a teacher to distill information into a sentiment analysis model that does not require mLLM input, thus significantly saving computational resources at inference. Performance for the mLLM+sentiment analysis model is close to SOTA for SIMS (especially), MOSI, and MOSEI datasets, while the performance for the teacher is still pretty good. The authors run a comparison with most of the popular MM fusion models/algorithms and perform a comprehensive ablation study.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Novel and Practical Approach: the idea of leveraging  mLLM knowledge via text output and then distilling this information in a teacher -student framework is novel.\n2. Comprehensive Evaluation: The paper includes thorough experiments across three datasets, multiple baselines (both traditional and mLLM-based), and extensive ablation studies examining each component's contribution.\n3. Technical Correctness: The authors do all the reasonable things without being especially novel in their MM architecture. The distillation is the most interesting part. \n4. Clear Motivation and Results: The cost-effectiveness argument is compelling and practical - the contribution of mLLM output is clear via the ablation study. Overall, persuasive results\n\nWeaknesses:\n1. The abstract and intro do little service to the paper. The main idea is not clearly exposed, i.e., explain what is the information that you are adding and in what format, explain the novelty of the paper in the distillation etc. Conclusions are a bit better. Still thorough rewrite is needed.\n2. You should at least explain where you sit in terms of mLLM SOTA, e.g., see this recent paper https://arxiv.org/pdf/2504.11082  -  in general your results are good but you are giving the impression that you are SOTA which is not true. \n3. Your literature review is ok and the methods you describe are solid - but i would like to see more focus on papers that introduce new ideas rather than having a ton of top performers in your studies, it would be nice to mabe also review ideas from https://lab-msp.com/MSP-Podcast_Competition/IS2025/\n4. The fusion and in general the architecture is of little novelty; however, the teacher-model part is interesting\n5. mLLM dependency is a concern - how you engineer the mLLM output is super-important and it is hard to tune as one can see from results from different mLLLMs\n6. Without the mLLM prompts results are a low - it is unclear to me if this a  weakness of the architecture or lack of tuning - please explain\n7. The method is not modular enough to be applied on top of any existing sentiment analysis model at its current form\n\nQUESTIONS:\n1. Could you use this on top a very strong baseline model - like the one mentioned above, how dependent is your mLLM prompt mixing approach on your AVT fusion architecture \n2. Prompt Sensitivity and Robustness: Given the significant performance difference between GPT-4o-mini and Gemini-2.0-Flash prompts, how sensitive is the method to prompt variations? \n3. Computational Cost Analysis: While you mention cost-effectiveness, can you provide concrete numbers comparing the total computational cost of your approach - also vs other SOTA methods like the ones i mentioned above\n4. How relevant is your approach for other multimodal tasks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a distillation-based method to introduce multimodal LLM (MLLM) to the multimodal sentiment analysis (MSA) task. The teacher model is trained with the context prompt from MLLM, called conditional alignment in this paper. Finally, experiments are conducted based on the three popular MSA datasets.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The motivation and insight of this paper are interesting. MLLM is a popular technique; how to introduce MLLM to the MSA task is a good motivation. \n\n2. The experiments are conducted based on popular datasets. CMU-MOSEI, CMU-MOSI, and SIMS are the three popular MSA datasets.\n\n3. The paper is well-organized and well-written.\n\nCons:\n\n1. This paper misses many related works. MAG-BERT[1], DMD[2], DLF[3], TCAN[4], MMML[5]. The performance of this paper is not SOTA. Although the SOTA result is not necessary for publication at the top-tier conference, it is not appropriate to claim that this paper achieves the SOTA performance. All the following papers should be included when discussing MSA tasks. If not, please show reasonable reasons for that. The MMML used another encoder for text; for fair comparison, it is better to use the same encoder for text.\n\n[1]  \"Integrating multimodal information in large pretrained transformers.\" ACL 2020.\n\n[2] \"Decoupled multimodal distilling for emotion recognition.\" CVPR 2023.\n\n[3]  \"DLF: Disentangled-language-focused multimodal sentiment analysis.\" AAAI 2025.\n\n[4] \"TCAN: Text-oriented cross attention network for multimodal sentiment analysis.\" Arxiv 2025\n\n[5] \"Multimodal multi-loss fusion network for sentiment analysis.\" arXiv 2023\n\n2. For teacher-student models, or distillation-based models, efficient metrics are generally needed. Such as the model size, inference time or FLOPs.\n\n3. There is no visualization analysis to show the unique contribution of this paper.\n\nQUESTIONS:\n1. For the MSA task, the THUAIR repo. is a good reference, but it is not all. It is necessary to discuss more papers published after 2023 so that this paper claims the right thing.\n\n2. How to obtain all the prompts when training the teacher model? Is it obtained once, or sample and generate the prompt for each data point?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-03"}