Prompt-Guided Distillation from Multimodal Large Language Models to Task-specific Models for Multimodal Sentiment Analysis

Authors: Haoyu Zhang, Xiaoying Tang, Wei Liu, Jian Luan, Tianshu Yu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal Sentimen Analysis, Representation Learning, Multimodal Large Language Model, Knowledge Distillation

Abstract:
Multimodal Sentiment Analysis (MSA) has made some progress with the advent of Multimodal Large Language Models (MLLMs). However, the scalability and the closed-source nature of some MLLMs imposes challenges for efficient application in the real-word. In this study, we explore an innovative pathway to infuse the capabilities of general MLLMs into task-specific small models for MSA. We introduce the Prompt-Guided Multimodal Framework (PGMF), a refined teacher-student framework designed to transfer knowledge from powerful, general MLLMs to smaller, efficient models. The PGMF-Teacher utilizes MLLM-generated prompts and a tailored conditional alignment module to achieve better MSA, while the PGMF-Student distills this expertise to predict independently of MLLMs' guidance. Extensive evaluations on two popular MSA datasets including SIMS and MOSI demonstrate that compared to previous task-specific small models, PGMF-Teacher achieves state-of-the-art performance with the help of MLLMs' prompts, while PGMF-Student achieve competitive results with fewer parameters and without relying on MLLMs' prompts. The proposed framework offers a novel way to equip task-specific small models with the capability of MLLMs.

Decision: reject
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730700342955

thread_type:
official_review

<thread_object>
creator: reviewer_wmdu

<content>
summary:
The paper  propose a novel framework that integrates the generalized knowledge of MLLMs to guide smaller, task-specific models for better MSA.
﻿
Specifically, visual and audio features are aligned with language features via two alignment modules: Visual-to-Language Alignment and Audio-to-Language  Alignment. These conditional alignment layers establish correspondences between modalities with the help of the prompt, facilitating effective multimodal fusion with the help of MLLMs. 
﻿
Both PGMF-Teacher and PGMF-Student can achieve good performance on two popular datasets (i.e., SIMS and MOSI), especially for PGMF-Student which can achieve improved performance without relying on prompt from MLLMs while maintaining fewer parameters. 
﻿
This approach also offers a novel way to empower task-specific small models with the capabilities of MLLMs.

strengths:
originality: The specific method of this paper has not been seen in other papers, so I believe it is original;

weaknesses:
It is currently a common practice to distill knowledge from large models to small models and achieve improvement. This paper uses a multimodal large model to identify the clues that play a decisive role in predicting emotional labels in each modality, and then integrates them into the small model training framework for improvement. The idea is relatively straightforward, and the innovation is not particularly prominent.

questions:
none
</content>
</thread_object>

</thread>
<thread>
creation date:
1730823044136

thread_type:
official_review

<thread_object>
creator: reviewer_pnkv

<content>
summary:
The paper presents a distillation framework for Multimodal Sentiment Analysis, where knowledge is extracted from Multimodal LLMs and used to bias the attention maps of a smaller task-specific model. This model is then used as a teacher for a task-specific student model, which does not require the LLM-generated prompts during inference. The method achieves competitive performance on CMU-MOSI and state-of-the-art performance on CH-SIMS.

strengths:
*  The presented method of prompt-guided attention transfer from MLLMs is a solid contribution and could be utilized for more tasks by the research community.
* Ablations regarding the effect of prompt-guidance, the effect of loss weight hyperparameters, and extension of the method to different architectures are included (though some are delegated to the appendix).
* Multiple-run averages, along with standard deviations are included.

weaknesses:
* The evaluation could include the CMU-MOSEI dataset, which is larger and more recent than CMU-MOSI.
* The "Searched best seed" related rows in Tables 1 and 2 should be removed. Optimized seed results are rather uninformative, since the seed is not a tunable hyperparameter.
* I do not support the choice of delegating the "Related work" section to the Appendix, since relation to prior work should be a key component of a research paper. Reducing the size of figures 1, 2, removing  the optimized seed results, and performing a general revision of  the paper should create space in the main part of the paper.
* In general  Sections 2.4, 2.5 are a bit verbose and the equations are rather uninformative. Equation 11, especially so. What exactly is the fusion operation? I think these sections could benefit from a revision both in terms of concreteness and clarity and in terms of length.
* The  attention scores in Figure 3 range from -0.0004 to  0.0006 (very close to 0 and a difference from high to low score in the 4th decimal point). I think this is very concerning for the soundness of the method. What does the model actually attend to? Could this be due to the choice of the Hadamard product operation for fusing the attention matrices, which makes the scores extremely sparse / close to zero?

The authors have addressed most of these weaknesses in the rebuttal.

questions:
My questions are included in the  "Weaknesses" section of the review
</content>
</thread_object>

</thread>
<thread>
creation date:
1730966579704

thread_type:
official_review

<thread_object>
creator: reviewer_9buk

<content>
summary:
This study proposes a Prompt-Guided Multimodal Framework (PGMF) to transfer the capabilities of large Multimodal Large Language Models (MLLMs) to smaller, task-specific models for Multimodal Sentiment Analysis (MSA). PGMF consists of a teacher model (PGMF-Teacher) and a student model (PGMF-Student). The teacher uses MLLM-generated prompts to achieve better alignment and sentiment analysis, while the student learns to predict independently. Experiments show that PGMF-Teacher achieves state-of-the-art performance, while PGMF-Student achieves competitive results with fewer parameters, providing an efficient way to enhance small models with MLLM capabilities.

strengths:
1. Leveraging Multimodal Large Language Models (MLLMs) to address the current challenges in the field of multimodal sentiment analysis represents a promising and worthwhile direction for exploration.
2. Embedding the teacher-student model paradigm within this domain is also a well-considered and potentially impactful approach.

weaknesses:
1. It appears that your CONDITIONAL ALIGNMENT primarily facilitates attention-based interaction between the GPT-generated prompts and the corresponding content. However, I am unclear about the specific significance of taking the dot product of these two attention maps. From my perspective, your alignment module seems to merely apply attention mechanisms followed by a dot product, which does not appear to introduce any substantive algorithmic novelty. Could you elaborate further on the theoretical or empirical contributions this approach provides beyond the existing methods?

2. Your multimodal fusion module appears to simply concatenate features from different modalities and feed them into a transformer encoder. This approach is quite common and widely adopted in existing literature.

3. In a word, it seems that the paper primarily applies the teacher-student model paradigm to the domain of multimodal sentiment analysis (MSA), incorporating GPT-generated content as prompts. While the motivation is sound, the implementation appears somewhat simplistic, lacking sufficient innovation to substantiate a significant contribution.

4. The selection of baselines in your comparison is quite limited, and notably, none of the baselines are from 2024. Given that this field remains highly active and rapidly evolving, I strongly recommend including more recent baselines from 2024 to provide a more comprehensive and current evaluation of your proposed approach.

5. The analysis presented in the "EFFECT OF EACH COMPONENT" section appears rather superficial and lacks depth, raising the concern that it may have been generated by AI without sufficient refinement or critical examination.

6. Since the goal is to train a student model with reduced complexity, it would be highly informative to include a comparison of parameter counts with other baselines. Such a comparison would help substantiate claims regarding the efficiency and compactness of the student model relative to existing approaches.

questions:
1. It appears that your CONDITIONAL ALIGNMENT primarily facilitates attention-based interaction between the GPT-generated prompts and the corresponding content. However, I am unclear about the specific significance of taking the dot product of these two attention maps. From my perspective, your alignment module seems to merely apply attention mechanisms followed by a dot product, which does not appear to introduce any substantive algorithmic novelty. Could you elaborate further on the theoretical or empirical contributions this approach provides beyond the existing methods?

2. Your multimodal fusion module appears to simply concatenate features from different modalities and feed them into a transformer encoder. This approach is quite common and widely adopted in existing literature.

3. In a word, it seems that the paper primarily applies the teacher-student model paradigm to the domain of multimodal sentiment analysis (MSA), incorporating GPT-generated content as prompts. While the motivation is sound, the implementation appears somewhat simplistic, lacking sufficient innovation to substantiate a significant contribution.

4. The selection of baselines in your comparison is quite limited, and notably, none of the baselines are from 2024. Given that this field remains highly active and rapidly evolving, I strongly recommend including more recent baselines from 2024 to provide a more comprehensive and current evaluation of your proposed approach.

5. The analysis presented in the "EFFECT OF EACH COMPONENT" section appears rather superficial and lacks depth, raising the concern that it may have been generated by AI without sufficient refinement or critical examination.

6. Since the goal is to train a student model with reduced complexity, it would be highly informative to include a comparison of parameter counts with other baselines. Such a comparison would help substantiate claims regarding the efficiency and compactness of the student model relative to existing approaches.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731508880730

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
# General Response

We appreciate the valuable time and effort from all reviewers, as well as the constructive comments and suggestions that contributed significantly to the improvement of our paper. **We are eager to engage in further discussions with you to address your concerns.**

**First, we would like to restate our motivation:**

We found that there are two limitations to apply MLLMs to MSA. 1) Although MLLMs have shown some improvement in MSA, their performance gains are often marginal and come at a high computational cost. 2) **Unlike the common practice in other fields** [1,2], using MLLMs to generate high-quality data for training small models is challenging for MSA due to the complexity of generating video, audio, and text data together. **These limitations** **motivate** **us to explore a different** **and more efficient** **direction: leveraging the guidance** **of** **MLLMs to help in training task-specific small-scale models for better MSA.** By involving MLLMs' prompt as guidance during the alignment and distillation process, we designed the PGMF that make the task-specific small-scale model to benefit from the MLLMs' knowledge. This presents a novel efficient framework.

**Second, we would like to restate the novelty is the framework rather than the model's architecture:**

1. At the framework level, our pipeline **for the first time** utilizes prompt outputs from MLLMs as conditional guidance within a teacher-student framework, effectively improving the alignment and learning process of the student model. The experimental results validate the effectiveness of this framework, demonstrating that it enables the student model to achieve state-of-the-art performance with reduced complexity. 
2. **We intentionally chose the simple** **and straightforward** **architecture modules** **to validate our framework**,which we believe can better demonstrate the effectiveness of our framework. By keeping the implementation straightforward, we can clearly show that the performance gains are due to the prompt-guided alignment **rather than any additional complex module designs**. 
3. **We believe that simplicity and effectiveness are valuable for the community.** There are many works [3,4,5] that have simple architecture but make a great contribution to the community. For example, in the MSA field, Self-MM [5] achieves significant performance despite using only MLPs in its network architecture. Similarly, our framework PGMF, although intentionally designed to be simple, offers a new perspective on using MLLMs to guide smaller models in multimodal tasks, which we believe can inspire further exploration and refinement in this area.

Thank you for your patience and suggestions. **We look forward to discussing our work with you.**

Sincerely,

The Authors


# Reference Used throughout the Rebuttal

[1] Chen, L. et al., 2023. Sharegpt4v: Improving Large Multi-modal Models with Better Captions. *arXiv* *preprint**arXiv:2311.12793*.

[2] Chen, L. et al., 2024. Sharegpt4video: Improving Video Understanding and Generation with Better Captions. *arXiv* *preprint* *arXiv:2406.04325*.

[3] Zadeh A. et al., 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. In EMNLP 2017.

[4] Yao-Hung Hubert Tsaiet al., 2019. Multimodal Transformer for Unaligned Multimodal Language Sequences. In ACL 2019.

[5] Wenmeng Yu et al., 2021. Learning Modality-specific Representations with Self-supervised Multi-task Learning for Multimodal Sentiment Analysis. In AAAI 2021.

[6] Feng, X. et al., 2024. Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis. In EMNLP 2024.

[7] Jin, Y., 2024. GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer Based Fusion Network for Multimodal Sentiment Analysis. *arXiv* *preprint* *arXiv:2408.14809*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732376864256

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-up on Review Feedback

comment:
Dear Reviewers,

I hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing my work. As the rebuttal deadline approaches, **we would greatly appreciate it if you could kindly provide feedback on our responses to your initial comments**. Your insights would be invaluable in helping us refine and strengthen our work.

Thank you very much for your time.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a distillation approach for Multimodal Sentiment Analysis, where knowledge is extracted from Multimodal LLMs and used to bias the attention maps of a smaller task-specific model. 

Two reviewers vote to borderline acceptance and one reviewer gave borderline rejection while pointing out some limitations such as limited originalty and lack of innovation.

Even if this paper in on the borderline, considering the ICLR quality, the contributions seem not to be sufficient for ICLR presentation.

So, AC recommends rejecting this paper.