Re-Imagining Multimodal Instruction Tuning: A Representation View

Authors: Yiyang Liu, James Chenhao Liang, Ruixiang Tang, Yugyung Lee, MAJID RABBANI, Sohail Dianat, Raghuveer Rao, Lifu Huang, Dongfang Liu, Qifan Wang, Cheng Han
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Representation Tuning, Large Multimodal Models, Parameter-efficient Fine-tuning

Abstract:
Multimodal instruction tuning has proven to be an effective strategy for achieving zero-shot generalization by fine-tuning pre-trained Large Multimodal Models (LMMs) with instruction-following data. However, as the scale of LMMs continues to grow, fully fine-tuning these models has become highly parameter-intensive. Although Parameter-Efficient Fine-Tuning (PEFT) methods have been introduced to reduce the number of tunable parameters, a significant performance gap remains compared to full fine-tuning. Furthermore, existing PEFT approaches are often highly parameterized, making them difficult to interpret and control. In light of this, we introduce Multimodal Representation Tuning (MRT), a novel approach that focuses on directly editing semantically rich multimodal representations to achieve strong performance and provide intuitive control over LMMs. Empirical results show that our method surpasses current state-of-the-art baselines with significant performance gains (e.g., 1580.40 MME score) while requiring substantially fewer tunable parameters (e.g., 0.03% parameters). Additionally, we conduct experiments on editing instrumental tokens within multimodal representations, demonstrating that direct manipulation of these representations enables simple yet effective control over network behavior.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730267743803

thread_type:
official_review

<thread_object>
creator: reviewer_rz9b

<content>
summary:
This paper adopted the parameter-efficient fine-tuning method Representation Tuning to the multimodal large language model domain. This paper used different representation editors for the vision encoder, LLM, and cross-modality projectors to optimize the visual representation, cross-modality representation, and multimodal representation. Experiments on several MLLM and image classification benchmarks show the efficiency and effectiveness of the proposed method. The paper further conducted controllability studies on image classification benchmarks to show the possible controllability of the proposed methods. Extensive ablation studies are furhter discussed for the designs of several hyper-parameters of the proposed method.

strengths:
1. This paper provides a promising and efficient PEFT method for MLLMs as an alternative to the commonly used LoRA-based methods.
2. The ablation studies are comprehensive for a better understanding of the proposed method and hyper-parameter design choices.

weaknesses:
1. The paper directly adapts the representation learning method in LLM to the MLLM in a rather straightforward way thus the technical contribution is limited.
2. The benchmark selection for comparisons is not comprehensive and convincing enough. MME is a relatively small-scale MLLM benchmark that has non-trivial variances. The paper should include more comprehensive and commonly used multimodal benchmarks like SEED, MMBench, MMMU, GQA, VQAv2, ChartQA, and DocVQA for more convincing comparisons.
3. The performance of other methods might have some problems. In Table 1 of this paper, the lora baseline is significantly worse than the full-finetuning one, but according to LLaVA's results (https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md), the performances should be similar. Besides, the proposed method adds tunable parameters in the vision encoder while other baselines use frozen vision encoders. Baselines with unfrozen vision encoders should also be added for comparison.
4. As the proposed method brings additional parameters and time costs in the inference phase, the efficiency advantage of the proposed method should be clearly verified in the training phase, including GPU memory usage and training speed comparisons,
5. To validate the generalization ability of the proposed method, the authors should include experiments with different vision encoders + different LLM types and sizes.
6. Although the paper claimed that the proposed method brings more interpretability and controllability to MLLMs compared with common practices, it is hard to see how could the method really help the interpretability and controllability of MLLMs. The controllability study is too customized with human priors and provides little help for MLLM controllability in the general case.

questions:
1. In Sec 3.2 Cross-modality Representation part, it says that the projector integrates representations from each layer of the visual encoder. Does this mean you combine vision features in different layers in the vision encoder as the final vision input for LLM?
2. What is the reason for applying prefix and suffix editors on textual tokens, as the prefix and suffix of different textual prompts at the same position might have very different semantics and meanings? Wouldn't this be harmful for the claimed interpretability and controllability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730575089237

thread_type:
official_review

<thread_object>
creator: reviewer_pdvr

<content>
summary:
This paper introduces a method for tuning large multi-modal models (LMM) in a efficient but effective way so that it can achieve similar performance to full fine-tuning, with an additional objective of having a controllability. The key idea of this paper is based on a prior technique that learns parameters that edits the representations. The main contribution of this paper is to make use of this technique for efficient tuning of LMMs, and the experiments show that this idea indeed is helpful and the paper did a good job of investigating the effect of various design choices.

strengths:
- Motivation is clear
- Intuitive and simple idea that works well
- Extensive analysis/ablation on the design choices

weaknesses:
Experiments on controllability is interesting but not conclusive. For instance,
- What would happen if you don't use ROI and do train with all tokens?
- What would happen if you fine-tune other baselines for this setup?
- What would happen if you use a sentence that has a same semantic meaning to 'Is the object an e in the image?' but with different structure and words, after fine-tuning? Would the model still be controlled as intended?

Additional weaknesses are:
- Figure 5 on the optimization landscape is interesting but I'm not sure how it is cherry-picked. Would there be a way to make this claim be supported by some metrics or more figures?
- Main method section feels a bit redundant to me, not much of a difference between each subsection. It could be nice to think of a better way to re-structure, remove redundancy, and think of a way to clearly explain what differences exists

Note: I'm not an expert in this area so I might be missing some experimental details. I'll check the other reviews on how the experimental setup is valid and how the results are good compared to other baselines.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687563473

thread_type:
official_review

<thread_object>
creator: reviewer_yzen

<content>
summary:
This paper introduces Multimodal Representation Tuning (MRT), a parameter-efficient fine-tuning method to enhance controllability and interpretability in multimodal large language models (LMMs). MRT addresses the challenge of adapting LMMs effectively with fewer parameters by leveraging token-level multimodal representation control, achieving superior performance with up to 21 times fewer parameters than similar approaches. The authors explore and improve model behavior control through MRT, illustrating benefits in various multimodal perception and cognition benchmarks.

strengths:
- MRT is intuitive, simple, and effective. It uses significantly fewer parameters while achieving strong results, making it suitable for resource-constrained applications.
- The approach provides granular control over the representation editing, enabling counterfactual output generation that enhances interpretability.
- The method is validated across multiple multimodal tasks, illustrating its effectiveness in diverse domains such as OCR, visual perception, and spatial reasoning.
- The paper is well-written and easy to follow.

weaknesses:
- The rank parameter is integral to MRT’s performance, but it currently requires manual tuning, which may limit practical adoption. While promising, the need for automated rank selection is highlighted as a limitation, suggesting a more autonomous rank-searching mechanism that could enhance usability.
- The empirical performance is decent, but could you elaborate on MRT's main contribution compared to ReFT instead of extending the interchange intervention idea into MLLM?
- Given its control over multimodal representations, MRT’s potential for misuse (e.g., manipulation of outputs) is acknowledged but not fully addressed in terms of mitigation strategies. Could you provide some examples of possible misuse of MRT utilizing controllability?

questions:
1. Could the authors elaborate on any plans to automate the rank-tuning process within MRT to simplify its application?
2. Section 4.3 is particularly interesting to me. However, I think the attacker can easily break the controllability only by changing the order of the text instruction. Do you have any potential solutions and ideas on that?
3. Instead of simple image classification, are there other qualitative examples of counterfactual controls (e.g., VQA)?
4. Any analysis on which set of layers L to intervene on for visual / cross-modality / multimodal editor?
5. What are some insights on fine-tuning only prefix/suffix tokens of textual embedding in the multimodal editor?
6. Minor Errata
   1. L84: hu2024bliva looks typo.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730784798997

thread_type:
official_review

<thread_object>
creator: reviewer_uvrb

<content>
summary:
The paper proposes a novel Multimodal Representation Tuning which can editing LMM representation and provide control. The paper introduces a representation editor $\phi$ based on linear representation hypothesis and interchange interventions, which can apply to different representations in LMM. The overall writing is clear. The experiments cover the comparison between the MRT and the other PEFT methods.

strengths:
1.	The overall writing is clear.
2.	The idea is general and could be applied to many different applications. I think this would be of interest to people in the LMM community.
3.	The proposed method is simple yet effective.
4.	The experiments clearly show the improvement of the MRT over the PEFT.

weaknesses:
1.	There are some typos. For example, ##hu2024bliva## in the "Multimodal Instruction Tuning" section of the related work.
2.	The ablation study is insufficient. I would expect more ablation experiments, such as applying MRT only to Visual Representation and Cross-modality Representation.
3.	There is too little theoretical analysis on why MRT is better than PEFT.

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732062629245

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
To all reviewers:

Thank you for your thorough review and insightful comments. We have revised our paper according to the suggestions. The major changes are summarized as follows:

* We have performed more ablation experiments to explore applying MRT to a single modality at a time, detailed in Appendix Sec. S4. We have also further discussed the future plan to conduct more theoretical analysis of MRT in Appendix Sec. S11. Additionally, we corrected the typo on Line 84 as suggested by **Reviewer uVrb**. 

* As suggested by **Reviewer Yzen**, we have added a discussion on prefix/suffix editing in Appendix Sec. S2, supplemented analysis of editing depth in Sec. 4.4, and provided more qualitative examples of counterfactual controls on the Text-VQA dataset in Appendix Sec. S6. Additionally, we discussed MRT's robustness and generalizability in Appendix Sec. S6, addressed potential misuse and mitigation in Appendix Sec. S10, and corrected the typo on Line 84. Key technical contributions have also been highlighted in Appendix S11.

* Based on the suggestions by **Reviewer PDvR**, we have included more discussion on the robustness of MRT’s controllability in Appendix Sec. S6 and reduced redundancy in Sec. 3.2. In addition, we have added more discussion on the optimization landscape in Appendix S11.

* We conducted additional experiments evaluating MRT on SEED and GQA datasets, as well as applying MRT to LMMs with different vision encoders and LLMs, presented in Appendix Sec. S3. Training time and memory efficiency comparisons were included in Appendix Sec. S5, and the visual representation selection procedure was clarified in Sec. 3.2. We have performed more controllability experiments in Appendix Sec. S6. Moreover, we have also highlighted our key technical contributions in Appendix S11, as suggested by **Reviewer RZ9b**.

All modifications have been marked in ${\color{blue} blue}$ in our revised submission.


Sincerely yours,\
Authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732256550443

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to the discussion

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you've devoted to reviewing our work. We understand that your schedule may be quite busy, and we are truly grateful for your valuable feedback. As we are presently in the discussion phase, we would greatly value the opportunity to engage in further dialogue with you. Our aim is to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss.

We look forward to the opportunity for further discussion with you. Thank you for your thoughtful consideration.
 
Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733165747196

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Author-Reviewer Discussion

comment:
Dear Area Chair and Reviewers,

We would like to express our sincere gratitude for your efforts in facilitating the discussion regarding our paper. As the discussion is coming to an end, we would like to provide a brief summary of the key points that have been discussed:

- We have included ablation studies to explore applying MRT to a single modality at a time and discussion of the future plan to conduct more theoretical analysis of MRT, as suggested by **Reviewer uVrb**. 

- In response to **Reviewer Yzen**'s suggestions, we have included a discussion on the prefix/suffix editing of textual-oriented representations and analysis of editing depth. We have also conducted thorough experiments on the extension to VQA task, robustness, and generalizability (i.e., prompt engineering) of MRT’s controllability. Moreover, we have discussed the potential misuse and mitigation of controlling model output and highlighted three key technical contributions of MRT.

- We have provided more discussion on the distinction of MRT’s counterfactual output control compared to other PEFT methods and robustness of MRT’s controllability with different prompt templates, based on **Reviewer PDvR**’s suggestions. Additionally, we have reduced the redundancy of the main method section and included more discussion on the optimization landscape.

- We have conducted additional experiments to evaluate MRT on SEED and GQA datasets, alongside applying MRT to LMMs with different vision encoders and LLMs and evaluating the generalizability of our method under different training datasets. We have also included more experimental results on the extension, robustness and generalizability of output control, and highlighted our key contributions. In addition, we have clarified the visual representation selection procedure and the choice of tuning prefix/suffix tokens on textual-oriented representations, as suggested by **Reviewer RZ9b**.

In summary, we would like to express our appreciation to all reviewers for acknowledging our responses. We are particularly grateful that Reviewer RZ9b has increased their score, and Reviewer uVrb, Yzen and PDvR have maintained their positive assessment. Although we understand that Reviewer RZ9b has not engaged in subsequent discussion due to the busy schedule, we believe that our response has effectively addressed the concerns through clear explanations and additional experimental results.

We would like to emphasize the contributions of our work, which have been acknowledged by the reviewers and are important to the community:

***Efficiency and Effectiveness:*** Existing PEFT approaches for LMMs tend to focus on reparameterization or adding additional heavy layers or trainable prompts, overlooking the potential of semantically rich multimodal representations. In contrast, MRT pioneers a fundamentally different approach by concentrating on directly editing these representations. Our approach eliminates the need for extensive parameter usage or complicated architectural modifications, making it significantly more computationally efficient. Despite utilizing only 0.03% of the tunable parameters, MRT achieves state-of-the-art performance across various benchmarks, proving that manipulation of multimodal representations can deliver both efficiency and effectiveness. This balance of reduced computational demands with high performance underscores MRT's impact on PEFT strategies for LMMs.

***Multimodal Integration:*** Our approach utilizes editing on semantically rich multimodal representation, considering not only multi-modality editing but multimodal integration. Unlike existing PEFT approaches on LMMs which either keep the cross-modality layer frozen or make it fully trainable, we introduce a lightweight cross-modality representation editor. This innovative design efficiently bridges the gap between visual and textual modalities, enabling effective alignment while maintaining computational efficiency.

***Controllability and Interpretability:*** MRT is the first attempt to enable token-wise control over LMMs through representation editing. By directly editing the semantic information of the image RoI and the textual target class indicator token, MRT offers an interpretable and intuitive mechanism for adjusting model predictions. This level of fine-grained controllability is difficult to achieve with existing baselines. Additionally, we provide a detailed analysis of why MRT outperforms other PEFT methods via visualizing the loss landscape, demonstrating that multimodal representation tuning enhances the generalization capabilities of LMMs.

Finally, we deeply value the constructive comments provided by the reviewers. In response, we have carefully refined our work based on the feedback received. Considering the contributions made, we hope that our work can provide new insights to the efficient tuning communities, and contribute to their further development.

Sincerely,\
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Multimodal Representation Tuning (MRT), a new parameter-efficient fine-tuning (PEFT) method for large multimodal models (LMMs). Instead of inserting additional trainable parameters into each layer, MRT directly edits and refines the latent representations at various points in the vision, language, and cross-modal processing stacks. The authors argue that focusing on representation-level adjustments offers several advantages: improved interpretability, fine-grained controllability over model outputs, and better performance than existing PEFT techniques with fewer tunable parameters.

Strengths:

-- MRT uses significantly fewer parameters than competing approaches yet maintains or exceeds performance. This resource-friendliness is appealing for real-world scenarios with limited computational budgets.

-- By editing representations directly, MRT enables counterfactual interventions and more transparent ways to understand how multimodal information flows and influences the final output.

-- The authors present experiments on diverse tasks and show consistent gains over existing baselines. The clear improvement across multiple multimodal benchmarks supports the robustness of the approach.

Weaknesses:

-- While the paper includes some ablation studies, reviewers note that further investigations—e.g., isolating edits on visual vs. cross-modal representations—could strengthen the findings. Additionally, performance on larger, more commonly used datasets (e.g., VQAv2, DocVQA) would bolster claims of general applicability.

-- The paper mainly provides empirical evidence but offers limited theoretical rationale for why representation editing yields these gains. Moreover, the important rank parameter is hand-tuned, leaving open the question of how to automate that process for wider adoption.

-- MRT’s controllability demonstrations are intriguing but somewhat narrow. More varied examples—especially across different NLP tasks or more complex queries—would strengthen the case for how effectively MRT can be leveraged in practice without risking misuse or adversarial manipulation.

Overall, the paper makes a notable contribution to parameter-efficient fine-tuning for LMMs. MRT’s approach to editing multimodal representations is clearly distinctive and yields strong empirical improvements. The reviewers appreciate the method’s simplicity, interpretability, and the strong reported results.  The AC agrees with the majority of the reviewers to accept the paper.