Re-Imagining Multimodal Instruction Tuning: A Representation View
Authors: Yiyang Liu
James Chenhao Liang
Ruixiang Tang
Yugyung Lee
MAJID RABBANI
Sohail Dianat
Raghuveer Rao
Lifu Huang
Dongfang Liu
Qifan Wang
Cheng Han
Venue: ICLR 2025 Poster (2025)
Keywords: Representation Tuning
Large Multimodal Models
Parameter-efficient Fine-tuning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Multimodal Representation Tuning for Zero-shot Multimodal Instruction Learning

Abstract: Multimodal instruction tuning has proven to be an effective strategy for achieving zero-shot generalization by fine-tuning pre-trained Large Multimodal Models (LMMs) with instruction-following data. However, as the scale of LMMs continues to grow, fully fine-tuning these models has become highly parameter-intensive. Although Parameter-Efficient Fine-Tuning (PEFT) methods have been introduced to reduce the number of tunable parameters, a significant performance gap remains compared to full fine-tuning. Furthermore, existing PEFT approaches are often highly parameterized, making them difficult to interpret and control. In light of this, we introduce Multimodal Representation Tuning (MRT), a novel approach that focuses on directly editing semantically rich multimodal representations to achieve strong performance and provide intuitive control over LMMs. Empirical results show that our method surpasses current state-of-the-art baselines with significant performance gains (e.g., 1580.40 MME score) while requiring substantially fewer tunable parameters (e.g., 0.03% parameters). Additionally, we conduct experiments on editing instrumental tokens within multimodal representations, demonstrating that direct manipulation of these representations enables simple yet effective control over network behavior.

Published as a conference paper at ICLR 2025
RE-IMAGINING MULTIMODAL INSTRUCTION TUNING: A REPRESENTATION VIEW
Yiyang Liu
James Chenhao Liang"
Ruixiang Tang' Yugyung Lee'
Majid Rabbani
Sohail Dianat
Raghuveer Rao Lifu Huang"
Dongfang Liu²
Qifan Wang"
Cheng Han"
1 University of Missouri Kansas City
*Rochester Institute of Technology 'Rutgers University
U.S. Naval Research Laboratory
U.S. DEVCOM Army Research Laboratory
Univerrity of California, Davis
Meta
ABSTRACT
Multimodal instruction tuning has proven to be effective strategy for achiev- ing zero-shot generalization by fine-tuning pre-trained Large Multimodal Mod- els (LMMs) with instmuction-folloinng data. However, the scale of LMMs continues to grow, fully fine-tuning these models has become highly parameter- intensive. Although Crrammteer-fficiee Fine-Tuning (PEFT) methods have been introduced to reduce the number of tunable parameters, significant performance gap remains compared to full fine-tuning. Furthermore, existing PEFT approaches are often highly parameterized, making them difficult to interpret and control. In light of this, we introduce Multimodal Representation Tuning (MRT), a novel approach that focuses directly editing semantically rich multimodal represen- tations to achieve strong performance and provide intuitive control over LMMs. Empirical results show that method surpasses current state-of toitte-ffthhhe trt- baselines with significant performance gains (e.g.. 1580.40 MME score) while requiring substantially fewer tunable parameters (e.g., 0.03% parameters). Additionally, we conduct experiments on editing instrumental tokens within multimodal represen- tations, demonstrating that direct manipulation of these representations enables simple yet effective control over network behavior.
INTRODUCTION
In this transformative era, artificial intelli- gence is undergoing a groundbreaking revolu- tion. driven by the rapid rise of Large Multi-
1600-
69
* 1580
1509
modal Models (LMMss) (Dumas et al. 2009)
1500-
8
Alayrac et al.) 2022, Yin et al.) 2023 Khattak (ct 2023). These models have demonstrated impressive capabilities across various multi- modal tasks, spanning remarkable capacities in natural language processing, computer vision, and beyond. Imagining future development, key objective in advancing LMMs is enhancing their zero-shot generalization ability to novel multimodal tasks. In this pursuit, multimodal instruction tuning has been introduced (Liu al. 2024), full fine-tuning pre-trained models with diverse multimodal instruction-folloving datasets, thereby enabling zero-shot generaliza- tion to previously unseen multimodal tasks.
MME
1400-
1398
83
62.0
12.3
1300
50.5
NMAWE
60
6122
*z
Oure MixLoRA
VPT M-FT
LoRA
8
Tunable Parameters (%)
Figure 1: MRT (ours) e.s. concurrent arts. Our method yields significant performance gains over state-of-the-art multimodal PEFT approaches MME and MMAvg benchmarks with consider- ably lower parameter usage (see Table I..
However, LMMs continue to grow in parameter size and complexity (e.g.. LLaVA Liu et al. 2024 leverages 7B and 13B backbone L.MSs and Flamingo (Alayrac et al. 2022) employs 70B LLM).. The standard approach of full fine-tuning LMMs from scratch presents significant challenges, as
"Equal contribution
1 "orresponning author

---

Published as a conference paper at ICLR 2025
researchers encounter difficulties in fine-tuning these pre-trained models both effectively and ef- ficiently. A promising solution, similar to vision and language domains, is to utilize Parameter- Efficient Fine- Tuning (PEFT) strategies (Han et al. 2023 2024b; [Shen et al. 2024). Despite achiev- ing promising effectiveness and efficiency, there are two main limitations in existing parameter- efficient methods. First, they typically require a substantial number of parameters attain sub-par performance to full fine-tuning, Meanwhile, the potential of fine-tuning rich semantic multimodal representations has been largely overlooked: Second, The parameters introduced in the PEFT proce- dure are abstract and independent of the physical characteristics of the problem being modeled (An-) gelov & Soares 2020). Consequently, they are challenging to interpret in manner that aligns with human understanding (Li al. 2018b Jin et al. c
This perspective raises two key questions: How can we achieve the effectiveness and efficiency of fine-tuning large-scale multimodal models? @ How can we explore the controllability of PEFT methods? These two questions form the foundation of our work. Our intuition that instead of merely modifying parameters in black-box manner, has been done in previous PEFT methods, we should explicitly investigate the potential of linearly interpretable representation engineering during the multimodal fine-tune tinettunig process. By doing so, ficiency but also foster a deeper understanding of the model's behavior, paving the way for advanced LMM efficiency and controllability.
not only improve the parameter ef-
In response to question 1, we propose an efficient and effective representation fine-tuning strategy Multimodal Representation Tuning (MRT), explore the extreme of tunable parameters (e-g., up to 21 times fewer parameters compared to LoRA) while achieving superior performance (e.g.. verses 4.7% higher performance on the MME benchmark (Fu al. 2023b) compared to the state-of- the-art baseline MixLoRA (Shen et al. 2024)) (see Figure 1p. To the best of our knowledge, MRT is the first work studying garameter-efficeee multimodal representation tuning, inspired by the current representation fine-tuning for language models (Wu et al. [202aa] b Turner et al. 2023
To address question @, we demonstrate that directly editing multimodal representations can ef- fectively control model behavior (see $3.3). Moreover, our findings indicate that precise behavior control offers valuable insights into the transparency and interpretability of PEFT methods, a topic that has been largely underexplored. We believe these insights establish foundational setup and perspectives for future research on multimodal representation understanding.
RELATED WORK
Multimodal Instruction Tuning, Transformers-based architectures currently dominate in LMMs. enabling breakthroughs in tasks such as visual question answering (Hu et al. 2024 Antol et al., 2015 Guo et al. 2023). image captioning (Özdemir & Akagünnüžz 2024). and visual commonsense reasoning (Chen et al. 2024, Park et al. (2024). A general structure of LMMs includes three main components (Liu et al. 20244] Li et al. 2023b): pre-trained modality encoder to encode modal features, a pre-trained L.M to reason tused multimodal data and perform prediction, and a cross- modality layer to align different modalities (e.g., linear projector in LLaVA (Liu et al., (024) and MiniGPT4 (Zhu et al. 2024), a GATED LATTN-DENSE layer in Flamingo (Alayrac et al, )022 ). An effective tuning method in improving the zero-shot capability of LMMs multimodal instruction tuning (Liu et al. 2024 Zhu et al. 2024 Dai et al. 2023). It refines LMMs by fine-tuning diverse dnstucction-folloiing datasets that embrace both user intent and desired responses, including machine-generated and human-annotated data. In this work, we explore parameter-efficient multi- modal instruction tuning LLaVA.
Parameter-Efficee Fine- TineTTnnngg Parameter-Efficient Fine-Tuning (PEFT) has emerged to solve the computational challenges of adapting large-scale models (e.g., LLMs, LMMs) downstream tasks (Wang et al. 2024: Liu et al. 2024). aiming to achieve comparable performance to full fine- tuning while updating only small fraction of model parameters or training customized learnable modules. Current PEFT strategies can be generally categorized into three groups: reparameteriza- tion, layer insertion and prompt tuning. Reparameterization methods (e.g., LoRA (Hu et al. 2022), IA3 (Liu al.) 2022 ) mainly focus on the reparameterization of the attention mechanism, offering balance between efficiency and performance. However, these methods still require a great amount of parameters while leaving a noticeable performance gap compared to full fine-tuning. Layer Insertion methods (e.g., Adapters (Long et al. 20244) generally insert learnable modules (e.g., fully- collecceodeeted layers) between attention or ML.. Nevertheless, they typically have higher parameter usage and ad-
2

---

Published as a conference paper at ICLR 2025
LIM
Edited
Original
Teo playful kittens, one lying Tunod
(
m
Frozzen
How nany cats
there the image?
down the other stand ing. on soft scc, poss
Dooda LooodLLaaa Layer
U
LLM Decrler Layer
Vision Encocker
Decodla Layer
V
S Vision Froower
V/
Ussion Encoder Layer Layer
Vision Encodar Laye Layer
Cross Mocality Layar
Wx+b
(a)
(b)
Figure 2: Overview of MRT. Representation editors v/ {vv. thyy Pr, ts) are the only tunable parameters while the entire model remains completely frozen. During fine-tuning, we jointly edit the visual representations in the vision encoder, the cross coos--modllity layer, and the prefix and suffix of rextual-oriented fraction in the multimodal representations in the LLM. These editors efficiently and effectively optimize the model representations during multimodal instruction tuning.
ditional burden during inference. Prompt-tuning (Jia et al. 2022, Yan et al. 2023) adds learnable soft tokens as a prefix to guide pre-trained models for specitic tasks. While prompt tuning is more parameter-efficient than Reparameterization and Layer Insertion methods, training only the prompt embedding could lead to sub-optimal performance when encountering more complicated tasks.
From different perspective, recent advances in representation engineering (Turner al., 2023 Zou al. 2023 Geiger et al. 2021) raise the exploration into representation tuning in Nature Language Processing (NLP) and Computer Vision (CV) fields, demonstrating promising results and superior parameter efficiency in comparison to existing PEFT methods. Specifically. RED (Wu et al. 2024a) proposes direct representation editing, utilizing element-wise scaling and bias for the entire representation of Transformers-basee layers. ReFT (Wu et al. 2024b) introduces intervention-base representation editing, steering partial representations of Transformers-basee projection matrix with orthonormal rows and linear projector. Although representation tuning has shown its exceptional capability on single modality (i.e., language), its effectiveness on multi- modalities is largely unexplored. Our method is the pioneering work to investigate the feasibility of multimodal intervention-basee representation tuning via rigorous structural design. Additionally, our experiments on instrumental token editing demonstrate that modifications within multimodal representations are highly effective, enabling precise counterfactual control over network behavior in the multimodal PEFT approach area that has not yet been sufficiently explored.
layers via low-rank
METHODOLOGY
In this section, introduce MRT, pioneer multimodal representation tuning approach for effec- tive and efficient LMM fine-tuning. We first introduce the preliminary of LMMs and notations in $3.1 The effective representation tuning with the designing of editors in visual, cross-men mooss-odallit,, and multimodal representation are presented in $3.2 The overall framework is shown in Figure
3.1 PRELIMINARY
Given vision-text Transformers-basse LMM F, which has been pre-trained on a substantial corpus of data and tasks, the model architecture is composed of three major components: vision encoder with hidden dimensionality d,,, a large language model T with hidden dimensionality d, and a linear eross- modallty projector C that aligns the dimensionality of visual features from d, to d,. The input of the model F an image I and text instruction T.
The processing of these inputs proceeds through the following steps: firstly, the vision encoder transforms the image into d,ddimensional visual tokens, denoted as T, = V(I) = (T.. T2 ..,..., T.m). where is the number of visual tokens generated by the encoder. Sec-
3

---

Published as a conference paper at ICLR 2025
ondly, the cross-modality projector C maps the visual representation I to the dimensional- ity of the language model, producing a de-dimensional visual embedding X₂ = C(T,) = C(T,") {x 2 "g ,,") Parallelly, the text instruction T is tokenized where n. represents the number of tokens with dy-dimensional space, forming textual embedding from the text instruc- tion. Lastly, the visual representation X, and the textual representation X₁ are combined through fusion mechanism, yielding joint multimodal representation X Concat Xe). Based on this fused representation, the large language model 'T generates a relevant linguistic response y F(I = T (Concat(X,.. X))).
into a sequence of textual tokens, X, = (okenize(T) = {ar ".....,,"""" =²
In our study, the primary objective is to fine-tune the pre-trained model, to enhance its zero-shot per- formance. While prior research has explored full fine-tuning strategies as well as parameter-efficien fine-tuning methods (see $2) we propose Multimodal Representation Tuning (MRT) that offers more computationally efficient approach. MRT has the potential to enhance performance signifi- cantly while minimizing resource consumption (see (4.2), presenting advantageous alternative to existing fine-tuning techniques from a representation view.
3.2
MUL.TIMODAL REPRESENTATION TCNING
MRT is inspired by the linear representation hypothesis (Wu al. 2024b) and interchange interven- tions (Geiger et al. 2021). As shown in Figure we apply representation editors for each layer of the vision encoder, LLM. and cross- cooss-modlliyy// layer, optimizing visual representation, coos-modallty representation, and multimodal representation simultaneously. Notably, during multimodal instruc- tion tuning, MRT only updates these editors, while the entire model remains completely frozen.
Representation Editor. We introduce a representation editor v formulated via the simple yet effective representation hypothesis. The editor modifies the original feature representation æ within specific subspace to reflect the desired intervention obtained from a linear projection Wx b, where both W and are learnable parameters. The editing operation is then confined to the subspace spanned by the rows of low-rank matrix U with orthonormal rows, so only targeted aspects of the representation are adjusted while preserving the remaining information. The editor & is:
+ -b
(1)
where U and W E Edixd: are low-rank matrices (i.e., di < de); di represents the rank of the subspace. U denotes the transpose of U. Specifically, Ux projects the original representation onto the subspace U, where Wz + b provides the target values within that subspace via linear transformation from the original feature representation x. The difference Wz + b - Ux computes the necessary interventions, which then mapped back into the original space via U'(.). By adding this intervention to the originals, we obtain the edited representation (i(x) that incorporates the desired modifications while maintaining the components orthogonal to the subspace U.
This editor facilitates controlled manipulation of representations by targeting linear subspaces, as multiple studies have shown that inttreeiiteerretbbll concepts are encoded linearly (Smolensky 1986 Rumelhart et al. 1986 Lasri et al., 2022, Guerner et al.) 2023 Mikolov et al., 2013). Con- sequently, linear subspace interventions can correspond to specific semantic attributes and modal- ities, thereby advancing research in LMM interpretability and controllability (see $3.3). Utilizing
a low-rank subspace with orthonormal rows not only enhances computational efficiency but also contributes to the stability and effectiveness of the intervention process.
Visual Representation. Without loss of generality, we maintain consistency with the previously established notation and recall the vision encoder V, which extracts visual features from given image I. For visual representation intervention, we define a set of visual representation editors, denoted v'v { . ... } where each individual editor vi corresponds to distinct visual low-rank representation editor that operates at the i-th layer of the encoder V. These editors function to modify the hidden visual representations T, at their respective layers within the encoder.
4

---

We concentrate editing solely the have already been extensively modified through dhy and the. We manipulate two consecutive seg- ments of the textual embeddings, corresponding a prefix tokens and suffix tokens. This process is facilitated by two sets of multimodal representation editors: u = (wr .., v%2) for the pre- Published as a conference paper at ICLR 2025
Specifically, each editor vi edits the complete set of hidden visual representations produced at its corresponding layer, expressed as:
Tx,1 (v. 0,1: T
To = (o D, T v,
where T represents the of visual representations i-th layer. They are sequentially edited by the corresponding editor v The process is applied across all m hidden representations at each layer, ensuring that each layer's output receives a layer-specific intervention. Finally, the edited visual representation from the second last layer is fed into the cross-modal projection layer C. (See $S2).
Cross-modality Representation. multimodal models, aligning visual representations within unified representation space is crucial. The cross-modality projector C plays an important role in this process. Consequently, we introduce an intervention in the cross-modality projector C, aiming to improve the alignment between visual and textual features.
Specifically, we define cross-modality editor, denoted as vhe The visual features X, are first processed by the cross-modality projector C, which integrates representations from each layer of the visual encoder V. The editor te is then applied to the output of C, yielding an optimized visual representation X,,, defined as:
= =.(CCIT T3 TTT }));
where T, represents the hidden visual representations from the final layer of the encoder V. This edited visual representation, X,, is subsequently combined with the corresponding textual repre- sentation Hr, producing a unified multimodal representation. The refined visual representation X., incorporating the output of the cross- modsliodylyt editor vic. is concatenated with the textual represen- tation X,, ensuring effective alignment of both modalities.
Multimodal Representation. In the previous discussion, the visual representation has been com- prehensively processed through th and the In this part, we shift toward the textual-oriented em- bedding tokens within the multimodal representations, where the visual and textual embeddings concatenated together (see I3.1).
representations, the image representations
fix tokens and vi. [w/, v/2 ...., { for the suffix tokens. Here, t/ and w denote the low-rank multimodal representation editors responsible for modifying the textual-oriented prefix and suffix embeddings, respectively, the i-th layer of the textual encoder T. Formally, the process of editing the multimodal representations across the j layers defined as:
X1 (-t 20 Wy 1. 27,1), sat
n-b-1 b-1
(2
F
( z.,)) 213
n- m- b-1
thd (2
where ....j....,.... " represent the visual tokens at j-th layer, and m o., represent the textual tokens. The prefix editors vig and suffix editors v apply the targeted intervention to the a prefix and b suffix textual tokens, following common practice (Geiger et al., 2021 Wu et al. 2024b). Altogether, by concatenating edited visual and textual tokens, we are able to adjust the intricate relationships between visual and textual information across layers. Further, as the visual and textual editing are decoupled, are then able to facilitate accurate LMM controllability (see $3.3)
3.3 CONTROLLABILITY THE BUTTERFLY EFFECT.
Controllable Text Generation (CTG) has been a recent surge of interest in the field of NLP for high-quality task-oriented generation, covering several conditions related lexical, structural,
5

---

Published as a conference paper at ICLR 2025
and semantic aspects (Zhang et al. 2022; Khalifa al. 2020: Erdem et al. 2022). However, many efforts (Zhag et al., 2023; Zeldes et al. 2020; Gao et al. 2020) designed to control the model
in implicit way to drive the generation of text satistying specific conditions; the transparency
and simplicity of CTG, however, remain problematic and misleading (Rudin) 2019 Rudin et al.,
[2022 Laugel al., [019 Arrieta et al. 2020). Existing methods for generation control can be broadly categorized into proc-proesssing and model behavior adjustment (Zhang et al. 2023 Liang al. 2024. Petrov et al., (023). Post-processing re-ranks the original next-token distributions in the textual decoder as filter to control the desired type of text while keeping the model com- pletely frozen. Though intuitive, it remains challenging to achieve better control performance. Even worse, in the multimodal scenario, the visual representation becomes entirely uncontrollable due to its decoder-oriented design. Model behavior adjustment utilizes strategies such as full fine-tuning, prompt tuning, and adapter to satisfy the controlled conditions. While effective, the behavior adjust- ment remains implicit, relying solely on full or partial parameter updates. The semantic meanings of representations within models, however, have largely gone unexplored.
In light of this view, we investigate the LMM controllability from a representation perspective, aiming to edit the actual semantics directly in flexible and explicit manner. We draw upon current research in LLM interpretability (Geiger et al.] [021] Wu et al. 2024b), where training a set of low- rank causal interventions on selected residual streams can effectively induce base LLM to follow human-desired instructions. Namely, given a singular set of representations, our design is able to manipulate them in targeted manner to achieve generalized control. In $4.3 we demonstrate that, even within complex multimodal settings, it remains feasible to interpret individual neurons and representations in isolation. We believe that this represents significant advancement towards multimodal interpretability and controllability.
EXPERIMENT
4.1 EXPERIMENTAL. SETUP
Implementation details. Following common practice (Liu al. 2024; Wang al.. 2024), we employ stage-one LLaVA (Liu et al. 2024) with CLIP-L (i.e., 24 Transformers- based encoder layers) as the vision encoder, a pre-trained modasty projector and Vieuna-7B-v1.3 (Chiang et al. 2023) (i.e., 32 Transformers- based decoder layers) as the backbone LLM in our pre-trained LMM (see I3.I.. For both visual representation editing and multimodal representation editing, we implement the same editor structure (see $3.2) For visual representations, we edit the entire visual representation in CLIP-L and the cross- modality layer. For multimodal representations, we apply More implementation
editing for both textual-oriented prefixes and suffixes in Vieuna-7B-vl. details and discussion on inference time are included in Appendix $52
Datasets. We conduct multimodal instruction tuning on Vision-Flan (Xu al. 2024 human- annotated multimodal instruction tuning dataset with 191 diverse tasks. Following common practice (Shen et al., 2024). we employ the scaled-down version containing up 1, 000 instances per task, resulting in a total of 191, 105 instances. For evaluation, we examine our method on the multimodal evaluation benchmark MME (Fu et al. (202a)), measuring both perception and cognition abilities across 14 subtasks (see ŞSI). We further investigate the model's 's capabilities using 7 multimodal datasets. Specifically, we utilize the Tex--VQA (Singh et al. 2019) for Optical Character Recognition, and the Visual Spatial Reasoning (VSR) (Liu et al. 2023) for reasoning. The perception capability is tested on CIFAR-10/10M (Krizhessky et al., 2009) and MNIST (Deng. 2012). Moreover, the SNLI-VE dataset (Xie al. 2019) evaluates Visual Entailment capabilities, while the POPE (Li et al., 2023c) dataset examines the object hallucination tendencies.
Evaluation Metrics. The MME incorporates both Perception and Cognition metrics for evalu- ation. For other multimodal datasets, we use ficuma-13B-I1. (Zheng et al.] 2024) to assess the accuracy of each prediction compared to the groundtruth, suggested by common practice (Shen ct al. 2024 Wang et al. 2024 Han et al. 2024b).
4.2
MAIN RESULTS
In Table !
we report a comprehensive zero-shot evaluation of MRT on eight multimodal datasets, comparing with several baselines. Specifically, we consider seven state-of-the-art PEFT methods, in-

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot Multimodal Evaluation. LLaVA slngn is the stage-one LLaVA without end-to- end fine-tuning, and LLaVApr indicates the fully fine-tuned LLaVA. The MMAvg represents the average score on the right seven tasks. Vision-Flan dataset is used for all fine-tuning processes. The best performance except FL is shown in bold, and the second best is shown in underline. As seen, MRT outperforms current state-of-the-art methods with far fewer trainable parameters (i.e., 0.03%). Note that the we have calibrated the M # para as it should include both 0.09% soft prompt params, and 1.87% Im head params for the overall parameter usage.
Method
# para
MME
Tex--VQA
VSR
SNLI-VE
CIFAR-IO
CIFAR-100
MNIST
POPE
MMAvg
ALaVA.
at
2024
1110.82
32.62
50,16
34.51
80.00
58.04
52.79
59,10
52.46
LLaVApp
et ar.
100%
1587.26
37.26
53.76
43.35
92.97
63.73
80.82
66.59
LoRA
Hu al. 2023
0.63%
1393.67
39.20
52.95
44.56
90.10
45.90
83.42
72.33
61.21
APrompe
Wang
20233
0.23%
1406,63
35.26
53.12
45.58
85.74
50.27
84.63
76.
61.52
PTUM
0.125
1354.62
53.75
$2.88
57.63
80.31
62.00
VPT
21740
0.06%
1398.74
33,68
53.93
32.62
76.49
52.31
94.73
79.60
60.48
ReFT Wo
20244
0.03%
1473.25
36.34
49,75
39.66
90.43
57.53
88.21
78.35
62,90
MPPT
Wiingg of
2014
1.96%
1503.98 98
34.48
53.19
32.89
89.29
59.14
95.54
81.26
64
MiaLoRA
(Snien et 2002
0.85%
1509.61
40.42
49.18
36.69
91.40
59.27
87.68
78.48
63.30
MRT
0.03%
1580.40
40.62
51.47
33.34
96,96
$7.20
95.63
79.30
64.93
cluding LoRA (Hu al. (022). APrompt (Wang et al. (023a). PTUM Yang et al. (223). VPT (Han
al. 2024b), M (Wang el. 2024). MixLoRA (Shen et al. 2024) and ReFT (Wu et al. 2024b). Here LoRA and MixLoRA are reparameterized methods, initializing and updating extra low-rank decomposition matrices within attention blocks; APrompt, VPT, PTUM, and M are prompt tuning methods. Differently, APrompt and VPT consider only inserting tunable soft prompts to a single modality (i.e., textual and visual space, respectively) while PTUM and MT are multi- modal prompt tuning approaches; ReFT is the most recent representation tuning approach for textu- ral modality. We do not include layer insertion methods in this comparison, as they typically require significantly higher parameter usage (Wu et al. 2024b Balne al., 2024), rendering them unsuit- able under the multimodal PEFT settings. Consequently, we have several key observations. First, MRT outperforms all PEFT methods with substantial performance gains. For example, our ap- proach achieves 4.70 % and 5.08% % improvements on MME compared to two state-of-the-ar PEFT baselines. MixLoRA and PT respectively. MRT can be further considered as a qualified alter- native to full fine-tuning, as it reaches 99.56% of the overall full fine-tuning performance on MME while introducing only 0.03% of the model parameters, demonstrating both its effectiveness and efficiency for large-scale multimodal model adaptation. Diving into the per-task performance, we also want to highlight that MRT outperforms the full fine-tuning LLa on TexI-VQA, CIFAR-IO, and MNIST tasks with large performance gap (i.e., 3.36%, 5.99%, 3.36%). Second, we observe that PEFT approaches focusing on multimodality (i.e., M MRT) generally outperform other methods that consider only single modality (i.e., APrompt, VPT, ReFT, MixLoRA). This indicates the significance of introducing cross- cross- modality interactions within MRT. The ablation study on com- ponent ablation in furtthrr proves that exploiting multimodal representation editing can result in higher performance. Third, similar to other PEFT approaches (e.g., PTUM, PT),,, MRT does not perform very well on visual entailment task, SNLI-VE. We hypothesize it's due to the complexity of logical relationship understanding, which might require more sophisticated task-oriented design.
4.3 CONTROLLARILITT RESULTS
We design our experiment on several image classification tasks, where we take an imagc-question pair as inputs. The LMM further answers the question based on the class prediction. As discussed in $3.3 our objective is to design targeted representation tuning that effectively intervenes in a few selected instrumental visual-based (i.e., visual and cross-modality tokens), and multimodal tokens to generate semantically counterfactual outputs.
Specifically, regarding that both visual-based features and textual-oriented target indicators in mul- timodal representations are rich in semantic information and play crucial roles in the image clas- sification task (Parekh et al., 2024), we decouple and study the LMM controllability via a set of representation editors & { (va, the, vif) from both modalities. Here fa indicates the multimodal representation editor for the targeted textual-orient token at the first layer of the LLM (i.e., not 1 or e/.,, but rather the specific token position intend to control).
Shown in Figure 3 for visual-based representation (i.e., T,, X.) editing, given that all images are represented as visual-based token patches of fixed length, our analysis here concentrates the
7

---

Published as a conference paper at ICLR 2025
semantically salient Regions of Interest (Rol), specifically the most informative visual-basee patches (e.g., objects for image classification). Note that we consider only Rols as candidates this setting, which is different from [3.2] The reason is that, during instruction fine-tuning, it is essential to consider all visual-based tokens for effective feature editing, whereas in targeted semantic control, only the Rols align the most to the paired question. Thus, editors 1/ and de are trained to edit only Rols to control the most important semantic information.
For editing of textual-oriented target indicators in multimodal representa- tions, we control the textual ques- tions to fixed template (More tem- plates are shown in Appendix $56): "Is the object an [indicator] in the image?" The representation editor ti is trained to modify only the to- ken corresponding to "[indicator]" within this sequence (i.e., the 5-th token). Given an image of class e, and the question "Iy the object e in the image?", an affirmative re- sponse (i.e., "Yes") represents cor- rect classification, while negative response (i.e., "No" ") refers to the in- correct one.
LMM w/o Intervention LMM with Intervention
Counterfactual "Nb"
YYes
Next Layer
Next Layer
4.l.0.
V
Rol
a
truck
in
Image question Pair
Is
abject
truck
the
image
Specifically, we target two differ- ent scenarios of counterfactual out- puts: i) Misclassification. Coun- terfactual output of miselassification on a specific class e while achiev- ing high classification accuracy for other classes. For training editors 2/1, change the training data where all labels of the targeted class counterfactual "No" while keeping the groundtruth labels "Yes" of other classes. ii) Misalignment. Counterfactual output of misaligning specific class into another class e. We train our editor 4/2 with misaligned image class with groundtruth of é. Note that 1/1 and 1 are two independent sets for each scenario.
Target Indicator
Figure 3: Controllabilty Pipeline on Image Classification. MRT offers LMM controllability from a representation per- spective, allowing for direct editing of representations with semantic meanings and enabling counterfactual interference with the results. Details shown in $4.3
In Table 2 conduct and report the results of control over counterfactual output 5 randomly selected classes from CIFAR-I0 10 (Krizhevsky et al.) 2009) (i.e., cat, dog, ship, frog and truck). For i), the trained representation editors 1b can modify the representation to produce counterfactual re- sponse (i.e., from "Yes" to "No") with 100% success rate, effectively causing the model to misclas- sify all class images when presented with the same template question prompts. Notably, interfering with class e does not prevent the model from classifying other classes correctly. For ii), the result demonstrates that the representation editors can fully control (i.e., 100%) the model to misalign all images from class into the targeted class e, while maintaining the capability of accurate classifi- cation of other classes. All together, our results clearly show that by simple yet intuitive token-wing representation editing. one can directly control the complex decision-making process, even consid- ering multimodal information as inputs. Furthermore, our insights may significantly advance L.M interpretability research, as the editing process is directly applied to both visual and multimodal rep- resentations, thereby regularizing trackable representations with certain properties (i.e., in our case, linear projection). By further designing explicitly the casual model, MRT can show potential promising solution for achieving ad-hoc interpretability (Wang et al. 2023b Subramanian et al. 2018 Chen et al. 2016 Jin et al.. 2024a
Table 2: Controlled Counterfact Rate is evaluated on two sce- narios: misclassification and misalignment.
Class
Misclassification
Misaligament Misalignment to
(L.VVA Ation)
Misclassification on e
Others
Others
(a) cat
18.8%
100%
0%
100%
(b) dog
17.3%
100%
0%:
100%
DS
(c) ship
21.8%
100%
0%
100%
0%
(d) frog
100%
(e) tuuck
21.4%
100%
0%
100%
0%
Han al. 2024a).
8

---

Published as a conference paper at ICLR 2025
MME
8-
1467.00
1475.93
1505.32
62:
1034
Visual- Visual-based bal-baadd
6-
1429.31
1580.40
1436.12
1522.87
Visual based
6
62.76
64.93
63.22
64.18
MMllugg Store
50
1473
4-
1474.33
1504.52
1518..7
1452.12
62.72
63.44
63.91
62.53
1454.43
1479.50
1489.83
1908
Multimedal
Multimodal
Figure 4: Impact of Rank. Each cell in the map corresponds to the evaluation score of model with multimodal rank (row) and visual rank (column). darker hue represents a higher score, whereas lighter hue indicates lower score.
4.4 DIAGNOSTIC EXPERIMENTS
Impact of Rank. In MRT, the number of ranks directly determines the number of tunable parame- ters. To further analyze the impact of rank w.r.t. model performance, we conduct comprehensive study on the number of ranks of iisual-based and multimodal editors on Vision-Flan. Specifically, we use grid search consider different combinations of ranks for visual-based. and multimodal editors, ranging from rank 2 to 8, respectively. We propose altering the visual and cross-modality editors to maintain the same rank count, eliminating the need for an additional manually set value, in alignment with 44.3 The results reported in Figure 4 As seen, the optimal configuration, yielding a peak score of 1580.40, achieved with risual-based rank 6, and multimodal rank 4. We stop at rank 8 because performance saturation is observed around this point. Further increasing the rank would result in increased parameter usage without significant performance improvement (e.g.. 1505.32 MME with visual-based and multimodal rank 8, and 1452. with visual-based rank and multimodal rank 8). This may result from slower convergence or overparameterization (Han
al. 2023 Hu al. 2022 Shen al. 2024 Zeng et al. 2024
Discussion on Optimization. We further investigate why MRT exhibits superior per- formance and generalizes effectively across different tasks from an optimization per- spective. Previous studies (Li et al.) 2018a] Ma al., (022) have shown that the geom- etry of the loss landscape plays crucial role in model generalization. Building on this insight,
depict the loss landscape in Figure [5] Here, we randomly choose two parameter directions, random directions has been shown to have minimal impact on the results (Li al.) 2018a). As seen, MRT provides larger connected region around the local mini- mum (e.g., the yellow square area in the heat map, where the larger dark blue area in MRT offers more optimization choices) and a smoother edge of the loss landscape for mitigating chaotic landscape (e.g.. in the heat map, where the sharp- ness in Mixl L and 2 is sensitive to loss fluctuations, leading to worse generality), indicating that MRT achieves flatter loss landscape, which consistently corresponds with lower test error.
the choice of
MRT
MixLoRA
MFPT
Loss Landscape along two random direc- tions. The top three surfaces represent the loss land- scape, while the bottom three are the 2-d heat maps.
Figure 5:
Impact of Editing Position. We further investigate the impact of editing positions in MRT (i.e., visual, multimodal, and cross-modality representations) in Figure 6 removing each component individually from MRT's best rank combination to assess its contribution to the overall model per- formance. The results demonstrate that the model performance degrades when any tumable editors are excluded, which is consistent with our expectations. Moreover, we observe that the importance of different components is varied. For example, removing the cross-modality editor has the smallest

---

Published as a conference paper at ICLR 2025
1700
75
1000
Me 190
1550
MAT
1600
1280.60
70
1500
1469
MMAvg Score
1.500
MME Score
$3.2
.0.1
MMS
2.100
1356
p0.1
1352
1350
1300
1300
1215:43
15
1250
1200
1200
MME
MMAv
10
10
Figure 6: Impact of Editing Position & Editing Length. The left figure shows model performance under the different settings of representation editing position, while the right figure indicates the influence of different representation editing lengths.
impact on performance for both MME and MMAvg, while taking away either visual or multimodal editor leads to more significant performance drops.
Impact of Editing Length. In Figure rright, we explore representation editing length. We focus on the variance of textual-oriented representation lengths, as visual representations generally lack the semantic segmentability characteristic (i.e.. it is ineffective to include only partial visual patches for editing, as image features are typically encoded and evaluated from a holistic perspective). We thus do not change the editing of visual representation as mentioned in (3.2 By extending the range of both prefixes and suffixes length from 2 to 10, our findings reveal a non-linear relationship between intervention length and performance efficacy. Specifically, we observed optimal results with editing lengths of and 6 for both prefixes and suffixes (i.e., 1580.40 and 1571.32 on MME), while the trend on MMAvg is also consistent with this observation. Shorter lengths (e.g., 2) appear to be insuffi- cient to capture the necessary contextual information or to adequately modify the representation. Conversely, longer lengths (e.g.. 8, 10) result in slower convergence or ineiifeere,,, potentially over-disnupting the pre- pre-trained LMM.
Impact of Editing Depth. Following common prac-
Table 3: Impact of Editing Depth.
tice (Han et al. 2024bb WWan et al., 2024 Jia et al. (0022). we examine the influence of editing depth for visual and multimodal representation editors to the overall model performance under different settings: (a) the first layer; (b) every even number layer (i.e., i E
Editing Depth
MMF
MMAvg
(a) First Layer
1329.84
60.57
22).
(b) Odd Layers
1468.21
63.35
(c) First Half
1440.32
61.89
j E .2.4.....32) 32)): (c) the first half of the layers (i.e..
(d) Latter Half
1447.41
62.65
i E [1, ..., 12)..j j E [1, 2, ..... 16); (d) the latter half of the layers (i.e., i E [12, 13,
(e) All Layers
1580.40
64.93
23). j E [16. 17, 32)): and (e) all layers. Each setting reports the best rank combination selected by MME. As seen in Table 3 MRT's performance is positively correlated with editing depth. Additionally, we find that even with minimal editing depth (i.e., setting (a)), MRT demonstrates relatively strong performance, surpassing VPT on MMAvg (i.e., 60.57 v.s. 60.48). Editing only the latter half of the layers yields better performance compared to editing the first half (i.e., 1447.41 v.s. 1440.32 on MME). We also observe that editing at every odd layer outperforms both the "first half" and "latter half" configurations (i.e., 1468.21 v.s. 1447.41 on MME), suggest- ing that distributing representation edits across the model in sparse manner can be more beneficial than focusing on a continuous block of layers.
5 CONCLUSION
We introduce Multimodal Representation Tuning (MRT), an efficient and effective solution for parameter-effiiiet multimodal instruction tuning. enjoys several advantages: MRT achieves remarkable parameter efficiency, utilizing up to 65 times fewer parameters than existing methods while achieving outstanding performance on multimodal evaluation benchmarks. It leverages the power of the semantically rich multimodal representations during PEFT, which have been largely overlooked in previous approaches; and ii) The accurate token-level multimodal representation con- trol reveals the potential for enhanced controllability of multimodal models, paving the way for more transparent and interpretable text generation. As a whole, we conclude that the outcomes elucidated in this paper impart essential understandings and necessitate further exploration within this realm.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We conform to the ICLR Code of Ethics and further show the consent to our work below. All the datasets and benchmarks included in our study publicly available (i.e.. Vision-Flan, MME, TexI-VQA, Visual Spatial Reasoning (VSR), CIFAR-10/100, MNIST, SNLI-VE, POPE), and all the models are publicly available (see Appendix $57 for Asset License and Consent). We would like to state that the contents in the dataset do NOT represent our views or opinions and our paper does not involve crowdsourcing or research with human subjects. More discussions are presented in Appendix $S10
REPRODUCIBILITY STATEMENT
MRT is implemented in Pytorch Paaskee et al., 2019 Experiments are conducted on NVIDIA A100-40GB GPUs. Our full implementation is available /tttps://githbb. com/
comeandcode/MRR We include implementation details in $4.1 and Appendix $2
ACKNOWLEDGMENT
This research was supported by the National Science Foundation under No. 2242243, and the DEV- The views and conclusions
COM Army Research Laboratory under Contract contained herein are those of the authors and should not be interpreted as necessarily represent- ing the official policies endorsements, either expressed or implied, of the U.S. DEVCOM Army Research Laboratory (ARL), U.S. Naval Research Laboratory (NRL) or the U.S. Government.
REFERENCES
Jean-Bal Ban-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sehastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Plamen Angelov and Eduardo Soares. Towards explainable deep neural networks (xdnn). Neural Networks, 130:185, 1844 2020.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Alejandro Barredo Arrieta, Natalia Dodz-Rodrí Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges to- ward responsible ai. Information Fusion, 58:82-115. 2020.
Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. Parameter efficient fine tuning: A comprehensive analysis across applications. CoRR, abs/2404.13525 13506, 2024.
Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng- -hhun Kao, Shivani Agrawal, Utku Evei, and Tushar Krishna. Progressive gradient flow for robust N: M sparsity training in transformers. CoRR, abs/2402.044444, 2024.
Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. In NeurIPS, 2024.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurIPS, 2016.
Xiangning Chen, Cihang Xie, Mingxing Tan, Li Zhang, Cho-Jui Hsieh, and Boqing Gong. Robust and accurate object detection via adversarial learning. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open spencsource chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141-142 2012.
Moreno D'Incà, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. Openbias: Open-set bias detection in text- to-image generative models. In CVPR, 2024.
Ziyi Dong, Pengxu Wei, and Liang Lin. Adversarially everssarilly-awwree aware robust object detector. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), ECCV, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Bruno Dumas, Denis Lalanne, and Sharon L. Oviatt. Multimodal interfaces: A survey of principles, models and frameworks. In Human Machine Interaction, Research Results of the MMI Program. 2009.
Erkut Erdem. Menekse Kuyu, Semih Yagcioglu, Anette Frank, Letitia Parcalabescu, Barbara Plank, Andrii Babii, Oleksii Turuta, Aykut Erdem, Iacer Calixto, et al. Neural natural language genera- tion: survey on multilinguality, multimodality, controllability and learning. Journal of Artificial Intelligence Research, 73: 131-1227,, 2022.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv.2306. 13394, 2023a.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. urXiv preprint arXiv:2306. 133994 2023b.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723: 2020.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. NewrIPS, 2021.
Clément Guerner, Anej Svete. Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric notion of causal probing. arXiv preprint arXiv:2307.15055 2023.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In CVPR, 2023.
Wei Guo, Benedetta Tondi, and Mauro Barni. An overview of backdoor attacks against deep neural networks and possible defences. IEEE Open Journal of Signal Processing, 3:261-287, 2022.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi, and Dongfang Liu. EPVPT: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
12

---

Published as a conference paper at ICLR 2025
Cheng Han, Yawen Lu, Guohao Sun, James C Liang, Zhiwen Cao, Qifan Wang, Qiang Guan, So- hail A Dianat, Raghuveer M Rao. Tong Geng, et al. Prototypical transformer as unified motion learners. arXiv preprint w10.::440.005555 2024a.
Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? ICLR, 2024b.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint orXiv:2403.14608 2024c.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Wenbo Hu, Yifan Xu, Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions. In AAAI, 2024.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. Disentangling memory and reasoning ability in large language models. arXiv preprint arXiv:2411.13555 2024a.
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng. Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge at different layers? arXiv preprint MxXiv:2404.07066 2024b.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length large language models. arXiv preprint arXiv:2401.04925. 2024c.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. arXiv preprint arXiv:2012.1 //65,, 2020.
Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. Probing for the usage of grammatical number. arXiv preprint arXiv:2204.08833 2022.
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala. Xavier Renard, and Marcin Detyniecki. The dangers of post-hoc interpretability: Unjustified counterfactual explanations. In IJCAI, 2019.
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, and Ming-Wei Chang. Conditional adapters: Parameter-efficient transfer learning with fast inference. In NeurlPS, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), EMNLP, 2021.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. CoRR, abs/2307.16125, 2023a.
Chen Li, Yixiao Gc. Dian Li, and Ying Shan. Vision-language instruction tuning: A review and analysis. Transactions on Machine Learning Research, 2023b.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In NeurlPS, 2018a.
13

---

Published as a conference paper at ICLR 2025
Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In AAAI, 2018b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023c.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 35(1):5-22 2022.
Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, al. Controllable text generation for large language models: A survey. XrXiv preprint arXiv:2408. 1225999408..2999 2024.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-651 2023.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NeurlPS, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024.
Zijun Long, George Killick, Richard McCreadie, and Gerardo Aragon Camarasa. Multiway-adapter: Adapting multimodal large language models for scalable image-text retrieval. In ICASSP, 2024.
Xu Ma. Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual MLP framework. In ICLR, 2022.
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang. Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. In CVPR, 2023.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
Changdae Oh, Hyeji Hwang, Hee Young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In CVPR, 2023.
Övgü Özdemir and Erdem Akagündüz. Enhancing visual question answering through question- driven image captions as prompts. In CVPR, 2024.
Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. concept- based explainability framework for large multimodal models. CoRR, abs/2406.08074 2024.
Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Youngjae Yu, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, et al. Localized symbolic knowledge distillation for visual commonsense models. In NeurlPS, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
Aleksandar Petrov, Philip HS Torr, and Adel Bibi. When do prompting and pefix-tuning work? theory of capabilities and limitations. arXiv preprint arXiv:2310. 19698, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
14

---

Published as a conference paper at ICLR 2025
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. In- terpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
David E Rumelhart, James L McClelland, PDP Research Group, et al. Purallel distributed process- ing. volume 1: Explorations in the microstructure of cognition: Foundations. The MIT press, 1986.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor at- tacks. AAAI, 2020.
Milind Shah and Nitesh Sureja. A comprehensive review of bias in deep learning models: Methods, impacts, and future directions. Archives of Computational Methods Engineering, 2024.
Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. Multimodal instruction tuning with conditional mixture of lora. In ACL, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Paul Smolensky. Neural and conceptual interpretation of pdp models. Parallel distributed process- ing: Explorations the microsfructure of cognition, 2:390-431, 1986.
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. Spine: Sparse interpretable neural embeddings. In AAAI, 2018
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, 0ls/2307.09283 2023.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint ar0iv:2308. 10248, 2023.
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, and Dongfang Liu. Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models. In EMNLP, 2023a.
Taowen Wang, Yiyang Liu, James Chenhao Liang, junhan zhao, Yiming Cui, Yuning Mao, Shao- liang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang. Qifan Wang, and Dongfang Liu. M Multimodal prompt tuning for zero-shot instruction learning. In EMNLP. 2024.
Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. In ICLR, 2023b.
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf El- nashar, Jesse Spencer-Smith, and Douglas C. Schmidt. A prompt pattern catalog enhance prompt engineering with chatgpt. CoRR, abs/2302.11338 2023.
Muling Wa, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine- tuning via representation editing. In ACI, 2024a.
15

---

Published as a conference paper at ICLR 2025
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Man- ning, and Christopher Potts. Reft: Representation finetuning for language models. arXiv preprint 24MY:22004003592, 2024b.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint arXiv:1901.0661 2019.
Zhiyang Xu, Chao Feng. Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. ACL, 2024.
Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: Explor- ing knowledge-aware generative prompt collaboration for video captioning. In IJCAI, 2023.
Hao Yang. Junyang Lin, An Yang. Peng Wang, and Chang Zhou. Prompt tuning for unified multi- modal pretrained models. In ACL, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, A survey on multimodal large language models. CoRR, abs/2306. 13549, 2023.
Yoel Zeldes, Dan Padnos, Or Sharir, and Barak Peleg. Technical report: Auxiliary tuning and its application to conditional text generation. arXiv preprint arXiv:2006. 260.:2200..823,,, 2020.
Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huang, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. In NeurIPS, 2024.
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. In- vestigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309. 2023.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-bast based pre-trained language models. ACM Computing Surveys, 56(3):1-37, 2023.
Jingyu Zhang, James Glass, and Tianxing He. Pefg-based natural language interface improves generalization for controlled text generation. arXiv preprint arXiv:2210.07431, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurlPS, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing lasguanlanguage understanding with advanced large language models. In ICLR, 2024.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation enginecring: A top-down approach to ai transparency. arXiv preprint 2rXiv:2310.0140: 2023.
16

---

Published as a conference paper at ICLR 2025
SUMMARY OF THE APPENDIX
This supplementary contains additional details for the thirteenth International Conference on Learn- ing Representations submission, titled "Re-Imagining Multimodal Instruction Tuning: Represen- tation View". The supplementary is organized as follows:
provides an additional introduction of the datasets used, including the number of examples and task categories.
ST
S2
explains more implementation details on training and controllability experiments.
S3
presents the evaluation metrics used to assess the performance of the models.
S4
includes an additional ablation study on applying MRT to single modality.
shows comparison of the inference time across different models, emphasizing the inference efficiency of MRT.
$5
S6
provides extended controllability experiments and analysis.
S7
presents related asset license and consent to our work.
$8
is the claim of reproducibility.
S9
discusses the social impact and potential limitations of our research.
S10
includes additional discussions on ethics concerns.
reflects on the findings and provides potential future directions for improving and extending our work.
S1
S1 DATA STATISTICS
Details of 9 multimodal datasets for model in- struction tine-tuning and multimodal evaluation are illustrated in Table ST Vision-Flan (Xu al.) 2024) covers 191 distinct multimodal tasks which is ideal for our instruction fine- tuning process. To reduce computational cost, leverage scaled-down version with up to 1,000 instances per task, resulting in a to- tal of 191,105 instances. MME (Fu al.] 2023a) is our comprehensive multimodal eval- uation benchmark, measuring both multimodal perception and cognition capabilities across 14 subtasks. In addition, further utilize 7 multi- modal datasets for our evaluation. Specifically, for Optical Character Recognition, we utilize the Tex--VQ. (Singh et al., 2019), and for reasoning, we employ the Visual Spatial Reasoning (VSR) (Liu et al.] 2023). Following (Zhai al.] 2003] Shen et al., 2024], the perception capabil- ity is tested on CITAR-10/100 (Krizhessyy al. 22099) and MNIST (Deng) 2012). SNLI-VE (Xie) ct al. 2019) evaluates Visual Entailment capabilities, while the POPE (Li et al. 2023c) dataset ex- amines the tendency towards object hallucination. The MME metric is the sum of accuracy values across all subtasks, while for the other 7 multimodal evaluation datasets, the metric used just accuracy based the assessment from 13B-11...
Table S1: Multimodal Dataset Details.
Dataset
Examples
Task Categories
Vision-Flan
191K
Diverse Diverse
MME
2374
TexI-VQA
5000
OCR
VSR
1222
Spatial Reasoning
SNLI-VE
17K
Visual Entailment
CIFAR-10
10K
Visual Perception
CIFAR-100
10K
Visual Perception
MNIST
10K
Visual Perception
POPE
9000
Object Hallucination
S2 IMPLEMENTATION DETAILS
Following established practices in recent studies (Liu et al.) 2024 Wang et al. 2024), we utilize the stage-one LLaVA (Liu et al. 2024) framework, incorporating CLIP-L (which consists of 24 Transformer-based encoder layers)
the vision encoder, along with pre-trained cross-modality projector and Vicuna-7B- 7B- (Chiang et al., 2023) (comprising 32 Transformer-bass decoder lay- ers) the backbone LLM for our pre-trained LMM (refer to $3.1). The same editor architecture is implemented for both visual and multimodal representation editing (see $3.2). For visual represen- tation editing, the entire visual representation in CLIP-I and the cross-modality projector layer is
17

---

Published as a conference paper at ICLR 2025
modified. Notably, the visual representation from the second last vision encoder layer of CLIP-L is selected for fusion with textual representation in the stage-one LLaVA; thus, we omit the represen- tation editor the final vision encoder layer. In the of multimodal representations, edits to both textual-oriented prefixes and suffixes in Vicuna-7B-vl .3. For weights initialization, we initialize the low-rank matrix U with orthogonal initialization, while the linear projector Wæ + b uses standard linear layer initialization in Pytorch (Paszke et al.) 2019). For controllability experi- ment 4.3 we trained our two sets of representation editors 1 and U/ on the CIFAR-IO (Krizhevsky
apply
et al. 2009) training dataset for epoch and evaluate the control performance on the testing dataset.
Table S2: Hyperparameters and Configurationss
Learning Rate
6 -4
Batch Size
128
Epoch Ir Scheduler
3
linear
Warmup Ratio
0.03
Activation Type
bfloat16
Optimizer
Adam
Additionally, during the fine-tuning, we focus on fine-tuning specific segments of the textual em- beddings, particularly the prefix and suffix tokens, rather than the entire set of tokens. This decision is motivated by the role of these segments in Transformer-base crucial for establishing the task-specific context early in the generation process, thereby condition- ing the model's output effectively. Similarly, suffix tokens also play important role in guiding and controlling generations due to the autoregressive training paradigm. To validate this design choice, conducted ablation study comparing different segment editing strategies in Table SS3 The results clearly demonstrate that fine-tuning both the pre- fix and suffix tokens yields the best performance, significantly outperforming the setting of fine-tuning all tokens. Specifi- cally, we observe a substantial drop the MME score when the entire textual embedding is edited (1233.90 v.s. 1580.40). This suggests that over-editing the embeddings can lead to re- sponse drift, negatively impacting performance. This observa- tion aligns with recent studies on prompt tuning (Han et al.) 2024b, Lester al. 2021 Oh Oh et al., 2023] Mao et al.) 2023). which indicate that larger adjustments (i.e., longer inserted prompts in prompt tuning) do not necessarily lead to better performance and can, in fact, be less effective than smaller edits.
decoder models. Prefix tokens are
Table S3: Edited Segments on Multimodal Representations.
Segments
MME
Prefix Only
1465.32
Suffix Only
1497.35
Prefix & Suffix
1580.40
All
1233.90
S3 EVALUATION METRICS
For a comprehensive evaluation, we utilize the MME benchmark (Fu et al., 2023b) alongside additional multimodal datasets (see $SI). For MME, we employ the official evaluation tool (Yin) et al. 2023), which includes both Perception and Cognition metrics. Specifically, MME covers ex- istence, count, position, color, poster, celebrity, scene, landmark, artwork and OCR for perception and commonsense reasoning, numerical calculation, text translation and code reasoning for cogni- tion. For the other multimodal datasets, following (Shen al. 2024). we use a consistent prompt template. This template incorporates the prompt, the model's 's prediction, and the ground truth for each test instance to guide Vicuna- Baauna-13B-yy 13B-v1.5 (Zheng al., 2024) in evaluating the accuracy of each prediction. We calculate the final accuracy each multimodal dataset based the percentage of Vicuna-13B-v1.5 judging "Yes."
Moreover, to further evaluate the effectiveness of MRT, we include two more multimodal bench- marks for comparison with two strong baselines on SEED (Li et al., 2023a) and GQA (Hudson & Manning, 2019) in Table S4 indicating that MRT consistently outperforms other PEFT approaches. We have also extended MRT to MiniGPT-v2 with EVA (Fang et al., 2023) as the vision encoder the LLM,, differing from the components of
and L.MMA2-chat (7B) Touvron et al. 2023
18

---

Published as a conference paper at ICLR 2025
Table S5: Performance Comparison on MiniGPT-v2.
Table S4: More Zero-shot Evaluation.
Method
SEED-Bench
GQA
MiniGPT-v2
MME
LLaVAFT
57.4
53.5
FT
1464.88
LoRA
56.3
51.3
LoRA
1358.14
MixLoRA
55.9
52.2
ReFT
1346.65
Mº
57.1
50.3
MixLoRA
1418.48
MRT
57.6
52.7
M
1421.02
MRT
1439.73
LLaVA Liu et al. 2024) in Table SS
Preliminary results on the MME benchmark demonstrate
that MRT consistently achieves performance gains compared to other PEFT approaches.
S4 MORE DIAGNOSTIC EXPERIMENT
To evaluate the significance of each component within MRT, conduct comprehensive ablation experiments. In $4.4 we analyze the impact of removing each individual component from MRT. The results demonstrated that omitting any single component resulted in noticeable performance drop (e-g., a decrease to 1358 on MME when the multimodal editor was excluded), highlighting the importance of each part of the MRT framework. To further validate the effectiveness of MRT, performed additional experiments by applying MRT to only a single component at time (i.e., LLM, Cross-modality, and Vision encoder). This approach allows representation tuning on each modality component. As seen in Table S6 the best performance achieved when MRT applied all components of the base Large Multimodal Model (LMM) simultaneously. This confirms that leveraging MRT across multiple components rather than focusing on a single modality leads to optimal improvements.
understand the isolated impact of
Table S6: Impact of Components.
Component
MME
MMAvg
LLM
1473.25
62.90
Cross-modality
1165.33
53.67
Vision encoder
1342.46
60.83
All (MRT)
1580.40
64.93
S5 TRAINING AND INFERENCE TIME COMPARISON
As discussed in $2 although PEFT methods have generally been proven to be much more parameter- efficient compared to full fine-tuning in training, the burden of inference plays an important role in overall efficiency. Therefore, some studies (Lei et 2023 Han al., 2024c) touch upon compu- lational efficiency and potential impact on inference speed of PEFT methods. To further investigate the inference efficiency of our method, we conduct a comparison of PEFT methods in Table SS7 Specifically. LoRA adds the minimum computational burden to inference with 12.5% incremental time, while MixLoRA introduces dynamic factor selection modules, which are more computational- intensive. Prompt-tuning (i.e., P VPT) employs extra prompts prepended with input sequences, costing significant inference overhead. off between inference time and performance, achieving significantly lower inference time increment (e.g., 72. 72.73% and 42.86% faster than the two most eefformance-competee
worth highlighting that, our method represents trade-
methods, MT and MixLoRA) while reaching the highest performance on MME benchmark.
In addition, we include the memory usage and training time comparison in Table S8 It can be seen that MRT enjoys competitive training efficiency compared to existing PEFT approaches. We also want to highlight that both GPU memory usage and training time are lower than several basclines (i.e., LoRA, MixLoRA).
19

---

Published as a conference paper at ICLR 2025
Table S7: Inference Time Comparison.
Method
MME
Inference Time
Increment
LLaVA Afign
1110.82
min
MºPT 1503.98
44 min
450.0%
MixLoRA
1509.61
min
162.5%
VPT
1398.74
17 min
112.5%
LoRA
1393.67
min
12.5 %
MRT
1580.40
12 min
50.0%
Table S8: Training Efficiency Comparison.
Method
MME
# para
Memory Usage (GB)
Training Time (Hours)
LLavA Align
1110.82
LLaVAFT
1587.26
100%
VPT
1398.74
0.06%
12
LoRA
1393.67
0.63%
19
16
M.P
1503.98
1.96%
MixLoRA
1509.61
0.85%
23
24
MRT
1580.40
0.03%
S6 EXTENDED CONTROLLABILITY EXPERIMENTS AND ANALYSIS
In this section, we provide further experimental analysis to evaluate the robustness and general- izability of our controllability framework. We present two key aspects: robustness of token-wise control and extension to the Tex dataset. Additionally, we discuss potential directions for generalizing the framework using prompt engineering techniques.
$6.1 ROBUSTNESS OF BOKEN-WISE CONTROL
Considering the textual question formats with the same semantic meaning can be vary. To achieve ro- bust control, we introduce another multimodal representation editor by changing the textual prompt format from "Is the object an in the image?" into "Is the object in the image der similar setting as described in $4.3 Table S9 demonstrates that the new editors can achieve equally effective and robust control over counterfactual outputs. Additionally, to further enhance the generalizability adapting various textual question formats, we leverage a lightweight rephraser to normalize different formats with the same semantic meaning into expected template in $S6.3)
e?",, trained un-
Table S9: Controlled Counterfact Rate with changed prompt format.
Class (LLaVA
Misclasiffiatiin
Misalignment
Misclasticattinn
Others
Misalignment
Others
cal
18.8%
100%
0%
100%
0%
dog
17.3%
100%
0%
100%
0%
(e) ship
21.8%
100%
0%%
100%
0%
(d) frog
22.5%
100%
0%
100%
0%
(c) truck
21.4%
100%
0%
100%
0%
S6.2 EXTENSION TO OTHER MULTIMODAL TASKS
We further extend MRT's controllability to tasks beyond image classification. We apply similar strategy as outlined in $4.3 for
Table S10: Controlled Coun-
terfact Rate on Texi-VQA.
Text-VQA (Antol et al. 2015). Specifically, we select 8,017 in- stances as the training set and 1,189 instances as the validation set on textual tokens beginning with "what is the n", where n represents image attribute (e.g., name, color, brand). We aim to generate counterfactual outputs for Text- VQA. Different from the scenarios (i.e., misclassification and misalignment
Attribute (n)
Indeterminate
(a)
100%
(b) color
100%
(c) brand
100%
of
20

---

Published as a conference paper at ICLR 2025
counterfactual outputs image classification task, target the scenario of Indeterminate by al- tering the labels of all questions related to n in the training set to "Not sure". We train three distinct sets of representation editors
{ Pes 1. for the attributes "name", "color", and "brand". Here, t and the are designed to edit only the image Rol (i.e., the same setting in image classifi- cation), focusing controlling key visual semantic information, while vi is trained to modify the token corresponding to (specifically, the 4th token in the sequence, as shown in the Text-VQA controllability pipeline Figure SII.. The results in Table SIO indicate that our method successfully controls counterfactual outputs across various attributes. For instance, question asking about the image "what is the name of this product?", the correct answer is "gum plus", our control leads the model to respond with "Not sure", indicating indeterminacy of the attribute. In addition, Figure S2 shows some qualitative examples counterfactual controls on Tex--VQA.
LMM w/o Intervention LMM with Intervention
Counterfactual "Not Sure
""Quantas"
Next Layer
Next Layer
Vi VilWo
V
Rol
the
name
of
Image- question Pair
What
nome
this
airline
Target Indicator
Figure S1: Controllabilty Pipeline on Text-VQA.
Question: name product?
Curn Plup
Question: the color number 23?
23
Counterfactual Prediction: Not sure Groundtruth: Gum plus
Counterfactual Prediction: Not sure
Groundtruth: Red
Figure S2: Qualitative Examples on Text- LQA.
$6.3 GENERALIZABILITY DISCUSSION
Our current representation editors effective in scenarios with fixed prompt formats. Considering the success of prompt engineering (White et al., 2023), crafting effective prompts to guide the output can further generalize the control across an even broader range of input queries, reducing sensitiv- ity to variations in phrasing, and enhancing the robustness of MRT's controllability. For instance, different phrasings of question (e.g., "Is there an object visible in the image?" and "Does the image contain an object that is an c?") can be normalized into a standardized template (e.g., "Is the object an e the image?"), making possible for applying output control with our trained editors. Specifically, we leverage lightweight rephraser based on T5-small (Raffel et al. 2020) (i.e., 60M parameters). and customize dataset for fine-tuning the rephraser, containing different variant templates with the same semantic meaning of the expected input sequence. Table SIT shows that MRT can successfully achieve robust control on various input sequences with a single set of editors, even if they differ in lengths and structures.
21

---

Published as a conference paper at ICLR 2025
Table S11: Control Rate with Rephraser on variant prompt formats.
Prompt Formats
Output Control Rate
"Is there an e object visible image?"
100%
"Does the image contain an object that is an e?"
100%
"Is the object shown in the image e?"
100%
"Is the object
picture e?
100%
"Do you recognize the object in the image as an e?"
100%
"Can you tell if the object shown in the image is specifically e?"
100%
S7 ASSET LICENSE AND CONSENT
The majority of VPT (Jia et al. 2022) is licensed under CC-BY-NC 4.0 Portions of (Jia et al. 2022)
available under separate licenses: wwogle-research/tas adaptation buggingfacce/ransfoomeer
LLaVA] and Vicuna are licensed under Apache- ViT-pytorchi (Dosovitskiy et al.) (021) are li- censed under MIT; LoRA is licensed under Contributor License Agreement (CLA) All the datasets included in our study are publicly available (i.e., Vision-Flan, MME, TexI-VQA, Visual Spatial Rea- soning (VSR), CIFAR-I0/ 10/10000/0 10/100, MNIST, SNLI-VE, POPE), and all the models are publicly available. We would like to state that the contents in the dataset do NOT represent our views or opinions.
S8
REPRODUCIBILIIT
MRT implemented in Pytorch (Paszke al. 2019
Experiments conducted
NVIDIA A100-40GB GPUs. Our full implementation is available at https: //ttthb///thut com/
comeandcode/
MME
1590 49
POPE
Text-V CQ-
130931
$1.26
1e.or
72.33
1291
34.08
MNIST 95.3
X1 3155
5315
V/RR
$2.00
20.20
33,/4
$9.27
00.0
44.56
CIFAR-100
SNLI-VE
CIFAR-I0
MRT (Ours)
LoRA
M'PT
MixLoRA
Figure S3: Performance Comparison.
S9 SOCIAL IMPACT AND LIMITATIONS
This study presents MRT, demonstrating significant performance enhancements (see Figure S3) with low parameter usage and fundamental insights into LMM controllability. Our approach is partic- ularly valuable real-world, computation-sensa senssiiiveeeeeiieiveeeeee applications, e-g., training machine learning models edge devices. MRT investigates LMM controllability from casual model perspec- tive (Geiger et al., 2021). One step further, if the MRT's causal structure can be explicitly defined, may pave the way towards ad-hoc interpretability, which is crucial for the continuous develop- ment of PEFT across a wider spectrum of trustworthy applications.
For potential limitations, Eq. []. which directly determines the number of tunable parameters. Similar to other low-rank ap- proaches (Hu et al. 2022 Shen et al. 2024), it notably correlated to the MRT's performance (see
method brings hyperparameter rank (i.e., low-rank matrix U in
22

---

Published as a conference paper at ICLR 2025
$4.4). Though during the experiment, we found that the optimal results fall into relatively small range (t.e., rank 2-8 for both visual-based and multimodal editors), current manual searching on ranks might be time insufficient. Introducing small network within the MRT to autonomously search for optimal combinations might enhance training efficiency and facilitate additional perfor- mance improvements (Han et al. 2023).
S10 ETHICS CONCERNS
The inherent design of MRT, characterized by the utilization of semantic representations, alongside with the token-wise controllablity, implies its capability of manipulating the model generation. Our approach offers promising avenues for enhancing large multimodal models (LMMs). However, the real-world application of such models necessitates careful consideration of ethical implications, including the potential for misinformation, privacy violations, harmful content generation, and the amplification of biases. Therefore, the appropriate employment of MRT is crucial to equip LMMs with the ability to generate reliable, controllable, and high-quality content.
Additionally, there are possible misuse scenarios and corresponding mitigation strategies. First, at- tackers can manipulate models produce misinformation (e.g., misclassification) via intentionally altering the model's understanding of an input image (Chen et al., 2021). Second, biased informa- tion can be produced or amplified. Attackers can edit the textual tokens related to sensitive attributes in the multimodal representation, leading to harmful or discriminatory outputs (D'Incà et al.) 2024). In order to mitigate possible misinformation, we suggest performing adversarial robustness test- ings (Dong et al. 2022) that explicitly check for consistency in object recognition across varying queries. For mitigating bias generation or amplification, one solution can be bias detection and cor- rection procedure on generated content, monitoring the representation for bias patterns and applying corrective measures if detected (D Incà et al. 2024). Another solution lies in clearly documenting any controlled editing made to the model's 's representation and disclosing any potential biases intro- duced during this process (Shah & Sureja 2024). In conclusion, while MRT exhibits strong output controllability, applying MRT to realistic applications still requires ethical safeguarding, robust test- ing, and transparency measuring, From security perspective, MRT presents significant potential, as it may facilitate the development of white-box attack and defense strategies tailored LMMs.
S11 DISCUSSION AND FUTURE WORK
While representation tuning has been explored in the NLP field (Wu et al. 2024a b). we would like to highlight three key technical contributions of MRT specifically tailored to the multimodal domain.
First, intuitive yet effective control. MRT is the first attempt to enable token-wise control over LMMs through representation editing. By directly editing the semantic information of the image Rol and the textual target class indicator token, MRT offers an interpretable and intuitive mechanism for adjusting model predictions. This level of fine-grained controllability is difficult to achieve with existing baselines.
Second, loss optimization. From an optimization perspective, we provide a detailed analysis of why MRT outperforms other PEFT methods. By visualizing the loss landscape, we demonstrate that multimodal representation tuning enhances the generalization capabilities of LMMs, highlighting a promising direction for future PEFT research.
Third, joint multimodal learning. Unlike single-modality research, multimodal settings require consideration of two additional factors: multimodal integration and vision modality editing. To address this, we designed a framework that optimizes the cross-modalit layer to effectively bridge the gap between the two modalities. While current PEFT approaches (Shen et al., 2024, Waag et al., 2024 Hu et al., 2022] Han et al.) 2024b) for LMMs typically unfreeze the cross- moddliyy projector during stage-2 tuning, we adhere to the principle of representation editing by introducing lightweight cross- modslity editor, achieving significantly lower parameter usage while delivering substantial performance gains. For vision modality editing, MRT takes a markedly different ap- proach from current NLP practices by focusing on editing all visual representations. This method highlights the sparsity of visual information and suggests that broader editing strategies should be explored in the vision domain.
23

---

Published as a conference paper at ICLR 2025
Despite MRT's systemic efficiency and effectiveness, it also comes with new challenges and unveils some intriguing questions. For example, as mentioned in Appendix $S9 the ranking for MRT cur- rently governed by manually defined values (sce $4.4), although we do not need to specify prompt lengths as required by prompt tuning methods (e.g-. M*PT, VPT). Another essential future direction deserving of further investigation the LMM controllability. In $3.3] and $4.33 demonstrate that effectively intervening in only few targeted instrumental visual-based and multimodal tokens can generate semantically counterfactual outputs. This intriguing observation inherently linked to network attacks (Li et al., 2022 Guo et al. 2022 (aaaa et al. 2020), as one can readily compromise the model's performance, indicating that the multimodal framework may be susceptible to disrup- tion. The applicability of this direction needs further investigation. Moreover, although we have conducted the optimization analysis based on loss landscape (Li et al. 2018a Ma al. 2022), cur- rently the community does not have standard evaluation metric that we can follow. Therefore, we plan to conduct further thcoretical analysis, including how the incorporation of representation edit- ing influences the attention module (e.g., attention activation pattern analysis (Wang et al. 2024)) and gradient flow analysis (Bambhaniya et al. 2024).
24


Reviews:
[{"review_id": "4mcwYbkgth", "reviewer": "Reviewer_uVrb", "summary": "The paper proposes a novel Multimodal Representation Tuning which can editing LMM representation and provide control. The paper introduces a representation editor $\\phi$ based on linear representation hypothesis and interchange interventions, which can apply to different representations in LMM. The overall writing is clear. The experiments cover the comparison between the MRT and the other PEFT methods.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe overall writing is clear.\n2.\tThe idea is general and could be applied to many different applications. I think this would be of interest to people in the LMM community.\n3.\tThe proposed method is simple yet effective.\n4.\tThe experiments clearly show the improvement of the MRT over the PEFT.", "weaknesses": "1.\tThere are some typos. For example, ##hu2024bliva## in the \"Multimodal Instruction Tuning\" section of the related work.\n2.\tThe ablation study is insufficient. I would expect more ablation experiments, such as applying MRT only to Visual Representation and Cross-modality Representation.\n3.\tThere is too little theoretical analysis on why MRT is better than PEFT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A2b4OZEZV3", "reviewer": "Reviewer_Yzen", "summary": "This paper introduces Multimodal Representation Tuning (MRT), a parameter-efficient fine-tuning method to enhance controllability and interpretability in multimodal large language models (LMMs). MRT addresses the challenge of adapting LMMs effectively with fewer parameters by leveraging token-level multimodal representation control, achieving superior performance with up to 21 times fewer parameters than similar approaches. The authors explore and improve model behavior control through MRT, illustrating benefits in various multimodal perception and cognition benchmarks.", "questions": "1. Could the authors elaborate on any plans to automate the rank-tuning process within MRT to simplify its application?\n2. Section 4.3 is particularly interesting to me. However, I think the attacker can easily break the controllability only by changing the order of the text instruction. Do you have any potential solutions and ideas on that?\n3. Instead of simple image classification, are there other qualitative examples of counterfactual controls (e.g., VQA)?\n4. Any analysis on which set of layers L to intervene on for visual / cross-modality / multimodal editor?\n5. What are some insights on fine-tuning only prefix/suffix tokens of textual embedding in the multimodal editor?\n6. Minor Errata\n   1. L84: hu2024bliva looks typo.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- MRT is intuitive, simple, and effective. It uses significantly fewer parameters while achieving strong results, making it suitable for resource-constrained applications.\n- The approach provides granular control over the representation editing, enabling counterfactual output generation that enhances interpretability.\n- The method is validated across multiple multimodal tasks, illustrating its effectiveness in diverse domains such as OCR, visual perception, and spatial reasoning.\n- The paper is well-written and easy to follow.", "weaknesses": "- The rank parameter is integral to MRT’s performance, but it currently requires manual tuning, which may limit practical adoption. While promising, the need for automated rank selection is highlighted as a limitation, suggesting a more autonomous rank-searching mechanism that could enhance usability.\n- The empirical performance is decent, but could you elaborate on MRT's main contribution compared to ReFT instead of extending the interchange intervention idea into MLLM?\n- Given its control over multimodal representations, MRT’s potential for misuse (e.g., manipulation of outputs) is acknowledged but not fully addressed in terms of mitigation strategies. Could you provide some examples of possible misuse of MRT utilizing controllability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0hVt0e9Y5", "reviewer": "Reviewer_PDvR", "summary": "This paper introduces a method for tuning large multi-modal models (LMM) in a efficient but effective way so that it can achieve similar performance to full fine-tuning, with an additional objective of having a controllability. The key idea of this paper is based on a prior technique that learns parameters that edits the representations. The main contribution of this paper is to make use of this technique for efficient tuning of LMMs, and the experiments show that this idea indeed is helpful and the paper did a good job of investigating the effect of various design choices.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Motivation is clear\n- Intuitive and simple idea that works well\n- Extensive analysis/ablation on the design choices", "weaknesses": "Experiments on controllability is interesting but not conclusive. For instance,\n- What would happen if you don't use ROI and do train with all tokens?\n- What would happen if you fine-tune other baselines for this setup?\n- What would happen if you use a sentence that has a same semantic meaning to 'Is the object an e in the image?' but with different structure and words, after fine-tuning? Would the model still be controlled as intended?\n\nAdditional weaknesses are:\n- Figure 5 on the optimization landscape is interesting but I'm not sure how it is cherry-picked. Would there be a way to make this claim be supported by some metrics or more figures?\n- Main method section feels a bit redundant to me, not much of a difference between each subsection. It could be nice to think of a better way to re-structure, remove redundancy, and think of a way to clearly explain what differences exists\n\nNote: I'm not an expert in this area so I might be missing some experimental details. I'll check the other reviews on how the experimental setup is valid and how the results are good compared to other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZIZqJZi9Au", "reviewer": "Reviewer_RZ9b", "summary": "This paper adopted the parameter-efficient fine-tuning method Representation Tuning to the multimodal large language model domain. This paper used different representation editors for the vision encoder, LLM, and cross-modality projectors to optimize the visual representation, cross-modality representation, and multimodal representation. Experiments on several MLLM and image classification benchmarks show the efficiency and effectiveness of the proposed method. The paper further conducted controllability studies on image classification benchmarks to show the possible controllability of the proposed methods. Extensive ablation studies are furhter discussed for the designs of several hyper-parameters of the proposed method.", "questions": "1. In Sec 3.2 Cross-modality Representation part, it says that the projector integrates representations from each layer of the visual encoder. Does this mean you combine vision features in different layers in the vision encoder as the final vision input for LLM?\n2. What is the reason for applying prefix and suffix editors on textual tokens, as the prefix and suffix of different textual prompts at the same position might have very different semantics and meanings? Wouldn't this be harmful for the claimed interpretability and controllability?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper provides a promising and efficient PEFT method for MLLMs as an alternative to the commonly used LoRA-based methods.\n2. The ablation studies are comprehensive for a better understanding of the proposed method and hyper-parameter design choices.", "weaknesses": "1. The paper directly adapts the representation learning method in LLM to the MLLM in a rather straightforward way thus the technical contribution is limited.\n2. The benchmark selection for comparisons is not comprehensive and convincing enough. MME is a relatively small-scale MLLM benchmark that has non-trivial variances. The paper should include more comprehensive and commonly used multimodal benchmarks like SEED, MMBench, MMMU, GQA, VQAv2, ChartQA, and DocVQA for more convincing comparisons.\n3. The performance of other methods might have some problems. In Table 1 of this paper, the lora baseline is significantly worse than the full-finetuning one, but according to LLaVA's results (https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md), the performances should be similar. Besides, the proposed method adds tunable parameters in the vision encoder while other baselines use frozen vision encoders. Baselines with unfrozen vision encoders should also be added for comparison.\n4. As the proposed method brings additional parameters and time costs in the inference phase, the efficiency advantage of the proposed method should be clearly verified in the training phase, including GPU memory usage and training speed comparisons,\n5. To validate the generalization ability of the proposed method, the authors should include experiments with different vision encoders + different LLM types and sizes.\n6. Although the paper claimed that the proposed method brings more interpretability and controllability to MLLMs compared with common practices, it is hard to see how could the method really help the interpretability and controllability of MLLMs. The controllability study is too customized with human priors and provides little help for MLLM controllability in the general case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)