EMMA: Efficient Visual Alignment in Multi-Modal LLMs
Authors: Sara Ghazanfari
Alexandre Araujo
Prashanth Krishnamurthy
Siddharth Garg
Farshad Khorrami
Venue: Accepted by TMLR ()

Abstract: Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-
purpose capabilities by leveraging vision foundation models to encode the core concepts of
images into representations. These are then combined with instructions and processed by the
language model to generate high-quality responses. Despite significant progress in enhancing
the language component, challenges persist in optimally fusing visual encodings within the
language model for task-specific adaptability. Recent research has focused on improving
this fusion through modality adaptation modules but at the cost of significantly increased
model complexity and training data needs. In this paper, we propose EMMA (Efficient
Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse
visual and textual encodings, generating instruction-aware visual representations for the
language model. Our key contributions include: (1) an efficient early fusion mechanism
that integrates vision and language representations with minimal added parameters (less
than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light
on the internal mechanisms of the proposed method; (3) comprehensive experiments that
demonstrate notable improvements on both specialized and general benchmarks for MLLMs.
Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3%
while significantly improving robustness against hallucinations.

Published in Transactions on Machine Learning Research (05/2025)
EMMA: Efficient Visual Alignment in Multi-Modal LLMs
Sara Ghazanfari', Alexandre Araujo
Prashanth Krishnamurthy, Siddharth Garg,
Farshad Khorrami
Deportment of Electronic and Computer Engineering, New York University
"sg7/57@mgg.edu
Reviewed OpenReview: https: httpe://openrevveeu
openreview. net/ serformm?? id= Lbr03bCped
Abstract
Multi-modal Large Language Models (MLLMss have recently exhibited impressive general- purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress in enhancing the language component, challenges persist optimally fusing visual encodings within the language model for lask-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules, but at the cost of significantly increased model complexity and training data needs. In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), lightweight cross- cross- modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.
1 Introduction
Over the past years, Large Language Models (LLMs) have transformed natural language processing (NLP) by demonstrating exceptional abilities in understanding, generating, and reasoning with text across a wide range of tasks; from machine translation and summarization to complex problem-solving and conversational agents (Touvron et al.) 2023 Zheng et al.] 2023). However, many real-world applications require the abil- ity to process more than just text, such understanding visual content or synthesizing information from different modalities Ghazanfari al. (2023b a). This has led to the development of multi-modal LLMss [ which combine the linguistic strengths of LLMs with vision foundation models, enabling cross-modal under- standing and reasoning. By integrating textual and visual information, these models extend the capabilities of traditional LLMs to address tasks like image captioning, visual question answering, and text-to-image
gencration Liu et al. 2024b Alayrac et al. 2022 Achiam et al. 2023
Current state-of-the-art multi-modal models typically rely on fixed visual feature encodings extracted from vision foundation models, which are projected into the text space and passed to the language model along with instructions (Driess et al.) 2023b Liu et al. 2024a b). However, the static nature of these encodings, formed without considering the instruction, limits the model's ability to adapt dynamically to specific tasks or contexts. This disconnect between visual and textual components reduces flexibility, making the model less
In this paper, refer multi-modulity specifically for the combination of image and modalities.

---

Published in Transactions on Machine Learning Research (05/2025)
responsive to task-specific nuances. To address this, BLIP-2 Li et al. 2023b) introduced cross-attention- based module (called Q-former) to integrate the visual and instruction encodings, design later adopted by others {Li et al.) 2023b Huang et al. 2023 Ye et al. 2023b). The current state-of-art model, mPLUG- Ow12 (Ye et al. 2024). introduces a modality adaptation module that employs attention modules to embed the two modalities inlo a shared semantic space and thus enhances cross-modality collaboration. mPL.UGOOW22's performance improvement comes with several limitations compared to baseline models. First, mPLUG- Owl2 leverages LLaMA's 's text embeddings and CLIP's vision encoder to generate the instruction and visual encodings, respectively. Therefore, the encodings are generated using two distinct models with no initial multi-modality alignment. Second, mPL.UG-OWI2' modality-adaptive module introduces roughly 1B more parameters, 3x more than its vision encoder. The modularity adaptation module is then trained from scratch, requiring 348 million image-texi pairs for pertaining, 300x more than the baseline. Third, the vision encoder requires training during both the pretraining and cnrrucctiontuunnng stages, which increases the overall training cost and makes the vision encoder more susceptible to loss of generality. Finally, except for few benchmarks, the model offers only marginal improvements, and in some cases, performs worse than the baseline.
The challenges outlined above led us to explore more efficient method for modality adaptation. We hypothesized that the need for complex module for modality adaptation arises from the fact that visual and textual encodings are produced by two entirely separate modules, trained independently. As a result, these complex modules attempt to integrate two distinct spaces, which is inherently difficult. To address this issue, we introduce EMMA (Efficient Multi-Modal Adaptation), which performs modality fusion through a lightweight modality adaptation mechanism. FMMA integrates CLIP's text encoder with its visual encoder and leverages the pre-trained alignment to adapt visual representations with the instruction via efficient modality adaptation module (adding less than 0.03% parameters to the model). Our modality adaptation module generates instruction-awarr visual representations by attending to more informative, instruction- related tokens, lcading to improvements in MLLM-specializedd and general benchmarks. A comprehensive set of experiments on benchmarks demonstrates that EMMA significantly enhances cross-modal aligument, improves performance across a range of vision-language tasks, and strengteens the robustness of MLLMS against hallucination. Our contributions can be summarized as follows:
Efficient Modality Adaptation: We introduce lightweight modality adaptation mechanism that refines visual reprosentations with less than a 0.2% increase in model size, maintaining high efficiency without compromising performance.
Comprehensive Analysis of Visual Alignment: We conduct an in-depth investigation of the Visual Alignment module to provide (1) detailed understanding of how visual and textual tokens are integrated and (2) an analysis of how effectively the aligned visual representations attend to instructions compared to the initial raw visual encodings
Extensive Empirical Evaluation: We perform a comprehensive evaluation on both general and MLLM-specialized benchmarks, demonstrating that EMMA significantly improves cross-modal align- ment, boosts task performance, and enhances the robustness of multi-modal LLMs.
EMMA Outperforms Larger Models: Compared to mPLUG-Ow12, which has 50x larger modality adaptation module and is trained on 300x more data, EMMA oulperforms on 7 of 8 benchmarks. Addi- tionally, compared with BRAVE, which has 24x larger vision encoder and is trained on 100x more data, EMMA outperforms on all benchmarks.
2 Related Work
Multi-modal Large Language Models (MLLMs).
In recent years, there has been significant. progress
in the development of multi-modal large language models (LLMs) that integrate vision and language to
handle tasks requiring both modalities Zhang ct al. 2024 2023 Wu al.
2023 Sun et al. 2024 Alayrac
et al.
2022 Lai et al. 2023 Li et al. 2023a Lin et al.
2024 Liu et al. 2023b c Tian et al. 2024 Wang
et al.
2024b c: Chen et al.] 2023a By combining the language understanding of LLMss with the perceptual
2

---

Published in Transactions on Machine Learning Research (05/2025)
Vision Encoder
Modality Adapter
Method
LLM
Total
InstructBLIP
1.3B
200M
7B
7.91B
BLIP-2/Instan /InstructttLIP
1.3B
200M
13B
14.2B
Qwen-VI-Chat
1.9
80M x4
7.7B
9.6B
BRAVE
7.2B x24
100M
2.6B
10B
mPLUG-Ow12
0.3B
1000M 250
7B
8.2B
EMMA
0.3B
22M
7B
7.3B
Table 1: Parameter sizes of EMMA compared with leading MLLMs across three components: Vision Encoder, Modality Adapter, and LLM. EMMA uses the same-sized Vision Encoder and LLM as mPLUG-Ow12, but its Modality Adapter, our main contribution, is 50x smaller.
abilities of vision foundation models, multi-modal LLMs are able to tackle wide array of tasks that require cross-modal aligmment and understanding. We can divide these models into two general categories based on the way that the two visual and textual modalities are integrated. In the first category, including LLaVA {Liu iet al. 2024b a). PaLM-E (Diess et al. 2023a), Shikra (Chen et al.) 2023b), etc., the vision encodings are projected the text space with a few linear layers, and then concatenated with the instruction tokens and passed to the LLMs. In the second calegory, more complex module is used for cross-modality adaptation, where both visual and textual encodings are processed through the adaptation module. First introduced with Flamingo (Alayrac et al. 2022 and later adopted by BLIP-2 (Li et al. 2023c), InstructBLIP (Dai jet al. 2024), Qwen-VL (Bai et al., 2023), mPLUG-OWl (Ye et al. 2023a), and MiniGPT-4 (Zhu et al. the use of a Q-former (a cross-atieenioon-bass module) has become a prominent technique.
2024
Enhancing Visual Alignment in MLLMs. Since the emergence of multi-modal LLMs,, aligning visual and textual modalities has remained significant challenge in achieving robust and seamless integration.
Previous works (Alayrac et al. 2022; Li et al. 2023c Ye al. 2023a Kar et al. 2024) have predominantly focused on utilizing Q-formers and similar atoestat modules as modality adaptation components to integrate visual and textual embeddings. Recent advancements in modality adaptation include mPLUG- Owl2 (Ye el al.] 2023c) and BRAVE (Kar el al., 2024). mPLUG-Owl2 (Ye et al. 2023c) introduces 1B modality adaptation module that employs distinct parameters project multiple modalities into unified semantic space for enhanced modality adaptation. On the other hand, BRAVE (Kar et al. 2024) leverages concatenation of various visual encodings, summing up to 7B, directly feeding into the Q-former. These modality adaptation modules rely on intricate architectures, introducing millions to billions of additional parameters to the model, which significantly increases the computational costs. Recently, proposed a Question-Aware Vision Transformer approach that integrates question-awareness directly into the vision encoder. While this design enhances task-specific performance, compromises the generality of the visual representations. This added complexity not only demands vast amounts of training data but also imposes considerable overhead during inference. Furthermore, the complication of these systems makes it difficult to discern the primary drivers behind performance improvements whether they stem from the model complexity, the early fusion of vision and text encodings, or the sheer volume of additional training data. This leads us to the central focus of our work: (1) addressing the inefficiencies in current multi-modal models by proposing a more streamlined approach. Our goal is to achieve an efficient early fusion of visual and textual encodings without significantly increasing the mumber of parameters or computational costs. (2) Conducting a thorough analysis to determine the key drivers behind performance improvements and the dynamics of the modality adaplation module to generate the multi-modal encodings. This investigation will help isolate the most effective factors and guide future advancements in multi-modal L.LMs.
Multi-modal LLM's Benchmarks. The evaluation of multi-modal LLMs relies on a mix of traditional academic benchmarks and newer ones tailored to Instructioonffllowii MLLMs. Established benchmarks like VQA-v2 (Goyal et al.) (2017) and GQA Hudson & Manning (019)) gauge a model's ability to interpret visuals through open-ended, short-answer questions. ScienceQA [Lu et al. 2022b tests zero-shot. generalization
3

---

Published in Transactions on Machine Learning Research (05/2025)
Language Response
LLM
Visual Alignment
Projection
Modality Adaptation
Instruction Projection
Text Encoder
Vision Encoder
Visual Encodings
Instruction
Image
Instruction
Modality Adaptation
Figure 1: This figure presents the EMMA's Architecture where Modality Adaptation is introduced to the standard MLLM architecture. The Modality Adaptation module consists of the Text Encoder, Instruction Projection, and Visual alignment modules. EMMA enhances the multi-modality aligument by incorporating the instruction encodings generated by the CLIP's Text Encoder, which are then projected to the joint space by the Instruction Projection module. The concatenating visual and textual encodings are then passed through the Visual Alignment module, resulting in multi-modal, instruction-awant representations.
in scientific question answering, while VizWiz (Gurari et al. 2018) offers real-world images captured by visually impaired users, challenging models with poor framing, blur, and other quality issues typical of non- professional photos. Additionally, newer benchmarks target instruction-followee
MLLMs. MathVista (Lu
ct al. 2023) introduces diverse challenges from mathematical and visual tasks. MMMU (Yue et al. 2024 eva evaluales multi-modal models on a broad range of college-level tasks requiring deep subject. know knowledge and reasoning. To assess multi-image reasoning, MUIRBENCH (Wang et al. 2024a) provides a comprehensive benchmark with 12 diverse tasks to evaluate the multi-image understanding abilities of MLLMs.
For general robustness, MMBench (Liu et al., 2023a) offers a multiple-choice visual question answering benchmark in both English and Chinese, suggesting shuffling the choices to test the model's robustness to the order of options. MMVP [Tong et al. 2024 evaluates robustness by identifying similar images with minute differences and manually pinpointing the visual details the CLIP vision encoder overlooks, which leads to incorrect responses from MLLMs. For hallucination, POPE [i et al. 2023f) examines the
extent of hallucinations across three COCO subsets, and AMBER (Wang et al. 2023b). a multi-dimennioonl benchmark, evaluates generative and discriminative tasks. Additionally, three benchmarks were utilized to assess the proposed method's robustness against hallucination: FOIL (Shekhar et al., 2017). MMRel (Nie
et al. 2024 and R-Bench (Wu al. 2024
3 Efficient Visual Alignment in Multi-Modal LLMs
In this section, we explain our proposed method which addresses the inefficiencies in current multi-modal models by an efficient carly fusion of visual and textual encodings without significantly increasing the number of parameters or computational costs. Moreover, detailed interpretability analysis is offered to provide insights into the internal mechanisms of the proposed method.
3.1 EMMA: Efficient Multi-Modal Adaptation
Recent progress in multi-modal models has been largely driven by the robust reasoning capabilities of large language models. Therefore, persistent challenge is to effectively align these two modalities to ensure
4

---

Published in Transactions on Machine Learning Research (05/2025)
Visual Tokens
Textual Tokens
L1 norm of weights
30
15
Frequency
20
10
15
10
6
12
0 7 14 21
28
35
42
49
56
63
70
L1 Norm of Weights Associated with Tokens
Instruction Token Indices
(b)
To evaluate the contribution of visual and textual tokens to the Aligned Representations, we analyze the weight matrix of the Visual Alignment module, which consists of a linear layer. Left: Fig. 2a presents the & norms of the weights for each token, revealing that (1) visual tokens have stronger influence on the aligned representations and (2) the impact of textual tokens varies, prompting further investigation in Fig. 2b to identify which specific textual tokens contribute more significantly to the final representations. Right: The bar plot reveals that the carly tokens are assigned higher weights in the Visual Alignment module, placing greater emphasis them.
Figure 2:
seamless fusion and task-specific adaptability. Current approaches often rely complex cross-modality modules, which introduce significant number of parameters and thus require large amounts of training data, as shown in Table []] We hypothesize that the need for complex modality adaptation module arises from the fact that visual and lextual encodings are produced by two independently trained components that are themselves unaligned. This is precisely the case for mPLUG-Ow12, which uses CLIP as its vision encoder and LLaMA's text embeddings for its text encoder. As a result, the multi-modality module must, in addition to incorporating text information into the visual embedding, also align the two embeddings.
To address this issuc, we propose simple but surprisingly effective idea-we use both CLIP's vision encoder and its text encoder in our multi-modality aligument module. By directly incorporating CLIP's text encoder into its visual encodings, Since CLIP's vision and text encoders were originally jointly trained, multi-modal adaptation is inherently embedded in their encodings, making its text encoder an ideal choice for encoding instructions. The strong, inherent alignment between the two modalitios allows for scamless integration, minimizing the need for complex cross-modal modules or extensive training to achieve aligmment. Further- more, CLIP has demonstrated strong performance across diverse tasks, making reliable foundation for multi-modal applications.
We refer to our proposed architecture EMMA- Efficient Multi-Modal Adaptation. Figure illustrates EMMA's architecture. The left side highlights the high-level structure, where the standard modules of a multi-modal LLM are shaded in gray, while EMMA's newly introduced Modality Adaptation module is shown in green. On the right, the details of the Modality Adaptation module are depicted. More specifically, let v(-) and t(-) represent the vision encoder and text encoder of CLIP, respectively. The visual encodings are defined as v == v(x) E gaxd where is the input image, and the text encodings as = t(i) E Rxxdd where i is the instruction. Later, the instruction encodings are then processed through the Instruction Projection module p gd + d which maps the instruction tokens to the same dimensional space the visual tokens. Once the visual and textual representations are generated, we introduce carly fusion via a lightweight module called the Visual Alignment module. This component, consisting of a simple linear layer, combines visual and textual tokens to create the model's multi-modal encoding. The Visual Alignment module, which consists of linear layer, forms the core of our proposed method. Let f R(++m)xd + Raxd represent the Visual Aligument module, where and m denote the number of visual and textual tokens, respectively. The Visual A Aiggmment module takes the concatenation of the visual and textual tokens as input to generate refined
5

---

Published in Transactions on Machine Learning Research (05/2025)
POPE
10
LLaVA-1.5
MMB-en
86
SQA
EMMA-1.0
66.13
71.0
8
Frequency
66
6
MMB-ch 60.2
58.3
24.7
62-0
02.70
GQA
4
23.46
34.0 MMVP
29.85
0.006
0.008
0.010
0.012
0.014
62.97
MuirB
Mutual Information between the Visual Tokens and Response Tokens
EMMA-1.0
63.18
LLaVA-1.5
A-Bench
(b)
Figure 3: Left: Figure 3a illustrates the mutual information between visual representations and response codings. EMMA's visual representations exhibit 1.5x higher mutual information with the response encodings compared to LLa'A's, highlighting its superior alignment with the instruction. Right: Figure 3b isolates the impact of FMMA's ' modality adaptation module by comparing it to the baseline model, LLa VA-1.5, while keeping all other parameters constant except for the architectural innovation.
visual tokens V. The complete pipeline can be expressed as follows:
v = f(v(x), p(t(1))
(1)
Note that the dimensions of this aligmment layer are designed to maintain the same number of visual tokens as the baseline model, ensuring consistency in the number of visual tokens passed to the language model. The Visual Alignment module plays a critical role in ensuring effective alignment between visual and instructions encodings, highlighting the most relevant tokens from the visual encodings in response the instruction, and thereby delivering more precise visual information to the language model. Its lightweight design facilitates (1) easier interpretability and analysis, and more importantly, (2) its simplicity mitigates overfitting when training small datasets and (3) reduces the training and inference time while outperforming the state-of- the-art models with 10x fewer parameters. Moreover, by leveraging the inherent alignment of CLIP's vision and text encoders, both encoders remain frozen during training. This prevents overfitting to the training data and preserves the generalization capabilities of the original encoders, enabling EMMA deliver robust performance across diverse tasks.
3.2 Analysis on Modality Adaptation by EMMA
In this section, we analyze the impact of the modality adaptation module introduced by EMMA.
The Mechanics of the Visual Alignment Module. The Visual Alignment module takes the concatena- tion of the visual and textual tokens (n+m tokens) as input to generate the n refined visual tokens. We begin our examination by scrutinizing the matrix W E (n++)) associated with it. By analyzing the norms of the weights corresponding to each token, The histogram of & norm of visual and textual tokens are demonstrated in Figure 2 As the weights of ual lokens are below and the weights of visual tokens are above 1, the & norm of the weights serve as a more indicative measure token importance, As expected, the visual tokens exhibit higher weights within the Visual Alignment module, signifying their greater influence on the generated multi-modal representa- tion. Another key observation that certain textual tokens exert more influence than others. To highlight the relative importance of each textual token, Figure 2b presents bar chart. illustrating their respective
identify which tokens, visual or textual, are most impactful.
6

---

Published in Transactions on Machine Learning Research (05/2025)
Is the butterfly's abdomen visible in the image?
Is the minion in the image smiling with its tongue out?
LLaVA-1.55
60
EMMA-1.0
50
Frequency
40
(a) Yes
(b) No
(b) No
(a) Yes
20
LLVVA-1.5
(a) (a) X
(a) (a)
X
10
EMMA-1.0
(a) (b) L
(b) (a)
10
20
30
50
60
70
90
L2 norm between MMVP Image pairs
(a) Instances of MMVP Benchmark
(b) & distance between the MMVP image pairs.
Figure 4: Is EMMA capable of discerning images by focusing on the aspect specified in the instruction? To address this, we utilize the MMVP benchmark, which contains closely resembling images, as shown in this figure. By leveraging EMMA's visual representations of the image pairs and calculating the lg norm, we observe increase in the distance between them, illustrated in Figure 4b] indicating EMMA's ability to diffcrentiate between similar images.
weights. CLIP's text encoder generates 77 textual tokens, shown along the x-axis, As depicted in the figure, the carlier tokens tend to have a more significant impact. This finding indicates that the alignment module effectively ident ifies the most informative textual tokens, as instructions lypically consist of brief prompts, with the crucial information concentrated in the early tokens and the remaining tokens often masked.
Mutual Information between Aligned Visual Tokens and Response Tokens. Another primary ob- jective of the Modality Adaptation module is to adapt the visual representations to the language modality, ensuring they are well-aligned with the language model and encapsulate the necessary information to accu- rately respond to the given prompt. To evaluate the contribution visual tokens to the language model's output, we use the Mutual Information, which quantifies the amount of information obtained about one random variable through the other. In this analysis, the L.LVV--n-Wild Liu et al. (2024b) benchmark is employed, which has set of 24 images with 60 challenging questions in novel domains. For cach of the 60 samples, visual representations are generaled using the visual modules of both LLaVA and EMMA. It is important to note that EMMA's visual module processes the image and prompt to produce instruction-awann reprosentations, whereas LLaVA generates instruction-agnostic encodings. Additionally, for cach sample, the corresponding answer is encoded using CLIP's text encoder. The Mutual Information between the visual and response encodings then calculated, with the results shown in Figure 3a As illustraaed,, the mean mutual information for EMMA is 1.5 times higher than that of LLaVA, underscoring the effectiveness of EMMA's Visual Alignment in steering the model toward accurate language responses.
Isolated Impact of the Modality Adaptation Module. To evaluate the improvements introduced by EMMA's modality adaptation, we follow the same training scheme as the bascline model (Liu et al. 2024a). Therefore the improvements demonstrated in Figure 3b are achieved solely by our proposed method,
EMMA. Across multiple benchmarks compared to the bascline model, LLaVA-1.5. EMMA consistently sur-
passes VL-155 in various cross-modal (asks, demonstrating significant improvements in MMVP (Tong 2024) (+9.3%), MuirBench (Wang et al. 2024a) (+5.62%). SQA (Lu et al. 2022b (+4.2%), MM- [Liu et al. 2023a (+1.9%), and POPE (Li et al. 2023e (-1.17%). These results underscore EMMA's
al.
Bench
effectiveness in enhancing cross-modal tasks that require both visual and textual comprehension.
Aligned vs. Vanilla Visual Tokens The primary objective of EMMA's modality adaptation is to align visual representations with the instructions, ensuring they emphasize the aspects of the image directed by the instructions. Our method achieves this by incorporating instruction encodings into the refinement process of the visual representations. In this section, explore the alignment capabilities of EMMA by examining
7

---

Published in Transactions on Machine Learning Research (05/2025)
Method
LLM
VB
#Params
PT
IT
VQA
VisWiz
SQA!
GQA
ORVQA
BLIP-2
Vicuna-13B
ViT-g/14
13B
129M
65.0
19.6
61
45.9
BLIP
Vicuna-7B
VIT-g/14
7.91
130.2M
34.5
60.5
49.2
BLII
Vieuna- 13B
ViT-g/14
14.2B
130.2M
33.4
63.1
49.5
Shikra
Vicuna-
VIT-L/14
13b
6.1M
77.4
47.2
IDEFICS
LLaMA-7B
VIT-H/14
9B
354M
50.9
35.5
38.4
IDEFICS
LLMMA-65B
VIT-H/14
80B
354M
60.0
36.0
45.2
Qwen-VI.-Che VL-Chat
Qwen-7B 7B
VIT-G/14
9.6B
1.4B
78.2
38.9
68.2
57
56.6
Vicuna-7B
VIT-L/14
7B
1.2M
78.5
50.0
66.8
62.0
QA-VIT
Vicuna-7B
VIT-L/14
1.2M
SU.5
36.5
mPLUG-OWL
LLaMA-7B
VIT-L/14
8.2B
348M
79.4
54.5
68.7
56.1
57.7
BRAVE
FlanT5-XL
100M
82.5
52.7
66.0
EMMA
Vicuna-7B
VIT-L/14
7B
1.8M
89.42
56.03
73.14
56.01
68.57
Table 2: Comparison between EMMA and previous methods academic task-oriented datasets. EMMA achieves state-offthe-art performance on 4/5 of the benchmarks while using the fewest number of parameters
and training data compared to the previous approaches, utilizing modality adaptation(PTT and IT stand
for Pretraining and Instruction fine-tuning, respectively). The symbol denoles the collection of vision
encoders utilized by BRAVE, including EVA-CLIP-g Fang et al. (023) ViT-L/14 Radford et al. (0021 SILCC-/16 Nacem al. (2025). Vhen et al. (2022). and DINOv2-L/14 Öquab et al. (2023).
the visual representations before and after aligument. To perform this analysis, we utilize the MMVP Tong
et al. (024)) benchmark, which is designed to expose the visual shortcomings of MLLMs. We focus images with visually similar encodings but subtle differences, shown in Figure 4a EMMA demonstrates 9.3% improvement on this benchmark, underscoring its ability to generate more distinguishable visual rep- resentations for such images. To empirically validate this, we compute the & norm of visual representations between randomly selected pairs of MMVP images, comparing the results for pre-aligned and post- aligned reprosentations. The histogram of the norms, shown in Figure 4b] reveals clear shift, indicating that the aligned representations are better at distinguishing between these images by focusing on instruction-relevant tokens.
4 Experimental Evaluation
In this section, we begin by comparing EMMA with state-of-the-at multi-modal LLMs, using the benchmarks introduced earlier. Following this, we conduct robustness analysis with focus hallucination. We conduct an ablation study to identify the optimal layer output from the text encoder for use textual reprosentations.
Implementation Details. For training, we follow the same two-slage instruction fine-tuning process as VLaVA. In the pretraining stage, only the Visual Alignment and Projection modules are trained, while the language model remains frozen. During the fine-tuning stage, the LLM is unfrozen and finc-tuned along with the two aforementioned modules. We employ CLIP-ViT-L-14, trained on 336² pixel images, the base image encoder and text encoder. The Visual Alignment module is initialized with the identity matrix for the visual tokens and all zeros for the instruction tokens to transfer all the visual tokens at the beginning of training. Morcover, the Visual Alignment module is designed to maintain the same number of visual tokens as the baseline model. The latest Vicuna v1.5 (Zheng al. 2023) used as the base L.M. FMMA uses the same set of hyperparamcters as the LLa For all the analysis performed in the Scction 3 we use the same dataset the baseline model, which is 558K and 665K samples for the pretraining and fine-tuning stages, respectively. In the Evaluation setting, we have preserved the same pertaining data but scaled the fine-tuning data to 1.2M samples, including LVVSSInstrructe (aorurttt//Waangggggg/gg et al. 2023a), CLEVRRJJhnnon et al. 2017), MtaWizGGurari et al. 2018), and Science (L et al. 2022a) training data. EMMA'S 's training data is the most elficient compared to all state-of-the-art methods (except for LLa VA), as shown in Table 3
Benchmarks. We evaluate FMMA across 10 benchmarks that span a diverse set tasks, including scientific question answering, visual question answering with poor image quality, integrated perception and reasoning tasks, visual dialogue, and general reasoning. The results of recent benchmarks designed specifically for
8

---

Published in Transactions on Machine Learning Research (05/2025)
Melhod
LLM
VE
#Params
PT + IT
MMBEN
MMB
MMMU
MathVisla
Muirbench
BLIP-2
Vicuma- 13B
ViT-g/l 14
14.2B
129M
InSI
Vicuna-7B
ViT-g/14
7.91B
130.2M
36
23.7
InstructBLIP
Vicmm-133 13B
VIT-g/14
14.2B
130.2M
25.3
Shikra
Vicuna-188 13B
VIT-L/14
13B
6.1M
58.8
IDEFICS
LLaMA-7B
VIT-H/14
9B
354M
48.2
25.2
19.8
IDFFCC
LLaMA-65B
VYT-H/14
S0B
354M
54.5
38.1
Qwen- VL-Chat
Qwen-7B
VIT-G/14
1.4B
60.6
56.7
35.9
LLaVA-1.5
Vicuna-7B
VIT-L/114
7B
1.2M
64.3
58.3
35.11
21.1
23.46
mPLUG-Ow12
LLMMA-TB
VIT-L/14
8.2B
348M
64.5
32.7
22.2
BRAVE
F5
10B
100M
EMMA
Vicuna-7B
VIT-L/14
1.8M
66.44
60.15
35.44
25.1
32.0
Table 3: This table compares EMMA with previous methods on specialized MLLM benchmarks. EMMA delivers the best performance on 3/5 benchmarks and secures second place on the remaining two with less than 0.5% difference. The symbol denotes the collection of vision encoders utilized by BRAVE, including
EVA-CLIP Fang al. 2023)) VIT- / Radford al. (2021), SILC-G/16 Naeem et al. (025). ViT-- Chen
ct al. 2022h, and DINOv2-L/ 14 Oquab et al. (2023).
instruction-folloingg large multi-modal models (LMMs) are presented in Table 3 while Table 2 outlines the performance on academie task-oriented benchmarks. EMMA emerges as the stale-of-the-arl model for 4 out of 5 academic task-oriented benchmarks, outperforming mPLUG-OwL which has 50x larger modality adaptation module and is trained on 300x more data, and BRAVE which has 24x larger vision encoder and is trained on 100x more data. On MLLM specialized benchmarks, EMMA delivers the best performance on 3 out of 5 benchmarks and is second best on the two others with less than 0.5% difference. In conclusion, out of 10 benchmarks, despite utilizing the simplest architecture,
EMMA achieves the best performance outperforming other MLLMs that rely on complex modality adaptation modules.
Robustness & Hallucination. Robustness and the ability to avoid hallucination are essential for multi- modal large langnage models (MLLMs), which are increasingly applied in areas like medical diagnostics to interpret complex text and image data. Hallucination, a significant security threat to MLLMs, occurs when the model generates information that does not accurately represent the provided images or text. As result, evaluating and mitigating hallucinations is a critical step in ensuring the reliability of MLLMS before real-world deployment, making it key focus in model performance asscssments. The hallucination evaluations in this section are performed utilizing two benchmarks, AMBER and FOIL, which do not rely on additional LLMs, thereby providing a direct and controlled means of assessing the model's ability to avoid hallucinations. These benchmarks (Wang et al.) 2023b Shekhar et al., 2017 Nie et al.] 2024 Wu et al. 2024) focus on specific challenges in multi-modal reasoning, lesting the model's accuracy in aligning textual and visual content without introducing erroneous information. AMBER consists of 7628, 4924, and 1663 samples for the Attribute, Relation, and Existence categories, respectively. FOIL contains total of 99,480 test samples, of which 92,705 are straightforward, allowing both LLaVA and our method to successfully avoid hallucinations. However, 6,775 samples present more challenging cases. We compare LLaVA-1.5 the bascline for our approach, with EMMA. The results, shown in Figure 5a indicate that EMMA outperforms the baseline, with significant performance gaps in two of the four benchmarks.
Ablations Text Encoder. We ablate the text feature's abstraction level in this section. The textual features can be optionally derived from either the final layer the penultimate layer of the CLIP Text Encoder. A comparison between the two methods of extracting textual fcatures is presented in Figure 5b demonstrating the clear superiority of features derived from the penultimale layer. We hypothesize that the final layer of CLIP captures more global and abstract semantics of the instruction, whereas the penultimate layer focuses on finer details. Additionally, since visual features are extracted from the corresponding layer in the visual encoder, using the same layer in the text encoder ensures both modalities operate at similar level of abstraction, promoting better alignment between them.
9

---

Published in Transactions on Machine Learning Research (05/2025)
91.4
LLaVA-1.5
POPE
EMMA-1.0
79.950.8
56.6
MMB-en
67.568.1
67.7
66.1
71.0
Accuracy
60
59.7
66.19 Sarri
69.9)
50.1
40
37.137.3
MMB-ch 60.2
59.3
6278
GQA
20.2
AMBER Attr
AMBER Rel
AMBER Exis
R-Bench IMG-LVL
R-Bench INS-LVL
MMRei DallE
MMRei VG
FOIL
63:48
MMVP
MuirB
Penulitmate Layer
Last Layer
A-Bench
(b)
Figure 5: Left: This plot compares EMMA's robustness against hallucinations with LLaVA, demonstrating consistent improvements over the baseline in avoiding hallucinations. Right: This radar chart compares EMMA's performance when leveraging textual features generated by either the penultimate or last layer of the text encoder, highlighting the advantage of features associated with the penultimate layer.
Ablation on the Visual Alignment Module. To evaluate the effectiveness of our proposed visual align- ment module, implemented as linear layer, we conduct a comparative analysis by replacing the linear layer with a cross-attention mechanism between the two modalities. Specifically, we compare the performance of three MLLLMs, all sharing the same architecture and training setup but differing in their modality adaptation modules:
VA
baseline model without any modality adaptation,
EMMA (Linear) utilizing a linear layer for modality adaptation,
EMMA AttesttA)))ntin)) using cross-attention for modality adaptation.
The results in Table show that EMMA with the linear adaptation module consistently achieves the best performance across most benchmarks. Moreover, EMMA with cross-attention shows improvements over the baseline on SQA and MuirB, highlighting the potential its adaptation mechanism. Nevertheless, its more complex architecture introduces risks of overfitting, as observed in benchmarks such as GCA, MMB, MMVP, and A-Bench. We posit that effective modality adaptation, i.e., refining visual representations through instruction- based guidance, requires carefully balanced interplay between architectural complexity and training set size. This balance becomes particularly crucial when operating under constraints such baseline setup with only 1.2 million training samples.
Method
Data
SQA
GQA
POPE
MMB-en
MMB-cn
MMVP
A-Bench
MuirB
LLaVA-115
1.2M
66.80
62.00
86.10
64.30
58.30
24.70
62.97
23.46
EMMA (Linear)
1.2M
71.00
62.78
87.27
66.19
60.20
34.00
63.48
29.08
EMMA (Cross-Attenlion)
1.2M
67.08
49.33
79.73
60.84
51.42
8.67
56.01
29.50
Table 4: A comparative analysis of EMMA's Visual Alignment module reveals that the linear adaptation layer delivers the most consistent performance gains. In contrast, the cross-attention variant demonstrates selective improvements but is prone to potential overfitting in certain scenarios.
10

---

Published in Transactions on Machine Learning Research (05/2025)
5 Conclusion
In this work, we addressed the inefficiencies in the modality adaptation modules employed by current multi- modal large language models. We hypothesized that the initial alignment between visual and textual en- codings plays critical role in determining the complexity level of the modality adaptation module and the necessary amount of training data. Our lightweight approach, EMMA (Enhanced Multi-Modal Adaptation), leverages CLIP's text encoder to generate instruction encodings and, by exploiting this initial aligmment, demonstrates that the modality adaptation module can be simple while still enhancing the alignment be- tween visual and textual modalitics. Through extensive analysis, we demonstrated that EMMA effectively produces instruction-aware visual representations aligned with the language model. Our experiments, eval- uated across multiple benchmarks, show that EMMA significantly outperforms state-of-the-art models that use modality adaptation modules 50x larger. Finally, our robustness analysis, particularly in hallucination avoidance, confirmed EMMA's 's superior ability to accurately process multi-modal data, even in challenging scenarios.
Acknowledgments
This paper is supported in part by the Army Research Office under grant number W911NF-21-1- 0155 and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010. Additional support was provided by the NYU IT High Performance Computing resources, services, and staff expertise.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl Xrx::2200 08774, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Mioch, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. NeurIPS, 35:23716-2366 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vasion-language model with versatile abilities. arXiv preprint 2rXiv:2308. 12966, 2023.
Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305. 13292, 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal Ilm's referential dialogue magic. arXiv preprinl arXiv:2306.1511 2023b.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-inage model. In ICLR, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Jungi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purport vision-language models with instruction tuning. NewrIPS, 36, 2024.
Danny Dricss, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint ar 23x7:22303.03378 2023a.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint XrXi::2003.032. 033378, 2023b.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation Icarning at scale. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 19358 19369, 2023.
Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Farshad Khorrami, and Siddharth Garg. Lipsim: A provably robust perceptual similarity metric. arXiv preprint arXim:2310.1827 2023a.
Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, and Alexandre Araujo. R-lpips: An adversarially robust perceptual similarity metric. arXiv preprint arXiv:2307.11157 2023b.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904 6913, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pp. 3608-3617 2018.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Newral Information Processing Systems, 36:72096-72109, 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988- 1997, 2017.
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. arXiv preprint arXiv:2404. 07204, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint Xrxim:208..0 000900 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXie preprint ar arXiv:2305.03720 033220 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730 19742. PMLR, 2023c.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2005.0635, 2023d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint urXiv:2305.10355 2023c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, pp. 292 305, 2023f.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large modal models. arXiv preprint ar Xrx::22311.0000 1006007, 2023g.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint or arXiv:2401.155 15947, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 26296 26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in newral information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprinl arXiv:2307. 06281, 2023a.
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centric tasks by interacting with 2023b.
chatgpt beyond language. arXiv preprint ar arXi::2305.050
Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, and Wenhai Wang. Controlllm: Augment language models with tools by searching on graphs. arXiv preprint arxi:22310.1 17796, 2023c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 35:2507- 15:2507-2521, 2022a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Chun Zhu, Oyvind Tafford, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Newral Information Processing Systems (NeurIPS), 2022b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint crXim:2310.02255 2023.
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Lue Van Gool, and Federico Tombari. Sile: Improving vision language pretraining with self-distillation. In European Conference on Computer Vision, pp. 38-55. Springer, 2025.
Jiahao Nie, Gongjie Zhang, Wenbin An, Yap-Peng Tan, Alex C Kot, and Shijian Lu. Mmrel: A relation understanding dataset and benchmark in the milm era. ar preprint arXiv:2406.088 091121 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. TMLR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, pp. 8748 8763, 2021.
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélic Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. Foil it! find one mismatch between image and language caption. arxio preprint arXiv:1705.0133 2017.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. In ICLR, 2024.
Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-inten inniinterleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. Xix preprint urXiv24011.00205 2024.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal lms.. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 9568 9578, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint Xrxvv:2302. 1. 13971, 2023.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image un- derstanding. arXiv preprint arxiv:2406. 09411, 2024a.
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpl-4v for better visual instruction tuning. orXiv preprinl ar Xix:2311.00511 2023a.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An lim-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.07399 2023b.
Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-sceing project v2: Towards general relation comprehension of the open world. ariiv preprint or Xiv:2402. 19474, 2024b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvide02: Scaling video foundation models for multimodal video understanding. arXiv preprint 2tv: 15377, 2024c.
Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in Ivlms. arXiv e-prints, pp. arXiv-2406, 2024.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. arXiv preprint ar Xic:2300 05519, 2023.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-docowl: Modularized multimodal large language model for document understanding. CoRR, abs/ 0ass/230..02245 02499, 2023a. doi: 10.48550/ ARXIV.237. 02499. URL hattps://dol org/ 10. 48550/arXiv. 2307 02499
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multi- modality. arXiv preprint arxiv: 23t::2304.1411 141788 2023b.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint ar Xrv: 042257, 2023c.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556 9366-956, 2024.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. Tn ICLR, 2024.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-offinterst. Xiv preprint vxii:2307.03601 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36:46595 46623, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. In ICLR, 2024.
15


Reviews:
[{"review_id": "8nvBNzFA4u", "reviewer": "Reviewer_P8Hw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The problem under study is intriguing and has consistently held significance in the field of vision-language models.\n2. This paper presents a systematic approach to designing the alignment module, which is notably lightweight compared to other methods, such as mPLUG2-Owl.\n3. In addition to conducting extensive experiments to demonstrate the effectiveness of the proposed module, the authors have also performed an in-depth analysis of the mechanisms driving EMMA's success.\n4. The paper is well-written and easy to read.\n\n## Weaknesses\n1. In this paper, the authors consistently emphasize the importance of achieving better alignment between visual and textual features. However, from my perspective, the key to EMMA's success lies in its ability to extract instruction-aware visual features. This ensures that the features fed into the LLM are more closely related to the instructions, allowing the LLM to understand them more easily.\n2. The implementation of EMMA is somewhat counterintuitive. Given that there is no direct interaction between the visual and textual features through a linear projection layer—since its role is merely to merge these features—it's puzzling why the authors did not opt for other approaches, such as cross-attention, to implement EMMA.\n3. A question arises regarding alignment. In the literature, better alignment typically refers to the alignment between the space of the vision encoder and that of the LLM. However, EMMA only incorporates text features from CLIP's text encoder into the vision features, and the space of CLIP's text encoder is not aligned with that of the LLM. So how better alignment is achieved?\n4. In Section 3.2, the authors assert that \"instructions typically consist of brief prompts, with the crucial information concentrated in the early tokens.\" However, in reality, this is not always the case. Given that CLIP's text encoder can only process 77 tokens, how do the authors propose to address the issue of longer instructions, such as those that include few-shot examples or chain-of-thought (COT) reasoning?\n5. I noticed that the authors only compare EMMA to methods that emerged about a year ago, and I encourage them to include comparisons with more recent works. Additionally, the training settings, such as the datasets used, are outdated. It is highly recommended that the authors update their training settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an approach to aligning visual and textual features. The authors suggest extracting instruction-aware visual features by integrating vision and text features before inputting the visual data into the LLM. Compared to previous methods, such as LLaVA, the proposed EMMA demonstrates superior vision-text alignment and achieves better results on benchmarks. Additionally, EMMA is more lightweight compared to other models like mPLUG-Owl2.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "beagN36WNg", "reviewer": "Reviewer_5AWv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Advantages:\n\nThe Efficient Multi-Modal Adaptation (EMMA) architecture is highly intuitive. By leveraging CLIP’s text encoder to collaborate with its visual encoder, EMMA naturally generates text-guided visual features. Such features allow the visual modality to better highlight task-relevant regions when injected into the large language model (LLM).  \nThis intuitive design leads to strong empirical performance, achieving significant improvements compared to standard baselines such as Llava 1.5.\n\nDisadvantages:\n\nHowever, the experimental evaluation presented in this paper is relatively simplistic. The chosen baseline, Llava 1.5, is somewhat outdated. Additionally, several critical design decisions of EMMA—such as the necessity of employing CLIP’s text encoder, the choice between cross-attention and self-attention mechanisms, or the potential benefits of additional optimization strategies and pretraining stages—lack comprehensive comparative experiments and explorations to clearly justify these design choices.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper designs an **Efficient Multi-Modal Adaptation (EMMA)** architecture, which fuses the CLIP text encoder and visual encoder early on to achieve a cross-modal representation before injecting it into Llama.  \nWe conduct extensive experiments to demonstrate the effectiveness of the EMMA structure, thoroughly analyze its various phenomena, and evaluate it on standard benchmarks, achieving significant performance improvements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would have appreciated seeing more detailed ablation studies and further elaboration on implementation details.", "broader_impact_concerns": "I do not have any broader impact concerns regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "btIGwispJy", "reviewer": "Reviewer_fhZ6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**strengths**\n\n**1. Leveraging CLIP’s Pretrained Semantic Alignment.**\nEMMA effectively leverages CLIP’s pretrained vision and text encoders, utilizing their inherent semantic alignment. The early fusion of visual and textual embeddings enhances semantic coherence efficiently, significantly reducing computational overhead.\n\n**2. Comprehensive Interpretability Analysis.**\nThe paper provides insightful analyses (mutual information, weight norms), clearly showing distinct contributions from visual and textual embeddings. It demonstrates that textual instructions serve as subtle guides for visual tokens, supporting EMMA’s design rationale effectively.\n\n**3. Lightweight and Efficient Alignment Module.**\nEMMA introduces an exceptionally lightweight alignment module (~0.2% additional parameters), significantly lighter than methods like mPLUG-Owl2. Its simple linear aggregation notably reduces training complexity and inference costs, enhancing practical applicability.\n\n**4. Strong Empirical Performance.**\nEMMA achieves state-of-the-art performance across multiple benchmarks (e.g., VQA, ScienceQA, VizWiz), with improved robustness against hallucinations. However, I have concerns about its reported superiority, as critical experimental details (dataset composition, fine-tuning settings) lack sufficient clarity, raising fairness and reproducibility questions.\n\n**weaknesses**\n\n**1. Inherent Limitation to Single-Round Tasks**\nEMMA’s lightweight alignment module is inherently limited to single-turn tasks, restricting its effectiveness in multi-turn conversational scenarios. This architectural limitation could inadvertently lead to shortcut learning or overfitting specifically to single-turn question-answer benchmarks. Since real-world multimodal applications frequently require multi-turn reasoning, context maintenance, and dialogue management, this significantly constrains EMMA’s broader applicability and impact in realistic, interactive settings.\n\n**2. Unclear Explanation and Ambiguous Technical Details**\nTechnical details, particularly about the linear adaptation layer (Section 3.1), remain unclear. Crucial implementation specifics—such as handling variable-length tokens, padding, masking, or aggregation—are inadequately explained, complicating reproducibility and rigorous evaluation.\n\n**3. Problematic Experimental Setting and Comparison Fairness**\nEMMA nearly doubles its fine-tuning dataset (665K → 1.2M samples) compared to its baseline (LLaVA-1.5) but directly compares results without clearly highlighting this difference. Performance gains may thus reflect increased data rather than architectural advantages, and unclear dataset details further reduce empirical credibility and reproducibility.\n\n**4. Limited Innovation Compared to Prior Work**\n Its core idea—integrating textual instructions into visual representations for alignment—bears strong resemblance to prior work such as _Question Aware Vision Transformer for Multimodal Reasoning_. Compared to that, EMMA offers relatively incremental innovation, primarily in terms of implementation simplification (e.g., replacing attention with a linear fusion layer) and utilizing the inherent alignment in CLIP model's visual and text modalities.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "EMMA proposes a lightweight visual-text alignment module that efficiently integrates CLIP’s pretrained vision and text encoders, which are inherently well aligned, significantly reducing computational overhead while maintaining strong cross-modal alignment. Through extensive interpretability analyses, EMMA demonstrates that textual embeddings effectively guide visual tokens, enhancing alignment quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Clarify and Elaborate on the Visual Alignment Module**\n* Please provide additional details on the Visual Alignment module, particularly how it handles variable-length visual and textual inputs. Including information on padding, masking, or aggregation strategies—as well as implementation-level details such as dimensions or initialization—would improve clarity and support reproducibility. A diagram could also be helpful for readers to better understand the architecture.\n* Moreover, some technical details require clarification. For instance, in Section 3.1, the paper introduces an “Instruction Projection” module  p: \\mathbb{R}^{d{\\prime}} \\rightarrow \\mathbb{R}^{d}  to align the instruction embeddings with the visual token space. However, since both modalities are encoded using CLIP’s vision and text encoders—which are pretrained to produce outputs in the same dimensional space—this projection appears unnecessary. If  d = d{\\prime}  by default in CLIP, the inclusion of this module is confusing and should be better justified or clarified.\n\n2. **Provide More Information on Fine-Tuning Data Composition**\nSince the paper reports strong performance gains, it would be helpful to include more information about the fine-tuning dataset, particularly how the 1.2M samples were constructed and what mixture of data sources were used. This would offer better context for understanding the results and allow others to reproduce the training setup more faithfully.\n\n3. **Address Limitations for Multi-Turn Tasks**\nEMMA appears to be optimized for single-turn tasks, and its applicability to multi-turn or conversational scenarios is unclear. Please discuss this limitation explicitly and consider outlining potential strategies or extensions that could support context-aware or multi-round reasoning.", "broader_impact_concerns": "N/A — no additional concerns beyond what is discussed in the Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SKCeEbLWHo", "reviewer": "Reviewer_tZkv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1.\tThe structure of EMMA is based on CLIP's joint training of text and vision encoders. The paper must address whether the approach generalizes across other vision-language architectures (e.g., DINOv2, PaLI) or performance is extremely CLIP-specific alignment-dependent.\n2.\tExperiments are done only with 7B-parameter LLMs. The paper does not state whether EMMA's efficiency holds for larger models (e.g., 70B+ LLMs) or whether performance gains degrade with size, limiting its generality.\n3.\tThough EMMA has less training data, comparison to models like mPLUG-Owl2 (trained from 300× more data) may not fully represent the impact of architectural efficiency. The paper should report whether baselines were re-trained under similar data constraints or whether findings are due to the intrinsic virtues of EMMA's architecture.\n4.\tCloser examination, e.g., attention pattern visualization or ablation on token importance, would lend stronger support to claims on alignment dynamics.\n\nRemark\n\nThe VE and parameters column in Table 2 and Table 3 seems to be inaccurate for EMMA.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces EMMA, a light module to enhance visual-textual alignment in multi-modal large language models. Current approaches prefer to use advanced modality adaptation modules, resulting in growing model size and large training data demands. EMMA addresses this by leveraging CLIP's pre-aligned vision and text encoders with a sparse linear layer for early fusion. This strategy maintains minimal parameter growth while boosting task performance. Comparison results show that EMMA outperforms state-of-the-art models on general and specialized tasks by up to 9.3%.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check the weakness part.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline submission with two weak accepts and two weak rejects. After carefully inspecting the paper, reviews and author feedback. AE recommends acceptance with minor revision because the benefits of the proposed approach outweigh the concerns pointed out by the reviewers in AE's view. In particular, the proposed approach is simple, intuitive, and demonstrates strong empirical performance on important and popular benchmarks. Regarding the minor revision, the AE recommends the authors to revisit the comments of reviewer P8Hw and incorporate parts of their responses into the method or even introduction section of the manuscript (e.g. aligning vision-text features vs making vision features instruction-aware, linear projection vs attention for combining CLIP vision/text features). Many of them are valid points that can benefit the readers. If possible (optional), adding some experiments (qualitative or quantitative) on how the proposed approach works with long-form instructions (e.g. few-shot, CoT) can enhance the value of this paper. Last but not least, given the strong performance of the proposed approach, it'd be very helpful for the community if authors can release the source code to aid other researchers reproduce the results reported in this paper.
Decision: Accept (poster)