EMMA: Efficient Visual Alignment in Multi-Modal LLMs

## Abstract

Abstract Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress
in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs.
In this paper, we propose EMMA ( E fficient M ulti- M odal A daptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.
Our code is available at https://github.com/SaraGhazanfari/EMMA

## 1 Introduction

Figure: Figure 1: This figure presents the EMMA’s Architecture where Modality Adaptation is introduced to the standard MLLM architecture. The Modality Adaptation module consists of the Text Encoder, Instruction Projection, and Visual alignment modules. EMMA enhances the multi-modality alignment by incorporating the instruction encodings generated by the Text Encoder which are then projected to the joint space by the Instruction Projection module. The concatenating visual and textual encodings are then passed through the Visual Alignment module, resulting in multi-modal, instruction-aware representations.
Refer to caption: x1.png

Over the past years, Large Language Models (LLMs) have transformed natural language processing (NLP) by demonstrating exceptional abilities in understanding, generating, and reasoning with text across a wide range of tasks; from machine translation and summarization to complex problem-solving and conversational agents (Touvron et al., ; Zheng et al., ). However, many real-world applications require the ability to process more than just text, such as understanding visual content or synthesizing information from different modalities (Ghazanfari et al., ; ). This has led to the development of multi-modal LLMs (^1^11In this paper, we refer use multi-modality specifically for combination of image and text modalities.), which combine the linguistic strengths of LLMs with vision foundation models, enabling cross-modal understanding and reasoning. By integrating textual and visual information, these models extend the capabilities of traditional LLMs to address tasks like image captioning, visual question answering, and text-to-image generation(Liu et al., ; Alayrac et al., ; Achiam et al., ).

Current state-of-the-art multi-modal models typically rely on fixed visual feature encodings extracted from vision foundation models, which are projected into the text space and passed to the language model along with instructions (Driess et al., ; Liu et al., ; ). However, the static nature of these encodings, formed without considering the instruction, limits the model’s ability to adapt dynamically to specific tasks or contexts. This disconnect between visual and textual components reduces flexibility, making the model less responsive to task-specific nuances.
To address this, BLIP-2 (Li et al., ) introduced a cross-attention-based module (called Q-former) to integrate the visual and instruction encodings, a design later adopted by others (Li et al., ; Huang et al., ; Ye et al., ).
The current state-of-art model, mPLUG-Owl2 (Ye et al., ), introduces a *modality adaptation module* that employs attention modules to embed the two modalities into a shared semantic space and thus enhances cross-modality collaboration. mPLUG-Owl2’s performance improvement comes with several limitations compared to baseline models. First, mPLUG-Owl2 leverages LLaMA’s text embeddings and CLIP’s vision encoder to generate the instruction and visual encodings respectively. Therefore the encodings are generated using two distinct models with no initial multi-modality alignment. Second, mPLUG-Owl2’s modality-adaptive module introduces roughly 1B more parameters, $3\times$ more than its vision encoder.
The modularity adaptation module is then trained from *scratch*, requiring 348 million image-text pairs for pertaining, $300\times$ more than the baseline.
Third, the vision encoder requires training during both the pretraining and instruction-tuning stages, which increases the overall training cost and makes the vision encoder more susceptible to loss of generality.
Finally, except for a few benchmarks, the model offers only marginal improvements, and in some cases, performs worse than the baseline.

The challenges outlined above led us to explore a more efficient method for modality adaptation.
We hypothesized that the need for a complex module for modality adaptation arises from the fact that visual and textual encodings are produced by two entirely separate modules, trained independently.
As a result, these complex modules attempt to integrate two distinct spaces, which is inherently difficult.
To address this issue, we introduce EMMA (Efficient Multi-Modal Adaptation), which performs modality fusion through a lightweight modality adaptation mechanism. EMMA integrates CLIP’s text encoder with its visual encoder and leverages the pre-trained alignment to adapt visual representations with the instruction via an efficient modality adaptation module (adding less than 0.03% parameters to the model). Our modality adaptation module generates instruction-aware visual representations by attending to more informative, instruction-related tokens, leading to improvements in MLLM-specialized and general benchmarks. A comprehensive set of experiments on benchmarks demonstrates that EMMA significantly enhances cross-modal alignment, improves performance across a range of vision-language tasks, and strengthens the robustness of MLLMs against hallucination.
Our contributions can be summarized as follows:

- •
Efficient Modality Adaptation: We introduce a lightweight modality adaptation mechanism that refines visual representations with less than a 0.2% increase in model size, maintaining high efficiency without compromising performance.
- •
Comprehensive Analysis of Visual Alignment: We conduct an in-depth investigation of the Visual Alignment module to provide (1) a detailed understanding of how visual and textual tokens are integrated and (2) an analysis of how effectively the aligned visual representations attend to instructions compared to the initial raw visual encodings.
- •
Extensive Empirical Evaluation: We perform a comprehensive evaluation on both general and MLLM-specialized benchmarks, demonstrating that EMMA significantly improves cross-modal alignment, boosts task performance, and enhances the robustness of multi-modal LLMs.
- •
EMMA Outperforms Larger Models: Compared to mPLUG-Owl2 which has $50\times$ larger modality adaptation module and is trained on $300\times$ more data, EMMA outperforms on 7 of 8 benchmarks. Additionally, compared with BRAVE which has $24\times$ larger vision encoder and is trained on $100\times$ more data EMMA outperforms on all benchmarks.

## 2 Related Work

#### Multi-modal Large Language Models (MLLMs).

In recent years, there has been significant progress in the development of multi-modal large language models (LLMs) that integrate vision and language to handle tasks requiring both modalities (Zhang et al., ; ; Wu et al., ; Sun et al., ; Alayrac et al., ; Lai et al., ; Li et al., ; ; ; Lin et al., ; Liu et al., ; ; Tian et al., ; Wang et al., ; ; Chen et al., ). By combining the language understanding of LLMs with the perceptual abilities of vision foundation models, multi-modal LLMs are able to tackle a wide array of tasks that require cross-modal alignment and understanding. We can divide these models into two general categories based on the way that the two visual and textual modalities are integrated. In the first category, including LLaVA (Liu et al., ; ), PaLM-E (Driess et al., ), Shikra (Chen et al., ), etc., the vision encodings are projected to the text space with a few linear layers, and then concatenated with the instruction tokens and passed to the LLMs. In the second category, a more complex module is used for cross-modality adaptation, where both visual and textual encodings are processed through the adaptation module. First introduced with Flamingo (Alayrac et al., ) and later adopted by BLIP-2 (Li et al., ), InstructBLIP (Dai et al., ), Qwen-VL (Bai et al., ), mPLUG-Owl Ye et al. (), and MiniGPT-4 (Zhu et al., ), the use of a Q-former (a cross-attention based module) has become a prominent technique.

**Table 1: Paremeter sizes of EMMA compared with state-of-art MLLMs across three components: Vision Encoder, Modality Adapter, and LLM. EMMA uses the same-sized Vision Encoder and LLM as mPLUG-Owl2 but its Modality Adapter, our main contribution, is 50$\times$ smaller.**
| Method | Vision<br>Encoder | Modality<br>Adapter | LLM | Total |
| --- | --- | --- | --- | --- |
| InstructBLIP | 1B $\times\text{3.3}$ | 0.2B$\times\text{10}$ | 7B | 7.91B |
| BLIP-2/InstructBLIP | 1B$\times\text{3.3}$ | 0.2B$\times\text{10}$ | 13B | 14.2B |
| Qwen-VL-Chat | 1.9B$\times\text{6.3}$ | 0.08B$\times\text{4}$ | 7.7B | 9.6B |
| BRAVE | 7.2B$\times\textbf{24}$ | 0.1B$\times\text{5}$ | 2.6B | 10B |
| mPLUG-Owl2 | 0.3B$\times\text{1}$ | 1B$\times\textbf{50}$ | 7B | 8.2B |
| EMMA | 0.3B | 0.02B | 7B | 7.3B |

#### Enhancing Visual Alignment in MLLMs.

Since the emergence of multi-modal LLMs, aligning visual and textual modalities has remained a significant challenge in achieving robust and seamless integration. Previous works (Alayrac et al., ; Li et al., ; Ye et al., ; ; Kar et al., ) have predominantly focused on utilizing Q-formers and similar cross-attention modules as modality adaptation components to integrate visual and textual embeddings. Recent advancements in modality adaptation include mPLUG-Owl2 (Ye et al., ) and BRAVE (Kar et al., ).
mPLUG-Owl2 (Ye et al., ) introduces 1B modality adaptation module that employs distinct parameters to project multiple modalities into a unified semantic space for enhanced modality adaptation. On the other hand, BRAVE (Kar et al., ) leverages a concatenation of various visual encodings, summing up to 7B, directly feeding into the Q-former. These modality adaptation modules rely on intricate architectures, introducing millions to billions of additional parameters to the model, which significantly increases the computational costs.
This added complexity not only demands vast amounts of training data but also imposes considerable overhead during inference.
Furthermore, the complication of these systems makes it difficult to discern the primary drivers behind performance improvements — whether they stem from the model complexity, the early fusion of vision and text encodings, or the sheer volume of additional training data. This leads us to the central focus of our work: (1) addressing the inefficiencies in current multi-modal models by proposing a more streamlined approach. Our goal is to achieve an efficient early fusion of visual and textual encodings without significantly increasing the number of parameters or computational costs.
(2) Conducting a thorough analysis to determine the key drivers behind performance improvements and the dynamics of the modality adaptation module to generate the multi-modal encodings.
This investigation will help isolate the most effective factors and guide future advancements in multi-modal LLMs.

#### Multi-modal LLM’s Benchmarks.

The evaluation of multi-modal LLMs relies on a mix of traditional academic benchmarks and newer ones tailored to instruction-following MLLMs. Established benchmarks like VQA-v2 (Goyal et al., ) and GQA Hudson & Manning () gauge a model’s ability to interpret visuals through open-ended, short-answer questions. ScienceQA (Lu et al., ) tests zero-shot generalization in scientific question answering, while
VizWiz (Gurari et al., ) offers real-world images captured by visually impaired users, challenging models with poor framing, blur, and other quality issues typical of non-professional photos. Additionally, newer benchmarks target instruction-following MLLMs.
MathVista (Lu et al., ) introduces diverse challenges from mathematical and visual tasks.
MMMU (Yue et al., ) evaluates multi-modal models on a broad range of college-level tasks requiring deep subject knowledge and reasoning.
To assess multi-image reasoning, MUIRBENCH (Wang et al., ) provides a comprehensive benchmark with 12 diverse tasks to evaluate the multi-image understanding abilities of MLLMs.
For general robustness, MMBench (Liu et al., ) offers a multiple-choice visual question answering benchmark in both English and Chinese, suggesting shuffling the choices to test the model’s robustness to the order of options. MMVP (Tong et al., ) evaluates robustness by identifying similar images with minute differences and manually pinpointing the visual details the CLIP vision encoder overlooks, which leads to incorrect responses from MLLMs.
For hallucination, POPE (Li et al., ) examines the extent of hallucinations across three COCO subsets. AMBER (Wang et al., ), a multi-dimensional benchmark, evaluates generative and discriminative tasks, including existence, attribute, and relation hallucination. FOIL (Shekhar et al., ) pairs images with both accurate and ”foil” captions—captions that are nearly identical but contain a single error—further challenging the model’s accuracy.

Figure: (a) Histogram of $\ell_{2}$ Norms
Refer to caption: x2.png

## 3 Efficient Visual Alignment in Multi-Modal LLMs

In this section, we explain our proposed method which addresses the inefficiencies in current multi-modal models by an efficient early fusion of visual and textual encodings without significantly increasing the number of parameters or computational costs. Moreover, a detailed interpretability analysis is offered to provide insights into the internal mechanisms of the proposed method.

### 3.1 EMMA: Efficient Multi-Modal Adaptation

Figure: Figure 3: Comparing EMMA with the state-of-the-art model LLaVA-1.5 demonstrates that EMMA surpasses the baseline across a range of benchmarks up to 9.3%.
Refer to caption: x4.png

Figure: Figure 4: Is EMMA capable of discerning images by focusing on the aspect specified in the instruction? To address this, we utilize the MMVP benchmark, which contains closely resembling images, as shown in this figure. By leveraging EMMA’s visual representations of the image pairs and calculating the $\ell_{2}$ norm, we observe an increase in the distance between them, as illustrated in Figure 5(b), indicating EMMA’s ability to differentiate between similar images.
Refer to caption: x5.png

Recent progress in multi-modal models is largely driven by the robust reasoning capabilities of large language models. Therefore, a persistent challenge remains in effectively aligning these two modalities to ensure seamless fusion and task-specific adaptability. Current approaches often depend on complex cross-modality modules, which introduce a significant number of parameters and
thus require large amounts of training data.
We hypothesize that the need for a complex modality adaptation module
arises from the fact that visual and textual encodings are produced by two independently trained independently that are themselves unaligned. This is precisely the case for mPLUG-Owl2, which utilizes CLIP as its vision encoder and LLaMA’s text embeddings for its text encoder.
As a result, the multi-modality module must, in addition to incorporating text information into the visual embedding, also first align the two embeddings.

To address this issue, we propose a simple but
surprisingly effective idea—we use both CLIP’s vision encoder *and* its text encoder in our multi-modality alignment module. By directly incorporating CLIP’s text encoder into its visual encodings,
Since CLIP’s vision and text encoders were originally jointly trained, multi-modal adaptation is inherently embedded in their encodings, making its text encoder an ideal choice for encoding instructions. The strong, inherent alignment between the two modalities allows for seamless integration, minimizing the need for complex cross-modal modules or extensive training to achieve alignment. Furthermore, CLIP has demonstrated strong performance across diverse tasks, making it a reliable foundation for multi-modal applications.

We refer to our proposed architecture as EMMA—
Efficient Multi-Modal Adaptation.
Figure 1 illustrates EMMA’s architecture. The left side highlights the high-level structure, where the standard modules of a multi-modal LLM are shaded in gray, while EMMA’s newly introduced Modality Adaptation module is shown in green. On the right, the details of the Modality Adaptation module are depicted. CLIP’s text encoder is used to generate instruction encodings, which are then processed through the Instruction Projection module to prepare them for fusion with the visual tokens. Once the visual and textual representations are generated, we introduce early fusion via a lightweight module called the Visual Alignment module. This component, consisting of a simple linear layer, combines visual and textual tokens to create the model’s multi-modal encoding. The dimensions of this alignment layer are designed to maintain the same number of visual tokens as the baseline model, ensuring consistency in the number of visual tokens passed to the language model.
The Visual Alignment module plays a critical role in ensuring effective alignment between visual and instructions encodings, highlighting the most relevant tokens from the visual encodings in response to the instruction, and thereby delivering more precise visual information to the language model. Its lightweight design facilitates easier interpretability and analysis and more importantly, reduces the training and inference time while outperforming the state-of-the-art models with $10\times$ less parameters. Additionally, their initial alignment allows the text and vision encoders to remain frozen during training, preventing the model from overfitting to the training data.

For training, we follow the same two-stage instruction fine-tuning process as LLaVA. In the pretraining stage, only the Visual Alignment and Projection modules are trained, while the language model remains frozen. During the fine-tuning stage, the LLM is unfrozen and fine-tuned along with the two aforementioned modules. To ensure that any performance improvements are not simply due to the addition of more data, we use the same dataset as the baseline model.
Figure 3 demonstrates the improvements achieved by our proposed method, EMMA, across several benchmarks compared to the baseline model, LLaVA-1.5. EMMA consistently outperforms LLaVA-1.5 on tasks such as MMVP (Tong et al., ) (+9.3%), MuirBench (Wang et al., )(+5.62%), SQA (Lu et al., ) (+4.2%), MMBench (Liu et al., ) (+1.9%), and POPE (Li et al., ) (+1.17%) showing consistent gains in cross-modal tasks requiring visual and textual understanding.

Figure: (a) $\ell_{1}$ norm of weights associated with textual tokens.
Refer to caption: x6.png

### 3.2 Analysis on Modality Adaptation by EMMA

In this section, we analyze the impact of the modality adaptation module introduced by EMMA.

#### Visual Alignment Module.

As explained earlier, the Visual Alignment module, which consists of a linear layer, forms the core of our proposed method. Let $f:\mathbb{R}^{(n+m)\times d}\rightarrow\mathbb{R}^{n\times d}$ represent the Visual Alignment module, where $n$ and $m$ denote the number of visual and textual tokens, respectively, each with a dimensionality of $d$. The Visual Alignment module takes the concatenation of the visual and textual tokens as input to generate $n$ refined visual tokens, and we begin our examination by scrutinizing the matrix $W\in\mathbb{R}^{(n+m)\times n}$ associated with it. By analyzing the norms of the weights corresponding to each token, we can identify which tokens, visual or textual, are most impactful.
The histogram of $\ell_{2}$ and $\ell_{1}$ norms of visual and textual tokens are demonstrated in Figure 5. As the weights of textual tokens are below 1 and the weights of visual tokens are above 1, the $\ell_{1}$ norm of the weights serve as a more indicative measure of token importance.
As expected, the visual tokens exhibit higher weights within the Visual Alignment module, signifying their greater influence on the generated multi-modal representation. Another key observation is that certain textual tokens exert more influence than others. To highlight the relative importance of each textual token, Figure 5(a) presents a bar chart illustrating their respective weights. CLIP’s text encoder generates 77 textual tokens, shown along the x-axis. As depicted in the figure, the earlier tokens tend to have a more significant impact. This finding indicates that the alignment module effectively identifies the most informative textual tokens, as instructions typically consist of brief prompts, with the crucial information concentrated in the early tokens and the remaining tokens often masked.

Figure: Figure 6: We evaluate the effectiveness of the Visual Alignment module by analyzing how well it refines visual representations to capture instruction-specific details. This is measured by calculating the mutual information between the visual representations and the response encodings produced by the text encoder. As illustrated in the figure, EMMA’s mutual information is 1.5 times higher than LLaVA’s, demonstrating its superior alignment with the instruction.
Refer to caption: x8.png

#### Enhancement in Modality Alignment.

The primary objective of EMMA’s modality adaptation is to align visual representations with the instructions, ensuring they emphasize the aspects of the image directed by the instructions. Our method achieves this by incorporating instruction encodings into the refinement process of the visual representations.
In this section, we explore the alignment capabilities of EMMA by examining the visual representations before and after alignment. To perform this analysis, we utilize the MMVP Tong et al. () benchmark, which is designed to expose the visual shortcomings of MLLMs. We focus on images with visually similar encodings but subtle differences, as shown in Figure 4. EMMA demonstrates a 9.3% improvement on this benchmark, underscoring its ability to generate more distinguishable visual representations for such images. To empirically validate this, we compute the $\ell_{2}$ norm of visual representations between randomly selected pairs of MMVP images, comparing the results for pre-aligned and post-aligned representations. The histogram of the norms, shown in Figure 5(b), reveals a clear shift, indicating that the aligned representations are better at distinguishing between these images by focusing on instruction-relevant tokens.

#### Mutual Information between Aligned Visual Tokens and Response Tokens.

Another primary objective of the Modality Adaptation module is to adapt the visual representations with the language modality, ensuring they are well-aligned with the language model and encapsulate the necessary information to accurately respond to the given prompt. To evaluate the contribution of visual tokens to the language model’s output, we use the Mutual Information which quantifies the amount of information obtained about one random variable through the other.
In this analysis, the LLaVA-In-Wild Liu et al. () benchmark is employed, which has a set of 24 images with 60 challenging questions in novel domains. For each of the 60 samples, visual representations are generated using the visual modules of both LLaVA and EMMA. It is important to note that EMMA’s visual module processes the image and prompt to produce instruction-aware representations, whereas LLaVA generates instruction-agnostic encodings. Additionally, for each sample, the corresponding answer is encoded using the Text Encoder. The Mutual Information between the visual and response encodings is then calculated, with the results shown in Figure 6. As illustrated, the mean mutual information for EMMA is 1.5 times higher than that of LLaVA, underscoring the effectiveness of EMMA’s Visual Alignment in steering the model toward accurate language responses.

**Table 2: Comparison between EMMA and previous methods on academic task-oriented datasets. EMMA achieves state-of-the-art performance on 4/5 of the benchmarks while using the fewest number of parameters and training data compared to the previous approaches, utilizing modality adaptation. (PT and IT stand for Pretraining and Instruction fine-tuning repectively.)**
| Method | LLM | #Parameters | PT + IT | VQA${}^{\text{v2}}$ | VisWiz | SQA${}^{\text{I}}$ | GQA | OkVQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP-2 | Vicuna-13B | 13B | 129M | 65.0 | 19.6 | 61 | 41 | 45.9 |
| InstructBLIP | Vicuna-7B | 7.91B | 130.2M | - | 34.5 | 60.5 | 49.2 | - |
| InstructBLIP | Vicuna-13B | 14.2B | 130.2M | - | 33.4 | 63.1 | 49.5 | - |
| Shikra | Vicuna-13B | 13b | 6.1M | 77.4 | - | - | - | 47.2 |
| IDEFICS | LLaMA-7B | 9B | 354M | 50.9 | 35.5 | - | 38.4 | - |
| IDEFICS | LLaMA-65B | 80B | 354M | 60.0 | 36.0 | - | 45.2 | - |
| Qwen-VL-Chat | Qwen-7B | 9.6B | 1.4B | 78.2 | 38.9 | 68.2 | 57 | 56.6 |
| LLaVA-1.5 | Vicuna-7B | 7B | 1.2M | 78.5 | 50.0 | 66.8 | 62.0 | - |
| mPLUG-Owl2 | LLaMA-7B | 8.2B | 348M | 79.4 | 54.5 | 68.7 | 56.1 | 57.7 |
| BRAVE | FlanT5-XL | 10B | 100M | 82.5 | 54.2 | - | 52.7 | 66.0 |
| EMMA | Vicuna-7B | 7B | 1.8M | 89.42 | 56.03 | 73.14 | 56.01 | 68.57 |

## 4 Experimental Evaluation

In this section, we begin by comparing EMMA with state-of-the-art multi-modal LLMs, using the benchmarks introduced earlier. Following this, we conduct a robustness analysis with a focus on hallucination. We conduct an ablation study to identify the optimal layer output from the text encoder for use as textual representations.

#### Implementation Details.

We employ CLIP-ViT-L-14, trained on 336^2 pixel images, as the base image encoder and text encoder. The Visual Alignment module is initialized with the identity matrix for the visual tokens and all zeros for the instruction tokens to transfer all the visual tokens at the beginning of training. Moreover, the Visual Alignment module is designed to maintain the same number of visual tokens as the baseline model. The latest Vicuna v1.5 Zheng et al. () is used as the base LLM. EMMA uses the same set of hyperparameters as the
LLaVA-1.5. For all the analysis performed in the Section 3, we use the same dataset as the baseline model, which is 558K and 665K samples for the pretraining and fine-tuning stages respectively. In the Evaluation setting, we have preserved the same pertaining data but scaled the fine-tuning data to 1.2M samples, which is the most efficient data compared to all of the state-of-the-art methods (except for LLaVA) as shown in Table 4.

#### Benchmarks.

We evaluate EMMA across 10 benchmarks that span a diverse set of tasks, including scientific question answering, visual question answering with poor image quality, integrated perception and reasoning tasks, visual dialogue, and general reasoning. The results of recent benchmarks designed specifically for instruction-following large multi-modal models (LMMs) are presented in Table 4, while Table 2 outlines the performance on academic task-oriented benchmarks. EMMA emerges as the state-of-the-art model for 4 out of 5 academic task-oriented benchmarks, outperforming mPLUG-Owl2 which has $50\times$ larger modality adaptation module and is trained on $300\times$ more data, and BRAVE which has $24\times$ larger vision encoder and is trained on $100\times$ more data.
On MLLM specialized benchmarks, EMMA delivers the best performance on 3 out of 5 benchmarks and second best on the two others with less than 0.5% difference. In conclusion, EMMA achieves the best performance in 7 out of 10 benchmarks, despite utilizing the simplest architecture, outperforming other MLLMs that rely on complex modality adaptation modules.

**Table 3: Robustness to Hallucination. This table compares EMMA’s robustness against hallucinations with LLaVA, demonstrating consistent improvements over the baseline in avoiding hallucinations.**
| Model |  | AMBER |  | FOIL |
| --- | --- | --- | --- | --- |
| Attr | Rel | Exis |  |  |
| LLaVA-1.5 | 20.23 | 37.1 | 91.43 | 50.08 |
| EMMA-1.0 | 26.15$+$5.92 | 37.34$+$0.24 | 94.50$+$3.07 | 66.13$+$16.05 |

#### Robustness & Hallucination.

Robustness and the ability to avoid hallucination are essential for multi-modal large language models (MLLMs), which are increasingly applied in areas like medical diagnostics to interpret complex text and image data. Hallucination, a significant security threat to MLLMs, occurs when the model generates information that does not accurately represent the provided images or text. As a result, evaluating and mitigating hallucinations is a critical step in ensuring the reliability of MLLMs before real-world deployment, making it a key focus in model performance assessments. The hallucination evaluations in this section are performed utilizing two benchmarks, AMBER and FOIL, which do not rely on additional LLMs, thereby providing a direct and controlled means of assessing the model’s ability to avoid hallucinations. These benchmarks focus on specific challenges in multi-modal reasoning, testing the model’s accuracy in aligning textual and visual content without introducing erroneous information.

Figure: Figure 7: This radar chart compares EMMA’s performance when leveraging textual features generated by either the penultimate or last layer of the text encoder, highlighting the advantage of features associated with the penultimate layer.
Refer to caption: x9.png

AMBER consists of 7628, 4924, and 1663 samples for the Attribute, Relation, and Existence categories, respectively. FOIL contains a total of 99,480 test samples, of which 92,705 are straightforward, allowing both LLaVA and our method to successfully avoid hallucinations. However, 6,775 samples present more challenging cases. We compare LLaVA-1.5, the baseline for our approach, with EMMA. The results, shown in Table 3, indicate that EMMA outperforms the baseline, with significant performance gaps in two of the four benchmarks.

#### Ablations on Text Encoder.

We ablate the text feature’s abstraction level in this section. The textual features can be optionally derived from either the final layer or the penultimate layer of the CLIP Text Encoder. A comparison between the two methods of extracting textual features is presented in Figure 7, demonstrating the clear superiority of features derived from the penultimate layer. We hypothesize that the final layer of CLIP captures more global and abstract semantics of the instruction, whereas the penultimate layer focuses on finer details. Additionally, since visual features are extracted from the corresponding layer in the visual encoder, using the same layer in the text encoder ensures both modalities operate at a similar level of abstraction, promoting better alignment between them.

## 5 Conclusion

In this work, we addressed the inefficiencies in the modality adaptation modules employed by current multi-modal large language models. We hypothesized that the initial alignment between visual and textual encodings plays a critical role in determining the complexity level of the modality adaptation module and the necessary amount of training data. Our lightweight approach, EMMA (Enhanced Multi-Modal Adaptation), leverages CLIP’s text encoder to generate instruction encodings and, by exploiting this initial alignment, demonstrates that the modality adaptation module can be simple while still enhancing the alignment between visual and textual modalities. Through extensive analysis, we demonstrated that EMMA effectively produces instruction-aware visual representations aligned with the language model. Our experiments, evaluated across multiple benchmarks, show that EMMA significantly outperforms state-of-the-art models that use modality adaptation modules $50\times$ larger. Finally, our robustness analysis, particularly in hallucination avoidance, confirmed EMMA’s superior ability to accurately process multi-modal data, even in the challenging scenarios.

### Acknowledgments

This paper is supported in part by the Army Research Office under grant number W911NF-21-1-0155 and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010. Additional support was provided by the NYU IT High Performance Computing resources, services, and staff expertise.

**Table 4: This table compares EMMA with previous methods on specialized MLLM benchmarks. EMMA delivers the best performance on 3/5 benchmarks and secures second place on the remaining two with less than 0.5% difference.**
| Method | LLM | #Parameters | PT + IT | MMB${}^{\text{EN}}$ | MMB${}^{\text{CN}}$ | MMMU | MathVista | Muirbench |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP-2 | Vicuna-13B | 14.2B | 129M | - | - | - | - | - |
| InstructBLIP | Vicuna-7B | 7.91B | 130.2M | 36 | 23.7 | - | - | - |
| InstructBLIP | Vicuna-13B | 14.2B | 130.2M | - | - | - | 25.3 | - |
| Shikra | Vicuna-13B | 13B | 6.1M | 58.8 | - | - | - | - |
| IDEFICS | LLaMA-7B | 9B | 354M | 48.2 | 25.2 | - | 19.8 | - |
| IDEFICS | LLaMA-65B | 80B | 354M | 54.5 | 38.1 | - | - | - |
| Qwen-VL-Chat | Qwen-7B | 9.6B | 1.4B | 60.6 | 56.7 | 35.9 | - | - |
| LLaVA-1.5 | Vicuna-7B | 7B | 1.2M | 64.3 | 58.3 | 35.11 | 21.1 | 23.46 |
| mPLUG-Owl2 | LLaMA-7B | 8.2B | 348M | 64.5 | - | 32.7 | 22.2 | - |
| BRAVE | FlanT5-XL | 10B | 100M | - | - | - | - |  |
| EMMA | Vicuna-7B | 7B | 1.8M | 66.44 | 60.15 | 35.44 | 25.1 | 32.0 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: * In line 295, the claim that leveraging the inherent alignment properties of CLIP encoders obviates the need for complex cross-modal modules and extensive training to attain alignment is deemed unwarranted, given that the instruction embeddings fed into the LLM are not inherently aligned with CLIP features. Indeed, the proposed EMMA incorporates a projection layer akin to LLaVA after its Modality Adaptation module and also adopts the pre-alignment training from LLaVA. The reviewer considers this projection layer a pivotal component of EMMA's alignment achievement. Conducting additional ablation experiments wherein this projection layer is removed could potentially strengthen the authors' argument.

* While Line 319 mentions the utilization of data from the same datasets, the 1.8M training samples employed by EMMA deviate in quantity from those used in any of the baselines, rendering it challenging to demonstrate the effectiveness of the Modality Adaptation module. Given that LLaVA 1.5 is fully open-source and shares a very similar structure with EMMA, adding the following experimental results would significantly strengthen this paper: (1) the outcomes of EMMA using the 1.2M training set from LLaVA 1.5, and (2) the results of LLaVA 1.5 utilizing the 1.8M training set from EMMA.

* The conclusion drawn in line 343 is not convincing. The findings in Figure 5(a) demonstrate that text tokens are indeed involved in the feature fusion process. Yet, they do not sufficiently establish that the alignment module effectively identifies the "most" informative textual tokens. The reviewer thinks the positions of the most valuable tokens for an arbitrarily given instruction are random, and the current framework adopting a simple linear layer, which allocates fixed importance weights for textual tokens during inference, can not adapt to this change.

* The reviewer praises the methodology illustrated in Figure 6, which employs Mutual Information to assess the importance of aligned visual representations in the reasoning process. However, concerns about its reliability are raised. A significant counterexample emerges in situations where the dataset requires responses strictly in YES/NO format, making the mutual information between the visual representations and the response encodings lack meaningful relevance.

### Minor:
* Which figure does the conclusion in line 338 correspond to?
soundness: 2
strengths: * Leveraging the inherent alignment properties of pre-trained CLIP text and vision encoders to implement an early fusion module is intuitive.
* The model's structural design is characterized by its simplicity and clarity.
* The analysis presents a novel perspective on modality adaptation.
summary: This paper introduces EMMA, a lightweight cross-modality module that implements visual-language early fusion, generating instruction-aware visual representations for the language model. This module adds less than 0.03% of parameters to the framework while boosting performance across multiple MLLM benchmarks.
weaknesses: * In line 251, it is mentioned that current methodologies depend on complex cross-modality modules, particularly highlighting mPLUG-Owl2. However, the paper does not provide a theoretical comparison with methods like LLaVA and Qwen-VL, which utilize straightforward projection layers that add only a minimal number of parameters to achieve visual-language alignment.

* The proposed EMMA introduces a Modality Adaptation module based on the correlation between word tokens in instructions and visual tokens. However, this methodology lacks performance guarantees when instructions are verbose or correlate poorly with visual tokens. The benchmarks selected in the paper are skewed towards VQA tasks, which inherently possess a high correlation between instructions and visual tokens. It is imperative that benchmarks assessing fine-grained perception capabilities, such as document understanding and OCR tasks (e.g., Chartqa[1] and Docvqa[2]), should also be evaluated to ensure comprehensiveness.

* Lack of comparison with visual-language early fusion works (e.g., QA-ViT[3]).

[1] Chartqa: A benchmark for question answering about charts with visual and logical reasoning.\
[2] Docvqa: A dataset for vqa on document images\
[3] Question Aware Vision Transformer for Multimodal Reasoning. In CVPR, 2024

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see weakness.

1. Can the authors provide more formulized descriptions of the proposed method? For example, this could include mathematical formulas or structured descriptions similar to those at the beginning of Section 3.2.

2. Can the authors include additional experimental comparisons with other adapter-based methods, such as [1-2] ? These would offer a more comprehensive evaluation of EMMA’s performance relative to other state-of-the-art adapters.

3. EMMA uses CLIP-ViT-L-14 as its visual encoder, but it's unclear whether the same visual model is used in the comparison models. Could the authors provide details on the visual models used by the other models in Tables 2 and 4, along with their parameter counts?

[1] MMA: Multi-Modal Adapter for Vision-Language Models

[2] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
soundness: 3
strengths: 1. Detailed analytical insights: The paper offers a thorough analysis of the modality alignment process through l1,  l2, and mutual information comparisons. This analysis provides deeper insights into how visual and textual tokens are integrated within the alignment module, enhancing the interpretability of the model's decision-making process.

2. Robustness against hallucinations: The empirical results highlight EMMA's superior ability to avoid hallucinations in multi-modal tasks, as demonstrated on benchmarks such as AMBER and FOIL. This robustness is critical for improving the reliability of MLLMs in real-world scenarios where model hallucinations can pose significant risks.
summary: This paper introduces EMMA (Efficient Multi-Modal Adaptation), a lightweight module designed to improve cross-modal alignment in multi-modal large language models (MLLMs). EMMA integrates visual and textual encodings using a low-parameter approach that leverages pre-trained CLIP models. Specifically, the method incorporates both CLIP’s vision encoder and its text encoder into the multi-modality alignment module, taking advantage of their pre-trained alignment to fuse visual representations with instruction encodings efficiently. The proposed architecture adds less than 0.2% additional parameters, making it highly efficient compared to existing models such as mPLUG-Owl2.
weaknesses: 1. Inconsistent writing and presentation: The overall writing of the paper is somewhat disorganized, making it difficult to follow and reducing its readability. For example, Section 3.1 is intended to introduce the EMMA method, but the section ends with a discussion of experimental results (in the last paragraph on page 6). Similarly, in Section 3.2, which is supposed to focus on the Analysis on Modality Adaptation by EMMA, the section starts by explaining model details, which would be better suited earlier in the paper. Additionally, Figure 1 is referenced only on page 6, even though it appears on page 2. Many figures and tables also lack proper axis labels and detailed captions, which further hinders the reader's ability to understand the data. For instance, it would be helpful to clarify the contents of the axes in Figures 2, 5, and 6. 

2. Unfair experimental comparisons: The paper does not provide a fair and direct comparison between different vision-language adapters, which is essential since the focus is on adapter efficiency. While achieving complete consistency between the vision and language models might be challenging, it would be beneficial to at least include an additional set of experiments that utilize other adapters, such as [1-2], alongside the same vision and language models. This would allow for a more fair comparison of their performance against the EMMA adapter. By keeping the vision and language models constant across different methods, the paper could offer a clearer evaluation of the specific contributions made by EMMA’s adapter design.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: Please see the weakness.

My major concern is about the novelty, both the slightly incremental structure based on LLaVA and the effectiveness of CLIP-text encoder.
soundness: 3
strengths: 1-Clear motivation: Improving effeciency is important for MLLMs.
2-Well-written paper: Easy to follow.
3-Substantial experiments and corresponding discussion.
summary: This paper proposes EMMA, a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model.
weaknesses: 1-The model is very similar to LLaVA: EMMA introduces CLIP-text encoder and an early fusion module.
2-The involvement of CLIP-text encoder is not very convincing: CLIP is trained for classification task only, but EMMA is a general-purposed instruction-aware MLLM. I am doubtful about the effectiveness when CLIP-text encoder deals with long instructions. You may make some visualization of the activation when CLIP encounter some long instructions / tasks apart from classification.

3-The discussion in Sec. 3.2 should incorporate more datasets apart from MMVP. Because I am not sure of the effectiveness of the CLIP-text encoder as above.

4-Unfair comparison: Table 1 without LLaVA, but Figure 3 only compare with LLaVA. Should compare LLaVA in Table1 and compare mPLUG-Owl2 and other methods in Figure 3. You may provide a table / figure to compare more methods.

5-The scale of Figure 3 is inappropriate, slight improves with significant distance, e.g. 64.3 -> 66.19 in MMB-en. I think a histogram bar will be better.

6-The claim in line 247-250 is too absolute: *Therefore, a persistent challenge remains in effectively aligning these two modalities to ensure seamless fusion and task-specific adaptability.* Actually, task-specific adaptability is important, but previous works solve it by LLM (very large scale).

7-Figure5: Two legends: purple and light green. What do dark green bars represent? Cannot understand right figure. Please give more decriptions.

8-Performances improve slightly in lots of datasets.

9-POPE dataset has many subsets, should provide detailed performances on each of them in a table.

10-If you mentioned hallucination, you should discuss and cite more papers. For example, there are relation hallucinations: R-Bench[1], and MMRel[2], and some datasets containing attribute hallucination: MME[3]. I do not ask you to evaluate on a bunch of datasets, you are just suggested to cite them and then the readers will have a clear picture of the hallucination tasks.

[1] Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in lvlms. In ICML, 2024
[2] Nie J, Zhang G, An W, et al. MMRel: A Relation Understanding Dataset and Benchmark in the MLLM Era[J]. arXiv preprint arXiv:2406.09121, 2024.
[3] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zux- 677
uan Wu. Synthesize, diagnose, and optimize: Towards fine- 678
grained vision-language understanding. In CVPR, 2024

Meta-review: This paper proposes a visual alignment design in multi-modal LLMs. It is conducted via an early fusion module with in-depth analysis and experimental validations. The reviews are mixed. [QZxi, Eq4D] raised several issues regarding a lack of theoretical comparison, lack of performance guarantees, and limited comparison with adapter-based approaches. During the discussion phase, the authors address these issues by providing additional experiments, and technical illustrations. [Eq4D] is generally satisfied while [QZxi] still raised unconvincing performance comparisons, and lack of claim support via ablation studies. Overall, the AC has checked all the files, and feels the current early fusion design are not lightweight in that the text encoder is introduced with additional instruction projection. It theoretically makes sense that adding additional interaction modules benefits vision-language correlations. The advantages of these largely own to the text encoder which utilizes pre-trained data. Meanwhile, the back-and-forth projection of text data (i.e., the input text is projected into CLIP space and then back to the text tokenized space) is not elegant and makes the design more engineering-like. The authors shall better improve the current design and welcome to the next venue.

Decision: Reject