EMMA: Efficient Visual Alignment in Multi-Modal LLMs

Authors: Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, Farshad Khorrami
Primary area: foundation or frontier models, including LLMs
Keywords: Multi-Modal LLMs, Modality Adaptation, Visual Alignment

Abstract:
Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress
in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs. 
In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730015505262

thread_type:
official_review

<thread_object>
creator: reviewer_p1vp

<content>
summary:
This paper proposes EMMA, a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model.

strengths:
1-Clear motivation: Improving effeciency is important for MLLMs.
2-Well-written paper: Easy to follow.
3-Substantial experiments and corresponding discussion.

weaknesses:
1-The model is very similar to LLaVA: EMMA introduces CLIP-text encoder and an early fusion module.
2-The involvement of CLIP-text encoder is not very convincing: CLIP is trained for classification task only, but EMMA is a general-purposed instruction-aware MLLM. I am doubtful about the effectiveness when CLIP-text encoder deals with long instructions. You may make some visualization of the activation when CLIP encounter some long instructions / tasks apart from classification.

3-The discussion in Sec. 3.2 should incorporate more datasets apart from MMVP. Because I am not sure of the effectiveness of the CLIP-text encoder as above.

4-Unfair comparison: Table 1 without LLaVA, but Figure 3 only compare with LLaVA. Should compare LLaVA in Table1 and compare mPLUG-Owl2 and other methods in Figure 3. You may provide a table / figure to compare more methods.

5-The scale of Figure 3 is inappropriate, slight improves with significant distance, e.g. 64.3 -> 66.19 in MMB-en. I think a histogram bar will be better.

6-The claim in line 247-250 is too absolute: *Therefore, a persistent challenge remains in effectively aligning these two modalities to ensure seamless fusion and task-specific adaptability.* Actually, task-specific adaptability is important, but previous works solve it by LLM (very large scale).

7-Figure5: Two legends: purple and light green. What do dark green bars represent? Cannot understand right figure. Please give more decriptions.

8-Performances improve slightly in lots of datasets.

9-POPE dataset has many subsets, should provide detailed performances on each of them in a table.

10-If you mentioned hallucination, you should discuss and cite more papers. For example, there are relation hallucinations: R-Bench[1], and MMRel[2], and some datasets containing attribute hallucination: MME[3]. I do not ask you to evaluate on a bunch of datasets, you are just suggested to cite them and then the readers will have a clear picture of the hallucination tasks.

[1] Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in lvlms. In ICML, 2024
[2] Nie J, Zhang G, An W, et al. MMRel: A Relation Understanding Dataset and Benchmark in the MLLM Era[J]. arXiv preprint arXiv:2406.09121, 2024.
[3] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zux- 677
uan Wu. Synthesize, diagnose, and optimize: Towards fine- 678
grained vision-language understanding. In CVPR, 2024

questions:
Please see the weakness.

My major concern is about the novelty, both the slightly incremental structure based on LLaVA and the effectiveness of CLIP-text encoder.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474260829

thread_type:
official_review

<thread_object>
creator: reviewer_eq4d

<content>
summary:
This paper introduces EMMA (Efficient Multi-Modal Adaptation), a lightweight module designed to improve cross-modal alignment in multi-modal large language models (MLLMs). EMMA integrates visual and textual encodings using a low-parameter approach that leverages pre-trained CLIP models. Specifically, the method incorporates both CLIP’s vision encoder and its text encoder into the multi-modality alignment module, taking advantage of their pre-trained alignment to fuse visual representations with instruction encodings efficiently. The proposed architecture adds less than 0.2% additional parameters, making it highly efficient compared to existing models such as mPLUG-Owl2.

strengths:
1. Detailed analytical insights: The paper offers a thorough analysis of the modality alignment process through l1,  l2, and mutual information comparisons. This analysis provides deeper insights into how visual and textual tokens are integrated within the alignment module, enhancing the interpretability of the model's decision-making process.

2. Robustness against hallucinations: The empirical results highlight EMMA's superior ability to avoid hallucinations in multi-modal tasks, as demonstrated on benchmarks such as AMBER and FOIL. This robustness is critical for improving the reliability of MLLMs in real-world scenarios where model hallucinations can pose significant risks.

weaknesses:
1. Inconsistent writing and presentation: The overall writing of the paper is somewhat disorganized, making it difficult to follow and reducing its readability. For example, Section 3.1 is intended to introduce the EMMA method, but the section ends with a discussion of experimental results (in the last paragraph on page 6). Similarly, in Section 3.2, which is supposed to focus on the Analysis on Modality Adaptation by EMMA, the section starts by explaining model details, which would be better suited earlier in the paper. Additionally, Figure 1 is referenced only on page 6, even though it appears on page 2. Many figures and tables also lack proper axis labels and detailed captions, which further hinders the reader's ability to understand the data. For instance, it would be helpful to clarify the contents of the axes in Figures 2, 5, and 6. 

2. Unfair experimental comparisons: The paper does not provide a fair and direct comparison between different vision-language adapters, which is essential since the focus is on adapter efficiency. While achieving complete consistency between the vision and language models might be challenging, it would be beneficial to at least include an additional set of experiments that utilize other adapters, such as [1-2], alongside the same vision and language models. This would allow for a more fair comparison of their performance against the EMMA adapter. By keeping the vision and language models constant across different methods, the paper could offer a clearer evaluation of the specific contributions made by EMMA’s adapter design.

questions:
Please see weakness.

1. Can the authors provide more formulized descriptions of the proposed method? For example, this could include mathematical formulas or structured descriptions similar to those at the beginning of Section 3.2.

2. Can the authors include additional experimental comparisons with other adapter-based methods, such as [1-2] ? These would offer a more comprehensive evaluation of EMMA’s performance relative to other state-of-the-art adapters.

3. EMMA uses CLIP-ViT-L-14 as its visual encoder, but it's unclear whether the same visual model is used in the comparison models. Could the authors provide details on the visual models used by the other models in Tables 2 and 4, along with their parameter counts?

[1] MMA: Multi-Modal Adapter for Vision-Language Models

[2] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631148138

thread_type:
official_review

<thread_object>
creator: reviewer_qzxi

<content>
summary:
This paper introduces EMMA, a lightweight cross-modality module that implements visual-language early fusion, generating instruction-aware visual representations for the language model. This module adds less than 0.03% of parameters to the framework while boosting performance across multiple MLLM benchmarks.

strengths:
* Leveraging the inherent alignment properties of pre-trained CLIP text and vision encoders to implement an early fusion module is intuitive.
* The model's structural design is characterized by its simplicity and clarity.
* The analysis presents a novel perspective on modality adaptation.

weaknesses:
* In line 251, it is mentioned that current methodologies depend on complex cross-modality modules, particularly highlighting mPLUG-Owl2. However, the paper does not provide a theoretical comparison with methods like LLaVA and Qwen-VL, which utilize straightforward projection layers that add only a minimal number of parameters to achieve visual-language alignment.

* The proposed EMMA introduces a Modality Adaptation module based on the correlation between word tokens in instructions and visual tokens. However, this methodology lacks performance guarantees when instructions are verbose or correlate poorly with visual tokens. The benchmarks selected in the paper are skewed towards VQA tasks, which inherently possess a high correlation between instructions and visual tokens. It is imperative that benchmarks assessing fine-grained perception capabilities, such as document understanding and OCR tasks (e.g., Chartqa[1] and Docvqa[2]), should also be evaluated to ensure comprehensiveness.

* Lack of comparison with visual-language early fusion works (e.g., QA-ViT[3]).

[1] Chartqa: A benchmark for question answering about charts with visual and logical reasoning.\
[2] Docvqa: A dataset for vqa on document images\
[3] Question Aware Vision Transformer for Multimodal Reasoning. In CVPR, 2024

questions:
* In line 295, the claim that leveraging the inherent alignment properties of CLIP encoders obviates the need for complex cross-modal modules and extensive training to attain alignment is deemed unwarranted, given that the instruction embeddings fed into the LLM are not inherently aligned with CLIP features. Indeed, the proposed EMMA incorporates a projection layer akin to LLaVA after its Modality Adaptation module and also adopts the pre-alignment training from LLaVA. The reviewer considers this projection layer a pivotal component of EMMA's alignment achievement. Conducting additional ablation experiments wherein this projection layer is removed could potentially strengthen the authors' argument.

* While Line 319 mentions the utilization of data from the same datasets, the 1.8M training samples employed by EMMA deviate in quantity from those used in any of the baselines, rendering it challenging to demonstrate the effectiveness of the Modality Adaptation module. Given that LLaVA 1.5 is fully open-source and shares a very similar structure with EMMA, adding the following experimental results would significantly strengthen this paper: (1) the outcomes of EMMA using the 1.2M training set from LLaVA 1.5, and (2) the results of LLaVA 1.5 utilizing the 1.8M training set from EMMA.

* The conclusion drawn in line 343 is not convincing. The findings in Figure 5(a) demonstrate that text tokens are indeed involved in the feature fusion process. Yet, they do not sufficiently establish that the alignment module effectively identifies the "most" informative textual tokens. The reviewer thinks the positions of the most valuable tokens for an arbitrarily given instruction are random, and the current framework adopting a simple linear layer, which allocates fixed importance weights for textual tokens during inference, can not adapt to this change.

* The reviewer praises the methodology illustrated in Figure 6, which employs Mutual Information to assess the importance of aligned visual representations in the reasoning process. However, concerns about its reliability are raised. A significant counterexample emerges in situations where the dataset requires responses strictly in YES/NO format, making the mutual information between the visual representations and the response encodings lack meaningful relevance.

### Minor:
* Which figure does the conclusion in line 338 correspond to?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732125560672

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank the reviewers for their interest, positive remarks, and comments. Below we emphasize two important comments on the contributions of our work and provide a summary of the changes in the revision of our paper. Please feel free to reach out if there are still questions or comments you would like us to address. 

### **1- On the importance of having a lightweight modality adaptation module**

EMMA is designed to provide an efficient modality adaptation that adheres to the following principles:

**Early Fusion of Visual and Instruction Encodings**: a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model.

**Harnessing CLIP’s Inherent Alignment**: EMMA is the first to leverage the inherent alignment between the visual and textual components of CLIP and enhance the efficiency of early modality adaptation in MLLMs, reducing the reliance on parameter-intensive modality adaptation modules.

**Lightweight and Data-Efficient Design**: The model requires significantly fewer samples for training or fine-tuning, avoiding the need for hundreds of millions of data points.

**Enhanced Robustness Against Overfitting**:
EMMA’s design minimizes the risk of overfitting, even with limited training samples (LLaVA's instruction tuning data of size 1.2M). This robustness ensures reliable performance across diverse datasets.

**Preservation of Visual Token Generality**:
The model carefully refines visual tokens by the instructions while preserving the tokens’ generality, maintaining their adaptability across various downstream tasks.

**Flexibility for Diverse Applications**: EMMA’s lightweight architecture allows efficient fine-tuning across diverse datasets and applications [1, 2] (In contrast to its counterparts, which employ complex modality adaptation modules requiring millions of data for finetuning).

The proposed method is intentionally designed with these considerations in mind, balancing simplicity and efficiency to meet these critical objectives.

### **2 - The direct effect of EMMA's modality adaptation**

To isolate the impact of adding the modality adaptation module to the baseline model (LLaVA), we conducted an initial set of experiments where the modality adaptation module was integrated into LLaVA's architecture while keeping all other settings the same as LLaVA (including training data, training setup, etc.). This ensures a fair evaluation of the capabilities of EMMA's proposed modality adaptation. The results, shown in Figure 5, reflect improvements attributable solely to the modality adaptation module, as it is the only factor that was modified. The table below is provided as a reference, derived from Figure 5.

| Method	| Data | SQA	| GQA | POPE | MMB-en |	MMB-cn | MMVP | A-Bench | MuirB	|
| ----  | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
LLaVA-1.5 |1.2M | 66.8 | 62.0  | 86.1 | 64.3 | 58.3 | 24.7 | 62.97 | 23.46 | 
EMMA | 1.2M | 71.00 | 62.78 | 87.27 | 66.19 | 60.2 | 34 | 63.48 | 29.08 | 
Difference | --- | **+4.2%** | +0.78% | +1.17% | **+1.89%** | **+1.9%** | **+9.3%** | +0.51% | **+5.62%**

### **3 - Summary of changes in the paper**

- The discussion on the direct impact of EMMA's modality adaptation has been relocated to the ablation study section for clarity and better context.
- Detailed descriptions of the model have been transferred from the analysis section (3.2) to section 3.1 to provide the full picture of our proposed method in one part.
- Mathematical formulations of EMMA have been included in section 3.1 to enhance the understanding of EMMA's methodology.
- Figure 1 has been repositioned closer to section 3.1, where EMMA's architecture is introduced, ensuring better alignment between the explanation and the visual representation.
- Missing details in the figures have been thoroughly added to provide a complete and accurate depiction.
- Additional hallucination benchmarks have been incorporated into Table 4 to provide a more comprehensive evaluation of hallucinations.
- The QA-ViT[3] method which was missed from the related works and evaluations was added to both sections.
  
Please note that, due to the revisions made to the paper, the numbering of figures and tables may have changed. To maintain consistency, all references to figures and tables correspond to the revised version of the paper submitted during the rebuttal period.

[1] LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day  

[2] PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding

[3] Question Aware Vision Transformer for Multimodal Reasoning. In CVPR, 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1733173674989

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you once again for your comments and feedback. We would like to kindly remind you that we are approaching the end of the rebuttal period, so please don’t hesitate to reach out if you still have any questions or comments about our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733196580104

thread_type:
official_comment

<thread_object>
creator: area_chair_kjgh

<content>
title:
Discussion due soon

comment:
Dear all reviewers,

Our reviewer-author discussion will end soon. For each of you, please check all the files and see if anything you'd like to discuss with authors.

Best, Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a visual alignment design in multi-modal LLMs. It is conducted via an early fusion module with in-depth analysis and experimental validations. The reviews are mixed. [QZxi, Eq4D] raised several issues regarding a lack of theoretical comparison, lack of performance guarantees, and limited comparison with adapter-based approaches. During the discussion phase, the authors address these issues by providing additional experiments, and technical illustrations. [Eq4D] is generally satisfied while [QZxi] still raised unconvincing performance comparisons, and lack of claim support via ablation studies. Overall, the AC has checked all the files, and feels the current early fusion design are not lightweight in that the text encoder is introduced with additional instruction projection. It theoretically makes sense that adding additional interaction modules benefits vision-language correlations. The advantages of these largely own to the text encoder which utilizes pre-trained data. Meanwhile, the back-and-forth projection of text data (i.e., the input text is projected into CLIP space and then back to the text tokenized space) is not elegant and makes the design more engineering-like. The authors shall better improve the current design and welcome to the next venue.