LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

Authors: Shaolei Zhang, Qingkai Fang, Zhe Yang, Yang Feng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Multimodal Models, Large Language Models

Abstract:
The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LLMs), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards efficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve a high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens in advance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729329142593

thread_type:
official_review

<thread_object>
creator: reviewer_cwer

<content>
summary:
This paper proposes an efficient LMM with minimal vision tokens. 1/ To achieve a high compression ratio of vision tokens while preserving visual information, the authors first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers, where they fuse visual information into text tokens. 2/ LLaVA-Mini introduces a novel modality pre-fusion to fuse visual information into text tokens in advance before feeding into LLM and a compression module to reduce #vision tokens into minimal ones. 3/ Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on GPU hardware with 24GB of memory.

strengths:
1/ The problem of how to improve efficiency of MLLM is important. 

2/ The authors conducted a very interesting and detailed analysis of how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers, where they fuse visual information into text tokens. 

3/ LLaVA-Mini introduces a modality pre-fusion module to fuse visual information into text tokens in advance before feeding into LLM and a compression module to reduce #vision tokens into minimal ones. 

4/ Experimental results are strong and convincing.

weaknesses:
1/ According to Sec 3, if I understand correctly, for each downstream task, this paper trains a new, independent compression and modality pre-fusion modules. In other words, for another task, we need to train another model. I wonder, how does this approach differ from task-specific distillation? It seems the reason why the efficiency can be improved significantly is because of removing unnecessary tokens for the target downstream task. Instead, would it be possible to train generic compression and modality pre-fusion modules in stage 1 that can be generalised to various downstream tasks?

2/ For the proposed approach, a new modality pre-fusion module has been introduced. I wonder if this is necessary. Instead, can we re-use the first few layers, which have high activation for vision tokens, as the fusion module. To further include the compression module, in the first few layers of LLM, the part digesting vision tokens can have two pathways, one pathway is to output the compressed tokens while the other pathway is to be gradually fused together with the text tokens.  

3/ When it comes to video, there are some recent streaming video LLM works such as "VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024" which also advocates the idea of representing each frame with minimal vision tokens to improve efficiency. It might be worthwhile to compare or just discuss.

questions:
Kindly refer to the weakness sec above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730291257353

thread_type:
official_review

<thread_object>
creator: reviewer_eqge

<content>
summary:
a. The paper proposes LLaVA-Mini, an MLLM with improved efficiency via token compression. Specifically, the authors first unveil that early layers of MLLM are important for visual understanding. On top of this finding,  the efficient MLLM is then designed with 2 strategies: 1) compressing input visual tokens using the query-based resampler and 2) fusing visual tokens with text instruction tokens in early layers of LLM.The paper is well written, and delivered with clear motivation and simple yet effective methods.

b. The authors conducted comprehensive experiments to demonstrate the effectiveness of the methods. On the other hand, LLava-mini manages to achieve competitive performance with much-reduced tokens.

strengths:
A. The paper is well written, and delivered with clear motivation and simple yet effective methods.

B. The authors conducted comprehensive experiments to demonstrate the effectiveness of the methods. On the other hand, LLava-mini manages to achieve competitive performance with much-reduced tokens.

weaknesses:
A. Since the token numbers are substantially reduced. LLaVA-mini may achieve inferior performance under scenarios demanding fine-grained visual details, e.g., text-oriented benchmarks including Text VAQ, especially when compared with models with longer visual sequence input （e.g., LLaVA-Next)

B. The paper lacks an important baseline, namely, keeping the visual tokens in early layers of LLMs and then compressing the visual token output on the L-th (4 in the paper) layer.

questions:
The authors claim that for video inputs, they train and infer via sampling frames at 1fps. In other words, the number of compressed visual tokens for videos of different lengths varies. I’d like to know if there’s any strategy (e.g, temporal embedding) to solve the discrepancy between training and inference (i.e., much longer videos).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704311322

thread_type:
official_review

<thread_object>
creator: reviewer_wp7g

<content>
summary:
This paper presents LLaVA-Mini, an efficient large multimodal model that significantly reduces computational overhead while maintaining competitive performance. The key innovation lies in its ability to represent visual information with as few as one vision token, achieved through two main components: a modality pre-fusion module and a vision token compression module. The authors first conduct a layer-wise analysis of how LLMs process vision tokens, revealing that vision tokens are most crucial in early layers. Based on this insight, they design the pre-fusion module to process visual information before it enters the LLM, and employ a compression mechanism to minimize the number of vision tokens. Experimental results across 11 image and 7 video benchmarks demonstrate that LLaVA-Mini achieves comparable performance to LLaVA-v1.5 while reducing vision tokens from 576 to 1, resulting in 77% FLOPs reduction and significant memory savings (from 360MB to 0.6MB per image). This enables processing of long videos exceeding 10,000 frames and reduces inference latency from 100ms to 40ms.

strengths:
1. Quality:
- Comprehensive empirical validation across 18 benchmarks (11 image-based and 7 video-based) demonstrates the robustness of the approach.
- The method achieves significant efficiency improvements (77% FLOPs reduction, 600x memory reduction) while maintaining competitive performance with the baseline.

2. Clarity:
- The motivation and problem formulation are well-articulated, clearly identifying the computational challenges in current LMMs.
- The paper presents a logical progression from empirical observations to proposed solutions.

3. Significance:
- The dramatic reduction in computational resources makes multimodal processing more accessible.
- The approach addresses a critical limitation in current LMMs, potentially enabling real-time applications and processing of high-resolution inputs.

weaknesses:
1. Writing:
- The attention weight analysis lacks crucial technical details. The authors do not specify how they handle multiple attention heads (whether through averaging, individual analysis, or selective visualization), making the results difficult to reproduce.
- The paper fails to clarify how attention scores are averaged among instruction, vision, and response tokens, which is essential for understanding the analysis methodology.
- Despite using LLaVA 1.5 and LLaVA NeXT as base models, which both incorporate high and low-resolution vision inputs, the analysis does not differentiate between these different types of vision tokens.

2. Technical Inconsistencies:
- There's a discrepancy between the visualization results and claims: while the authors argue that attention to vision tokens decreases in latter layers, the entropy visualization shows persistently high values for vision tokens in these layers, creating confusion about the actual attention distribution.
- The paper lacks sufficient details about the pre-fusion transformer architecture and implementation, which is crucial for reproduction and understanding the method's effectiveness.

3. Architectural Design and Claims:
- The paper's title and main claim about "one vision token" is potentially misleading. The implementation still requires processing full projected vision tokens in the pre-fusion transformer, suggesting that the computational efficiency gains are not solely from token reduction.
- The ablation studies (lines 446-457) emphasize the importance of the pre-fusion module, raising questions about the actual contribution of the token compressor. The paper should explore the effectiveness of using only the pre-fusion module without compression.

4. Computational Analysis:
- The paper lacks a detailed analysis of the computational overhead introduced by the fusion module, particularly regarding the processing of N^2×5 vision tokens plus l_q text tokens.
- The training costs during the instruction tuning stage, including memory and FLOPs comparisons with LLaVA, are not adequately addressed, making it difficult to assess the true efficiency gains during training.

5. Logic:
- The presentation of "Modality Pre-fusion" and "Vision Token Compression" would benefit from a chronological reorganization to better reflect the processing pipeline (image encoder -> vision token compression -> modality pre-fusion).

questions:
See "Weakness". 

Additional question:

1. Entropy Visualization Clarification:
- Could the authors explain the apparent contradiction between the claim of decreased attention to vision tokens in the latter layers and the high entropy values shown in the visualization?
- What causes high entropy in vision-related attention weights in the latter layers despite lower attention weights?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730920336732

thread_type:
official_review

<thread_object>
creator: reviewer_rv2y

<content>
summary:
LLaVA-Mini proposes a compute and memory efficient Multi-modal by compressing the vision tokens that is passed to the LLM. The work achieves this by introducing a technique of modality pre-fusion where visual representation is fused into the text tokens and compression of vision tokens using cross-attention with learnable compression queries. LLaVA-Mini achieves comparable performance to existing models like LLaVA-v1.5 across several benchmarks, while significantly improving efficiency in terms of FLOPs, latency, and memory usage.

strengths:
- The finding that vision tokens are primarily crucial in early layers is key to the design of the efficient early fusion architecture, a significant contribution to reducing computational overhead in LLMs.
- Overall, the paper is well written and easy to follow. The methods, concepts are well explained and tables/ figures convey key findings. The qualitative visualization examples such as the cross-attention in compression module helps provide intuition. 
-   LLaVA-Mini achieves significant performance, reducing FLOPs by 77% and VRAM usage by more than 99% per image compared to LLaVA-v1.5, without compromising much in terms of model accuracy. This efficiency could make LLaVA-Mini highly suitable for edge devices, making it as a practical solution for real-time multimodal interactions.
- The authors provide exhaustive ablations experiments on number of pre-fusion layers/ vision tokens etc.

weaknesses:
- The paper does not explain the complexity of plugging this compression method to existing multi-modal pipelines. The lack of open source implementation of the code might hinder reproducibility.
- While the paper highlights successful qualitative examples, it would be valuable if it could also discuss any failure cases encountered with the compression technique, along with potential solutions for addressing them.
- It would be great to understand how model efficiency scales across different hardware platforms.

questions:
- Have the authors done any ablation on using SOTA token pooling methods for the vision token pooling and compare against them (other than average pooling).
- Can the authors highlight results on more vision centric benchmarks and show the impact of different number of vision tokens on these e.g., CV-Bench [1]





[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
</content>
</thread_object>

</thread>
<thread>
creation date:
1732452141888

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks all reviewers for their valuable comments!

comment:
We sincerely thank the reviewers for their recognition of our work and for providing thoughtful and constructive comments that have significantly contributed to improving our work. Based on the reviewers’ suggestions, we have carefully revised and strengthened the paper.

Since the public discussion phase nears its conclusion, if the reviewers have any additional questions, we would be delighted to address them promptly.

Once again, we are deeply grateful for the reviewers’ insightful comments and support.
</content>
</thread_object>

</thread>
<thread>
creation date:
1741158390893

thread_type:
public_comment

<thread_object>
creator: ~liang_chen10

<content>
title:
Reporting Academic Plagiarism in an Accepted ICLR 2025 Paper (LlavaMini, Submission Number: 13752)

comment:
Dear ICLR 2025 Program Chair, AC, Reviewers of LlavaMini

I am writing to report a serious case of academic plagiarism in a paper accepted for ICLR 2025. The paper titled "LlavaMini" has substantially plagiarized my work "FastV, https://arxiv.org/abs/2403.06764" which was published at ECCV 2024.

Upon thorough examination of the LlavaMini paper, I have identified clear instances where my methodologies, technical approaches, and results have been appropriated without proper attribution. LlavaMini even does not cite FastV given so many similarities.  Most notably:

As documented in a public GitHub issue (https://github.com/ictnlp/LLaVA-Mini/issues/14#issuecomment-2692746529), Figure 4 in LlavaMini is strikingly similar to Figure 4 in FastV, featuring identical token nomenclature and even the same color scheme. The resemblance is so precise that it strongly suggests use of our codebase. What's particularly concerning is that LlavaMini presents these elements—specifically the finding that "Vision Tokens are More Important in Early Layers"—as novel contributions, when this was in fact a key insight published in our work eight months ago.

I find it troubling that all reviewers and the Area Chair failed to identify this plagiarism, especially considering that FastV has garnered over 100 citations in the community since its publication. A paper should not be accepted when one of its main claimed contributions is directly taken from prior published work.

I have attempted to contact the authors of LlavaMini regarding this matter but have not received any direct response. As a researcher who regularly submits to and reviews for ICLR, I believe this issue must be addressed seriously to maintain the integrity and reputation of the conference.

Thank you for your attention to this serious matter. I am available to provide any additional information or evidence required for your investigation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presented a novel way of compressing visual tokens for images and videos in multimodal LLMs. The authors started with an analysis on how the LLMs handle visual tokens in the middle layers and found that most tokens only play important roles at eearly stages. Based on this observation, the authors proposed a new visual token compression algorithms to conduct pre-fusion for visual tokens before feeding them into LLMs. As a result, the model can support down to one visual tokens for LLMs while still retaining a great performance across variou multimodal benchmarks.

The main strength of this work, as pointed out by the reviewers, is that the authors proposed a new way of compressing long sequence of visual tokens into a smaller one for multimodal LLMs. The authors did thorough analysis on how LLMs cope with visual tokens, from which they derived a neat approach to compress the visual tokens by modality pre-fusion. The authors further showed solid improvement of efficiency while reserving the effectiveness. The discussions between authors and reviewers are pretty engaged.

A few weaknesses were pointed out by the reviewers, including the paper writing, missed report of complexity for the fusion module, etc. After the rebuttal, the authors mostly addressed the concerns and revised the submission accordingly. One more concern about this work is whether this work applies for those tasks which require fine-grained information, such as DocVQA, as pointed out by Reviewer eQGe. The ACs believe this part requires more study for the proposed method.

Given that all reviewers gave positive rating to this work, and most of the concerns have been addressed by the authors during rebuttal, the ACs recommend an acceptance to this work.