LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
Authors: Shaolei Zhang
Qingkai Fang
Zhe Yang
Yang Feng
Venue: ICLR 2025 Poster (2025)
Keywords: Large Multimodal Models
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LLMs), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards efficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve a high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens in advance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.

Published as a conference paper at ICLR 2025
SLAVA-MINI. EFFICIENT IMAGE AND VIDEO LARGE MULTIMODAL MODELS WITH ONE VISION TOKEN
Shaolei Zhang Qingkai Fang Zhe Yang! Yang Feng
1.2.+
Key Laboratory of Intelligent Information Processing.
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
KKey Laboratory of AI Safety, Chinese Academy of Sciences
University of Chinese Academy of Sciences, Beijing, China
zhangshaolei 20z iit..itit. .tc.cttccc.ooo. ac.cn,
fengyange ac..iictt...ooo..ommmo ac. cn
ABSTRACT
The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically code visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LL.ss), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards ef- ficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens ad- vance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLAVA-Mini is a unified large multimodal model that support the understanding of images, high-resolution images, and videos in efficient manner. Experiments across 11 image-based and 7 video-based bench- marks demonstrate that LLaVA-Mini outperforms LLa VA-v1.5 with just vision token instead of 576. Efficiency analyses reveal that LLaVA-Ming can reduce FLOPs by 77%. deliver low-latency responses within 40 milliseconds, and pro- cess over 10,000 frames of video on the GPU hardware with 24GB of memory [T]
INTRODUCTION
Large multimodal models (LMMs), such as GPT-4o (OpenAl 2024), equip large language models (LLMSs (OpenAl 2022; 2023) with the ability to understand visual information, exhibiting com- mon trend toward low-latency responses to enable real-time multimodal interactions. Recently, the most widely adopted LMMs (Liu et al., 2023b 2024a Zhu et al. 2024), exemplified by the LLaVA series (Liu et al. 2023b), involves embedding image patches into vision tokens through a vision encoder (Radford et al. 202T) and incorporating them into the LLM's context to facilitate visual information comprehension, leading to strong performance image and video understanding.
However, the substantial computational costs of LMMs present ongoing challenges. Unlike LLMs (Tovrron et al. 2023a Dubey et al. 2024), which only process textual inputs, LMMs must in- corporate large number of additional vision tokens into the LLM's context represent visual information (Liu et al.) 20235), significantly increasing computational complexity. For instance, in the widely used vision encoder CLIP ViT-L/336px, single image is encoded into 24 24 576 vision tokens (Radford et al. 2021), where integrating such a large number of vision tokens into
Corresponding author: Yang Feng.
'Code: httpps://githtii www/iccnl////e hatppp/.laa---iin (CTNiP/11ava-mmn--11 3-
Model: https::/hugginngfacc.

---

Published as a conference paper at ICLR 2025
the context of parameter-heary LLM results in significant computational overhead and higher infer- ence latency. This issue becomes even more pronounced in high-resolution image modeling (which requires more vision tokens per image) (Liu et al. 2024b) or video processing (which involves pro- cessing more images) (Maaz et al. 2024 Lin et al., 2023a). Therefore, developing efficient L.MSs is essential for achieving GPT--o-lite like low-latency multimodal interactions.
The computational demands of LMMs are primarily driven by model scale and the number of tokens in the input context. Existing approaches to improving LMM efficiency typically focus on model downsizing (Chu et al. 2023 2024 Yuan et al. 2024a] Zhou et al. 2024a) or quantization techniques Yuuan et al. 2024b), but often overlook another critical avenue: reducing the number of vision tokens to shorten the imput context. Some token reduction methods rely on predefined rules to reduce the number of tokens output by the vision encoder (Bolya et al., 2023, Shang et al., 2024 Li et al.)
2024e] Ye et al. 2024d] Hu et al.) 2024), which leads to the loss of visual information and inevitably results in performance degradation (Wang al., [024 Fan et al. 2024).
In this paper, we aim to develop efficient LMMs by minimizing the number of vision tokens while maintaining comparable performance. To this end, we begin by exploring a foundational question: How does the LMM (particularly the LLaVA architecture) understand vision tokens?
LLaVA -Mini
LLaVA -v
FLOIS: 77.1% 1% reduction
66
Speed: 2.92x faster
VIAA//maaee: 360MB - -06MB
MMBench Accuracy (%)
Latenoy 11 3ms - 40ms
64-
62
Through layer-wise analysis (refer SecB), we observe that the importance of vision tokens changes across different layers of LLM. In the
60
early layers, vision tokens play a crucial role,
58-
Image LMMs
LLaWi-v1.5
receiving considerable attention from the fol- lowing text tokens (e.g., user input instructions and responses). However, as the layers deepen, the attention devoted to vision tokens decreases sharply, with most attention shifting towards the input instructions. Notably, even when we en- tirely remove vision tokens in some later layers, LMM keeps certain visual understanding capa- bilities. This finding suggests that vision tokens more critical in early layers, where text to- kens fuse visual information from vision tokens.
56-
LLaVk-v1. 1 Ioaken avg)
Image/Videa LMMs
LLatA-rruMerre
54
Video- Video-LLavA LLavA
VeCo-LLaMA
LIaMANID
LLaWA PakenPaceee
LLaVA-Mini
MOT-LLawe
52
10 #Vision Tokens/lmage
100
576
Figure 1: LLaVA-Mini achieves comparable performance to LLaVA-v1.5 using only I vision token instead of 576, yielding efficient computa- tion, lower latency, and reduced VRAM usage.
Based on this finding, if the fusion process can be shifted from the early layers of LLM to perform before LLM, we can significantly reduce the number of vision tokens fed into the LLM without sacrificing performance. Along with this idea, we propose LLaVA-Mini, an efficient and high-quality LMM with minimal vision tokens. LLa LLM fuse visual information into the instruction text in advance, and employs compression module to highly compress the vision tokens before inputting them into LLM, thereby enhancing efficiency while preserving high-quality visual understanding. Under extreme settings, V.--Mni requires only one vision token per image fed into L.MM backbone, offering significant advantages in inference time and memory consumption for high-resolution image and long video processing.
introduces a modality fu--inson module before
Experiments across wide range of 11 image-based and 7 video-based understanding benchmarks show that VA-Minn achieves performance comparable to VA- (Liu et al, 2023b) while using only 1 vision token instead of 576 (compression rate of 0. 0.17%). With minimal vision tokens, LLaVA-Minn offers substantial benefits in terms of computational efficiency (77% FLOPs reduction) and lowering GPU memory usage (360 MB - 0.6 MB per image), shown in Figure [ As a result, LLaVA-Mini decreases inference latency of image understanding from 100 ms to 40 ms and also enables the processing of long videos exceeding 10,000 frames (over 3 hours) on an NVIDIA RTX 3090 with 24GB of memory, paving the way for low-latency multimodal interactions.
RELATED WORK
As Large multimodal models (LMMs) are increasingly deployed in real-time applications (OpenAl 2024), enhancing their efficiency has become critical concern. Recent efforts focus either
2

---

Published as a conference paper at ICLR 2025
reducing the model size the number of tokens that fed into LMM. To reduce LMM's s model size, previous methods directly replace the LLM backbone with a smaller one (Chu et al. 2023 2024
Yuan et al. 2024a Zhou et al., 2024a), while directly reducing the parameter scale impact the
LLM backbone's capabilities, resulting in performance declines visual tasks (Shang et al 2024
Another efficiency determinant for LMMs is the context length provided to the LLM backbone, including vision and text tokens. In practice, the number of vision tokens can be substantial, partic- ularly when processing high-resolution images and videos. For image-based LMMs, token merging
(Bolya et al. 2023), PruMerge (Shang et al. 2024), and TokenPacker (Li et al. (024e) aggregate vision tokens based on similarity. Qwen- VL (Bai et al. 2023) and MQT-LLaV. VA (Hu et al., 2024)
utilize Q-former (Li et al.] 2023a) to compress vision tokens into fixed length. However, directly
reducing vision tokens inevitably results in the loss of visual information (Fan et al. 2024).
For video-based LMMs. Video-ChatGPT (Maaz et al.) 2024), VideoChat (Li et al. 2024c), Video- LLaVA (Lin et al., 2023a), and Video-LLaMA (Zhang et al. 2023). select a fixed number frames from videos of varying lengths. MovieChat (Song et al.) 2024a) applies memory techniques to condense videos into a fixed-length representation. Such frame selection or merging methods may lose some key frames or misunderstand the temporal information of the video (Zhou et al. 2024b).
Previous methods have primarily focused on token reduction on the vision encoder. LLLVA-Minn takes this step further by exploring how vision tokens and text tokens interact within the LLM backbone, and accordingly introduces a modality pre-fusion module, enabling an extreme compres- sion of vision tokens (1 vision token fed into LLM) while achieving comparable performance.
3 HOW DOES LLAVA UNDERSTAND VISION TOKENS?
To compress visual tokens while preserving visual understanding, we sought to figure out how LMMs understand visual tokens. Given the complexity of this issue, our preliminary analysis concentrated on the LLaVA architecture (Liu et al. 2023b), focusing on the role of visual tokens (particularly their quantity) in LMMs from an attention-based perspective (Xiao al. 2024
3.1 LLAVA ARCHITECTURE
LLaVA (Large Language and Vision Assistant) (Liu et al. 2023b) is an advanced multimodal ar- chitecture that integrates vision and language processing capabilittes. Building upon vision Trans- formers (ViT) (Dosovitskiy et al., (202]) for visual inputs and LLMs for text, LLaVA can generate language response X based on the given language instruction X and visual inputs X
Typically, a pre-trained CLIP ViT-L/14 (Radford et al. 2021) and a projection layer are employed to encode the visual inputs X" into vision tokens (i.e., continuous representations) HV. Then, vision tokens HY and language instruction's 's embedding H are fed into an LLM,, such as Vicuna (Chiang cet al. 2023) or Mistral, to generate the response X In practice, vision tokens are often inserted into the middle of the language instruction, so the inputs of LLM can be formally represented as:
H H. H HY, H k+1 H²)
(1)
where I, and ly denote the lengths of the vision tokens and language instruction, respectively. For instance, in LLaVA-v1.5, the system prompts are positioned before the image (i.e., Hi,... .2,....,1.. ,H),, while the user inputs follow the image (i.e., H ;H,,, (Liu et al. 2023b).
3.2 PRELIMINARY ANALYSES
We begin by analyzing the significance of visual tokens LMMs to guide the strategies for com- pressing vision tokens. Specifically, we evaluate the importance of visual tokens at each layer of LMMs from an attention-based perspective. Our analysis encompasses several LMMs, including 1.VVA-1..5-Viumm 7B. LLa LA-v1.-1.5 --.unnaaa LLaVA-v1. 1...-v....Msttaa----
and LLaVA-NeXT. Vicuna-7B (Liu et al. 20236 2024b), to identify common characteristics across models of varying sizes and training datasets. Appendix A gives the formal expression of the preliminary analyses.
Vision Tokens are More Important in Early Layers To find out which layers in LMM the vision tokens play more important role, measure the attention weights assigned to different token
3

---

Published as a conference paper at ICLR 2025
Alter
(a) LLaVA-vI. Vicuuna--
(b) LL V3euna-13B 13
(c) LLaVA-v1G-MMsttal--
(d) LLAVA-NoXT-Vicun-788
Figure 2: Layer-wise variation of attention weights assigned to different types of tokens (including
instruction, vision, and response) in LMMs. "A-BB" means the atfention weights from A to B.
Agtrro Ennoey
Apeera Trenrry
Entopy
Rapprraa Batracien
Tesparen FatuCCa
Layer
Uie
(a) LLaVA-v1.5- V.B
LLaVA-vI -.Vi-un.--
(c)
LL
LLAVA-NeTT-Viemm--
Figure 3: Attention entropy assigned to different types of tokens across different layers in LMMs.
types (including instruction, vision, and response) at each layer. As shown in Figure Visual tokens receive more attention in the earlier layers, but this attention sharply decreases in the deeper layers, with over 80% of the attention being directed towards instruction tokens. This finding is consistent with the previous conclusion (Chen al. 2024). This change in attention suggests that vision tokens play central role in the early layers, with the instruction tokens seeking relevant visual information from vision tokens through attention mechanisms. the later layers, the model relies more on instructions that have already fused the visual data to generate responses.
Most Vision Tokens are Focused Early Layers To further assess the importance of individual visual tokens, we calculate the entropy of the attention distribution at each layer. As shown in Figure 3 we find that the entropy of attention toward visual tokens is much higher in the earlier layers, indicating that most visual tokens are evenly attended to in the early layers.
Layer
Layer
Layer 10
Layer
Layer 30
Figure 4: Attention visualization at different layers in LLaVA-v1.5 (color bar: logarithmic scale).
To intuitively illustrate the layer-wise variations in the importance of visual tokens, Figure[avisualizee the attention distribution across each layer of LLaVA-v1.55 Almost all visual tokens receive broader atten- tion in the early layers, while only some visual tokens are focused in the later layers. These observations suggest that all visual tokens are crucial the early layers, and reducing their quantity inevitably results in loss of visual information. This explains why previous methods of direct token reduction will compromise visual understand-
1241 1500 8:12
ing capabilities (Shang et al. 2024 Ye al. 2024d
Hu al. 2024
To further substantiate our finding that visual tokens are particularly critical in the early layers, we evaluated the visual understanding abil- ity of LMMs when visual tokens were dropped at different layers. Specifically, we measured the performance of VLaV on the GQA (Hudson & Manning. 2019) and MMBench (Liu et al.) (024c). with visual tokens being dropped at layers 1-4, 5-8,
Figure 5: Performance of LLaVA-v1.5 when remov- ing all vision tokens in var- ious layers of LMM.
29-32, respectively. As shown in Figure removing visual to- kens in the early layers leads complete loss of visual understanding ability, while removing tokens in the higher layers has a minimal effect, with the model retaining much of its original per- formance. In conclusion, our analyses and ablation study reveal that vision tokens play crucial role the early layers of LLaVA, where text tokens fuse visual information from the vision tokens this stage. This insight inform strategies for compressing vision tokens.
4

---

Published as a conference paper at ICLR 2025
MINI
LLaVA-Mini
Langnage Response
(a) Compression
II 576 tnkens
attention A
Large Language Model
fast inference vision token milliseconds, (.6B/image)
C learnablee
queries
cinstructino::
<Responsoo:
11't
token / image
IP
Cross-Attertion
Compression
Modality Pre-fusion
position ancoding
1I
ANNAA 00
(b) Modality Pre-fusion
H'f S76 lokens! image
H
Projection Vision Encoder
Embedding
Transformer Blocks
X
What smoge/ridee?
a
Language Instruction
Visual Inputs
Black
vision H
text H
Figure 6: Architecture of LLaVA-Mini, Left: LLaVA-Mini represents each image with vision token. Right: Detailed view of the proposed query-based compression and modality pre-fusion.
4 LLAVA-MINI
We introduce LLaVA-Mini, an efficient large multimodal model with minimal vision tokens. Like previous work, LLaVM-Mini uses a vision encoder to encode an image into several vision tokens. To enhance the efficiency, LLaVA-Mini significantly reduces the number of vision tokens fed into LLM backbone through a compression module. To retain visual information during compression, based previous findings that vision tokens play crucial role in the early layers for fusing visual information, LLaVA-Mini introduces a modality pre-fusion module before the LLM backbone to follows.
fuse the visual information into the text tokens. The details of LLaVA-Min
4.1 ARCHITECTURE
The architecture of LLaVA- is illustrated in Figure 6 For the visual inputs X', pre-trainn pre-trained CLIP vision encoder (Radford al.) (02I)) is employed extract visual features from each image. These features are then mapped into the word embedding space via projection layer, producing vision tokens H RVV xd,, where 2 is the number of vision tokens and d, is the LLM's M's embed- ding dimension. For the language instruction X LLM's embedding layer used to generate text token representations H4 E XR,, where l is the number of text tokens.
Vision Token Compression To enhance the efficiency of LMMs, LLaVA-Mini reduces the num- ber of vision tokens fed into the LLM backbone by utilizing a query-based compression module. To learn compression of the vision tokens, LLaVA-Mini introduces x C' learnable compression queries Q" These queries interact with all vision tokens H' through cross-attention (Li et al. 2023a Ye et al. 2024a), selectively extracting the important visual information to produce x C compressed vision tokens A E x To preserve the spatial information in the image dur- ing compression, we introduce a 2D sinusoidal positional encoding PE(-) (He et al. 2021] on the learnalle queries and original vision tokens. Formally, the compression
be expressed
f A H" where A = Softmax (QQ + PE(Q") (H" (H' PE(H")))
(2)
where E 1 is the similarity and A are C C compressed vision tokens.
Modality Pre-fusion The compression of vision tokens inevitably results in some loss of visual information. To retain as much visual information during compression as possible, LLaVA-Min introduces modality pre-fusion before the LLM backbone, enabling text tokens fuse relevant visual information from all vision tokens in advance. Based on our previous observations, where this fusion stage occurs implicitly within the early layers of the LLM, the modality pre-fusion module f(-) consists of N Fussion Transformer blocks Vaswanni et al. 2017). where each Transformer block shares the same structure and hyperparameters with L.M backbone. Vision tokens H and text
5

---

Published as a conference paper at ICLR 2025
tokens H are concatenated and fed into the pre-fusion module, and the outputs corresponding to the text tokens are then extracted as fusion tokens, expressed as:
H f (Concat (H", H))) 1-4y :
where Hy R xd are fusion tokens of text representations with related visual information.
Finally, the compressed vision tokens H and fusion tokens H of text representations with related visual information (C²² + 1 tokens total) are fed to LLM together generate the response.
4.2 HIGH-RESOLUTION IMAGE AND VIDEO MODELING
LLa uses minimal vision tokens to represent visual information, making it possible to han- dle high-resolution images and videos much more efficiently.
High-Resolution Image The resolution of LMM is typically determined by the vision encoder, such as CLIP's ViT-L, which encodes resolution of 336*336 pixels. To perceive images at higher resolution, we divide each image into four sub-images by splitting it horizontally and verti- cally into two parts (Liu et al. 2024b). Each these sub-images processed by the vision encoder and projection individually, yielding N2 x vision tokens with a high resolution of 672*672 pixels. The proposed compression module is then employed to reduce these N2 x 4 vision tokens into C compressed vision tokens A The modality pre-fusion module takes the sub-images (N2 x vision tokens), the original image (N2 vision tokens), and the language instruction (ly text tokens) as inputs, and then generates lg fusion tokens Ho with richer global and local visual information. Finally, the number of tokens input the LLM is C2 + bgg Note that when handling resouutsot images, C' is set slightly higher than in standard-resoluse
settings to preserve more details.
When handling videos, LMMs often extract multiple frames from the video (Li et al. 2023b), which incurs sigmificant computational costs. For instance, in the case of LLaVA-v1.5, ex- tracting frames a rate of I frame per second (fps) from an 8-second video results 576 4608 vision tokens, leading to substantial VRAM usage. LLLVA-Minni can represent each image with min- imal vision tokens, providing significant advantage in processing long videos. For a video consist- ing of M frames, LLaVA-Min processes each frame individually, generating C22 vision tokens and l fusion tokens per frame. C2 vision tokens from each of M frames are sequentially concatenated to yield total of M x C2 vision tokens, i.e., H. Then, lg fusion tokens corresponding to M frames aggregated through pooling operation to generate the video's 's fusion tokens f14 As result, the number of tokens fed to the LLM is reduced from MN2 + lg to MC² + lg for a video M frames.
Video
4.3 TRAINING
VA-Mini follows the same training process LLaVA, consisting of two stages.
Stage 1: Vision-Language Pretraining In this stage, compression and modality pre-fusion mod-
ules are not yet applied (i.e., the 2 vision tokens remain unchanged). LLaVA-Miní learns to align vision and language representations using visual caption data. The training focuses solely on the projection module while the vision encoder and LLM remain frozen (Liu al. 2023b).
Stage 2: Instruction Tuning In this stage, LLaVA-Mini is trained to perform various visual tasks
based on minimal vision tokens, using instruction data. Compression and modality pre-fusion are introduced to LLa VA-Mini, and all modules except the frozen vision encoder (i.e., the projection, compression, modality fue-iu and LLM backbone) are trained in an end-to-end manner.
EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Benchmarks We evaluate LLaVA-Mini on image and video understanding tasks. Experiments are conducted on 11 image benchmarks and 7 video benchmarks. Refer to Appendix [o details.
Baselines LLaVA-Mini is an image/video LMM, so we compare with several advanced image-
based and video-based LMMss. Detailed description of baselines refer to Appendix D

---

Published as a conference paper at ICLR 2025
Table 1: Performance on 11 image-based benchmarks. 'Res.' is resolution and *Viiion Tokens' is the number of vision tokens fed to LLM backbone. *** indicates that involving extra training data.
Vission Tokens
VOA
SaQA
VQA'
POPE
MME
SEED
MM-
(%)
Vicuno-1
19.6
46.4
38.1
22.4
InstruciBLIP
Vicuna 78
224
89.2
34.5
00.3
50.1
36,0
53.4
50.9
26.2
IDEFICS-9B
LLaMA-7B
2241
64
50.9
38.4
35.5
25.9
48.2
DEFFCCS800
LLaMA-4S8
36.0
Owen-VLL
Qwen-7BB
25.2
56.3
Owen-VI.-Chal
Qwas-78
44H
256
78.2
57.5
38.9
68.2
G1.5
1487.5
60L5
58.2
SPHINX
LLaMA-138
224
289
78.1
62.6
29,9
099.3
51.6
S0.7
1476,1
66.9
56.2
73.5
36.0
56,0
SPHINX-IL
LLaMA- 13B
2390
80.7
63.1
44.9
70.6
61.2
87.2
1470.6
65.9
57.9
76.9
40.2
59.0
PPLUGGOOW
LLaMA-7B
54.5
57.8
36.2
Video-LLaYA
Vicuno-78 7M
214
256
74.7
60.3
48.1
66.4
51.8
84.4
60.9
73.1
32.0
LLaYA-w15
Vicuna 7B
336
576
75.5
62.0
50,0
66.8
58.2
85.9
1510.7
64.3
58.6
63.4
30.5
56.3
LMMs with Jewer visiom tokens
MOT-LLLVA
Vicuno 7B
48.5
74..5
41.7
19.5
MOT-LLLVA
Vicunn-78
336
36
73.7
58.8
$1.0
66.8
81.9
1416.1
63.4
59.6
27.8
MOT-LLLYA
Vicuno-78 78
336
256
76.8
61.6
53.1
67.66
54.4
1434,5
64.3
64.6
29.8
PruMerge
Vieuna 2M
68.5
76.3
1350,3
60.9
PraMergess
Vicuno-78
84.0
1462.4
LLDMA-VID
Vicuno-788
3366
55.5
6888
49.0
83.1
VoCo-LLaMA
Vicuna 28
236
72.3
57.0
05.4
81.4
1323.3
58.8
53.7
TokenPacker
Vicuna-78
50.2
86.2
29,6
MaaAA-Min
Vicuno-78
336
77.6
60.9
56.2
70.4
57.0
54.4
1466.0
65.6
58.5
68.9
36.6
57.9
A cumpare LiaVi--1..
0.17%
-0.9
-1.1
46.1
43.6
-1.3
-1.5
-44.7
+1.3
-0.1
45.5
-6.1
+1.6
Vicuna
78.9
58.5
85.3
67.5
60.2
33,9
A compure
LLaVA v1.5
11.1%
+0.4
0.2
+8.5
+2.9
+0.9
0.6
-33.9
+3.2
+1.6
+3.9
+3.4
+2.4
MLaVA-Mimi
LLaMA-3.I-
(Imaee Video)
SB Instract
336
57.4
83.1
85.3
1522.7
63.0
37.2
Table 2: Performance on video-based open-ended generative benchmarks. We report accuracy (%) for question-answert and scores (1-5, higher is better) for question-answer and generative perfor- mance. Results marked with bold and underlined indicate the best and second best, respectively.
#Visco
Video-based Question-Ausee
Video-based Generative Performance
Methods
ATrames
Tokens Frume
Acc.
MSVD-QA
MSRVTT-QA
Score
Aetivityet-QAA
Acc.
Scare
Correctness Detall Contextual Temporal Consistency
LLaMA Adapter
54.9
2.7
2.32
1.98
2.15
2.19
VideoClut
16
32
56.3
2.8
45.0
2,5
26.5
2,2
2:23
2.50
2.53
1.94
2.24
2.30
Vidoo-LLaMA
16
64
51.6
2,5
20.6
1.8
12.4
1.1
1.6
2.18
2.16
1.82
1.79
1.99
Vidco-ChatGPP
100
64.9
2.7
2.37
BT-Adapter
100
~266
67.5
3.2
45.7
3.2
2.68
2.69
3:27
2.34
2.46
2.69
MovieChat
2048
32
75.2
3.8
$2.7
2.6
45.7
3.4
2.76
2.93
3,01
2.24
2.42
2,65
LIAMA-VID
18ps
2
69.7
3.7
57.7
3.2
47.4
3.3
2.96
3,00
3.53
2.46
2.51
2,88
Vidco-LLaVA
256
70.7
50.2
2.45
LLaVA-Mial
Mpps
20.9
4.0
99.5
3,6
$3.5
3.5
2.97
2.98
3.61
2.48
2.67
2.94
Configuration For a fair comparison, LL VA-Miii employs the same configurations LL- v1.5 (Liu et al. 2023b), using the CLIP ViT-L/336px (Radford et al. 202I) as the vision encoder and Vicuna-vI.S... 1.57B (Chiang et al. 2023 as the LLM backbone. The compressed hyperparameter is is set to 1, meaning vision tokens are compressed to one token. The number of modality pre- fusion layers N Nusion is set 4. LLa
uses the same training data as LLa (Liu jet al., 20236), using 558K caption data for pretraining and 665K instruction data for instruction tuning. The high-resolution version with 672*672 pixels (refer to 4.2 is denoted LLaVA- Mini-HDD To capture more visual details, the compressed hyperparameter C. of L.LaVA-Mini-HD is to i.e., compressing 64 vision tokens. For vidco processing, LLaVA-Mini extracts frame per second (1 fps) from the video and sets C' = 1 to represent each frame with one vision token.
To further explore the potential of LLaVA-Mini, we introduce a variant that uses the CLIP ViT- L/336px (Radford et al. (021)) as vision encoder and the advanced LLaMA-3.1-88 11--3.188B-Intrruee (Dueey al., 2024) as LLM backbone. During instruction tuning. we combine 665K image instruction data from LLaVA (Liu et al., (2033)). 100K video instruction data from Tidoo-ChatGPT (Maaz al. 2024), and part of open-source data (Li et al. 2024a), resulting in million training samples. I.a. VA-Mini is trained using 8 NVIDIA A800 GPUs Training details are provided in Appendix B
5.2 MAIN RESULTS
Image-based Evaluation We compare LLaVA-Mimi with LLaVA-v1.5 across 11 benchmarks to thoroughly assess the performance of LLAVVAMinni with minimal vision tokens. The results
7

---

Published as a conference paper at ICLR 2025
Table 3: Performance on MVBench (accuracy). Detailed scores are reported in Appendix H
Methods
Action
Object
Position
Scene
Count
Attribute
Pose
Character
Cognition
Avg.
MPL.G-Owl
28.4
33.0
25.0
29.0
29.3
42.0
24.0
31.0
25.3
29.7
Video-ChatGPT
32.1
40.7
21.5
31,0
28.0
44,0
29.0
33.0
30,3
32.7
Video-LLaMA
28.3
32.5
40.0
29.3
VideoCbal
38.0
41.2
26.3
48.5
27.8
44.3
26.5
41.0
27.7
35.5
LLaMA-VID
36.5
37.5
34.0
Video-LLaVA
48.0
46.5
27.8
84.5
35.5
45.8
34.0
42.5
34,2
43.1
LLaVA-Mini
52.1
43.2
31.8
85.5
37,5
44.5
29.5
52.0
35.0
44.5
Table 4: Results on MLVU (aceuracy) of long video under- standing. Evaluation includes Topic Reasoning (TR), Anomaly Recognition (AR), Needle QA (NQA), Ego Reasoning (ER), Plot QA (PQA), Action Order (AO), and Action Count (AC).
Table 5:
Results on
EgoSchema a long-form video benchmark (~ minutes) for first-perre first-person
(accuracy).
view temporal reasoning.
Holistic
Single Detail
Multi Detail
Methods
#Frames
Avg.
IR
AR
NQJ
ER
PQA
AO
AC
Methods
#Frames EguSchema
Avg. Video Duration (minute)
7
10
13
11
Random
20
Max Video Duration (minute)
20
543
139
20
13
137
130
143
mPLUG.OWl
16
31.1
Video-ChatGPP
100
26.9
24.0
40.3
42.0
29.9
25.1
31.1
31.3
Intern
16
32.1
MovieChat
2048
29.5
25.0
24.2
24.7
25.8
28.6
22.8
25.8
Viato-Ch
100
36.2
Movie-LLM
Ifps
30.0
29.6
24.7
24.1
20.5
24.8
26.1
VideoChat
16
42.2
TimeChat
96
23.1
27.0
24.5
28.4
25.8
24.7
32.0
30.9
TimeChat
96
33.0
LLaMA-VID
16ps
50.8
32.7
23.9
27.8
33,2
LLaMA-VID
1ps
38.5
MA-LMM
1000
51.9
35.5
43.1
38.9
35.8
25.1
24.1
36.4
Video-LLLaV
38.4
LLaVA-Mini
1pps
76.0
50.0
44.5
37.5
49.0
24.3
18.4
42.8
LLaVA-Mini
1ips 51,2
reported in Table ! where LLaVA-Mini achieves performance comparable to LLaVA-v1.5 while using only 1 vision token instead of 576. Previous efficient LMMs with fewer vision tokens often merged similar tokens directly after the vision encoder (Shang et af.) 2024; Ye et al. 2024d), re- sulting in significant loss of visual information and negatively impacting visual understanding of LMMs. For instance, LLaMA-VID. VoCo-LLaVA, and MQT-LLaVA, which reduce vision tokens 1-2 tokens, lead 5% performance drop on average. contrast, LLa VA-Mini employs modal- ity pre-fusion to integrate visual information into text tokens before compressing the vision tokens, achieving performance comparable to LLaVA-v1.5 more, LLiVA-Mini-H shows an average performance improvement of 2.4% over LLa to high-resolution image modeling. Note that VA-MA-- has a computational load 8.13 TFLOPs, which remains lower than ALLVAA-1..5's 8.55 TFLOPs. More efficiency analyses refer to Sec5.3 Overall, LLaVA-Mini preserves strong visual understanding capabilities while compressing vision tokens, enhancing the usability of efficient LMMs in visual scenarios.
token compression rate of 0.17%. Further-
due
Video-based Evaluation We compare LLaVAs VA-Mini with advanced video LMMs on widely used video-based benchmarks. The results are reported in Table 2 and where LLa VA-Mini demon- strates superior overall performance. Video LMMs such as VidcoChat (Li et al. 2024c), Video- LLaVA (Lin al. 2023a), and Video-LLaMM (Maaz et al., 2024) much more vision tokens to represent each frame, and thereby can extract only 8-16 frames from a video due to the limited con- text length of LLMs, which may result in the loss of visual information in some frames. In contrast, LLaVA-Mini uses one vision token to represent each image and accordingly can extract frames from the video a rate of frame per second, thus performing better on video understanding.
Extrapolation to Long Videos Furthermore, we compare VA-Minn with advanced long-video LMMs (can process video over 100 frames) on long-form video benchmarks. MLVU (Zhou al.) 2024b) and EgoSchema (Mangalam et al.) 2023). Note that LLaVA-Minn is trained only on Video- ChatGPT instruction data and has not been exposed to any long video data, so performance on long videos is entirely derived from the length extrapolation capabilities of its framework (Press) LLaVA- exhibits significant advantages long video
ct al. 2022). As shown in Table understanding, By representing each frame as one token, LLaVA-Mini facilitates straightforward extension to longer videos during inference. In particular, LLaVA-Mini is only trained on videos shorter than I minute (< 60 frames), and performs well MLVU's long-form video, which encom- passes videos over hours (> 7200 frames) during inference. Overall, with one vision token per frame, I.La LLLVA-MMnn -ini demonstrates high-quality video understanding in efficient manner.
8

---

Published as a conference paper at ICLR 2025
FLOPS and infretcc Latency onsooucion 3369336)
FLOPS and Inforence Litency (tesikiion 67249722
VIAM Usage LOTg VISCO
Leary
Valer
Chur
L
VIEE" COT
Space
Naftamm
X0 200 MiDhumaaaion
140
120
5005
Figure 7: FLOPs and latency of LLa LLaVA-Mini.
Figure 8: FLOPs and latency of LLaVA-Mini-HD.
Figure 9: VRAM usage (3- hour video) of LLaVLA-MMin
Table 6: Performance of LLaVA-Mini with different numbers of modality fre-fusion layers N fusion-
Table 7: Performance of LLaVA-Mini with various vision tokens.
Pre-fusion
#Vision Tokens
FLOPs
Performance
VVision Tokens
Performance
Methods
Methods
Res.
#Layers
(T)
VQA
GQA
MMB
VQA
GQA
MMB
S76
8.55
78.5
62.0
64.3
M.VA-1155
336
576
78.5
62.0
64.3
1
0.96
72.4
$4.2
57.7
336
77.6
60.9
65.6
LLaVA-Mini
(w/o
16
1.16
74.1
55.4
59.2
336
77.7
60.9
66.7
pre-fusion)
64
1.79
75.3
56.7
62.1
336
16
78.1
61.3
66.6
144
2.85
76.9
58.9
64.9
LLaVA-Mini
336
78.5
61.6
67.5
VLaVi-Mini
1
1.21
74.8
55.5
60.4
672
16
78.5
61.5
67.4
(w/
1.46
76.0
672
78.9
pre-fusion)
1.81
76.9
59.1
64.9
672
144
79.3
62,3
67.9
1
1.96
77.6
60.9
65.6
672
576
80.0
62.9
68.1
5.3 EFFICIENCY
With the performance comparable to LLaVA-11.5, we further explore the computational efficiency offered by LLaVA-Ming Figures 7 illustrate the advantages of LLaVA-Mini in terms of com- putational load, inference latency, and memory usage, where FLOPs are calculated by calflops Ye 2023). and latency is tested on the A100 without any engineering acceleration techniques.
FLOPs and Inference Latency As shown in Figure LLaVA-Mini significantly reduces com- putational load by 77% compared to LLaVA-v1.5, achieving a speedup of 2.9 times. LLaVA- achieves response latency lower than 40 ms, which is crucial for developing low-latency real-time LMMs. As shown in Figure X when modeling at high resolutions, the efficiency advantages of VA-Min are even more pronounced, yielding 82% FLOPs reduction and 3.76 times speedup.
Memory Usage Memory consumption poses another challenge for LMMs, particularly in video processing. Figure 9 demonstrates the memory requirements of LMMs when processing videos of varying lengths. In previous methods, each image requires approximately 200-358 MB memory (Liu et al. 2023b Lin et al.] 2203a). limiting them to handle only about 100 frames on a 40GB GPU. In contrast, LLa with one vision token requires just 0.6 MB per image, enabling it to theoretically support video processing of over 10,000 frames on RTX 3090 with 24 GB of memory.
6 ANALYSES
6.1
SUPERIORITY OF MODALITY PRE-FUSION
The proposed modality pre-fusion is central to LLaVA-Mini, as it integrates visual information into text tokens in advance, facilitating extreme compression of vision tokens. To investigate the ef- fects of modality pre-fusion, we conduct an ablation study in Table 6 Without pre-fusion, token compression results in a performance drop of around 5%, even with 144 vision tokens retained, the performance of LMMs falls short of LLa methods often exhibit poor performance Ye et al., 2024d) or can only achieve compression rate of over 40% (Shang et al. 2024). Notably, under the same FLOPs, increasing the number of fre-fusion layers yields greater benefits than increasing the number of compression vision tokens. This sup-
This also explains why previous token merging
9

---

Published as a conference paper at ICLR 2025
Image:
Instruction:
How much Ray-Ban Mcta Wayfarer?
Instruction: What activity does the video show?
60s video
LLaVA-Mini: The Ray-Ran Meta Wayfurer is priced at S299 USD.
The video shows group men suits and sports uniforms standing a stage, holding a trophy, and colebrating their victory.
I.LaVA-Vin::
Latency: 18mz, Vistor token
Ray-Ban Meta Wayfarer sunglasses shown the image $299.00.
GPT-4e: The price
Latency: 14s, Vision tokensi 1+60x60
The video shows group of playing soccer on field. They seen celebrating with one another and holding up a trophy.
Video-I.LaVA:
Video-LLaVA: The price of the Ray-Ban Mela Wayfarer is $129.99.
Lotency: 80ns, Vision takens 250
Latency Visien 256-8-2018
LLaVA-w1.5: The Ray-Ban Meta Wayfarer is priced at S199.
Video-ChatGGT The video shows group of men bolding troph and celebrating on a stage L7a, Vision tokeess 256-100-356
Latency: 110me, Vision takens: 576
Figure 10: Case of image understanding.
Figure 11: Case of video understanding.
ports our preliminary analysis, which indicated that vision tokens exhibit varying importance across different layers and vision tokens are more critical in early layers. Investing more computational overhead in earlier stages where vision tokens are more important results in better performance.
6.2 EFFECT OF COMPRESSION
Table 8: Effect of query- -bsed compression.
VA- employs query-based compression achieve a high compression ratio for vision tokens. We
#Vision Tokens
Performance
Compression
FLOPs
compare the performance of query-based compression with direct average pooling in Table 8 Query-based compression can adaptively capture important features in the image while requiring only minimal additional computational cost, demonstrating a significant advan- tage. Appendix gives a visualization of the compression process and more detailed analysis.
VQA
GQA
MMB
Average Pooling Query-based
1.96T
76.1
59.8
64.0
+2.42G
77.6
60.9
65.6
Average Pooling Query-based
2.01T
76.9
60.3
65.1
77.7
66.7
6.3 PERFORMANCE WITH VARIOUS VISION TOKENS
uses I vision token for standard images and 64 for high-resolution images. We explore the potential of LLaVA-Mini when further increasing the number of vision tokens (larger C) in [ The results indicate that as the number of vision tokens increases, LLaVA-Mini continues improve in performance. In particular, LLa outperforms VLaVA-v1.5 when both using 576 vision tokens, demonstrating its effectiveness when computational resources are plentiful.
6.4
CASE STUDY
Figures 10 and present examples of LLaVi-Mini in image and video understanding tasks (refer to Appendix G for more cases). Despite using only one vision token, LLaVA-Mini performs effec- tively in capturing visual details, such as accurately identifying price information (OCR) in website screenshots. For video understanding, Video-LLaVA extracts 8 frames per video, regardless of video duration (Lin et al. 2023a). Training on only frames (sometimes missing key frames) can cause hallucinations (Khattak et al. 2024), encouraging LMM to forge information beyond the extracted frames. For instance, given a celebration scene, Video-LLa VA mistakenly imagines "a group of men playing soccer field' before the celebration. This fixed-length frame extraction is forced com- promise due to the large number vision tokens required per image while LLM's context length limited. In contrast, LLaVA-Mini, utilizing just one vision token per frame, can process videos at fps, resulting in more robust video understanding. Overall, LL- ensures strong visual understanding while enhancing efficiency, making it a practical solution for multimodal interaction.
7 CONCLUSION
We introduce LLa VA-MM,,, an efficient LMM with minimal vision tokens. LLaVA-Mini excels in image and video understanding while exhibiting superiority in computational efficiency, inference latency, and memory usage, facilitating the real-time multimodal interaction with efficient LMMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by National Natural Science Foundation of China (Grant 66.62276260).
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile languaee model for understanding, local- ization, text reading, and beyond, 2023. URL Uttpttss::: / org/ abs/ 2308 12966
Daniel Bolya, Cheng- Chan Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster, 2023. URL http::://axxiv.org////
2210.0320 09461
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.). Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 190-200, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https: 7/aclanthology. wwwptthooooooooooPo.o.eo wwtp:/l-.g//p01/10001111
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//1 //mmsys.org/// wttp:///////o//.///////////////e og/p///222232 -
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16888 2023.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint ar arXiv:2402.03766, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference Learning Representations, 2021. URL https: // //openreview. www.ewewww........ wwwprrrewwwweeeeefrffoeoeee
cbF.TTTTy
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan
11

---

Published as a conference paper at ICLR 2025
Mísra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Hontala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh. Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileamu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Buevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadch Yazdan, Beau James, Ben Maurer, Benjamin Leonbardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic. Brian Gamido. Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Lc, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanaa, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov. Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean -aanistt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthia Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael I.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli,
12

---

Published as a conference paper at ICLR 2025
Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary VeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models, 2024. URL httph:tss////m /atpss://axxi..oo org/ abs/ 2407 21783
Xiaoran Fan, Tao Ji, Changhao Jiang. Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Mousi: Poly-vssual-exper vision-language models, 2024. URL https:
7/arxiv. abt//ab///eee http:/abs/2400.111
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL
2306. 13394
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: Memory-augmenter large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. URL
06377
Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large vision-language models, 2024. URL https://arxiv.
Brg/abs/24055 19315
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real- wol-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), June 2019.
Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan, Muzammal Naseer, Fed- erico Tombari, Fahad Shahbaz Khan, and Salman Khan. How good is my video Imm? com- plex video reasoning and robustness evaluation suite for video-Imms, 2024. URL https:
03690
13

---

Published as a conference paper at ICLR 2025
H Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, et al. Introducing idefics: An open reproduction of ofte-of-the-art visual language model. 2023. URL https///huggingfuce. co/blog/idefics. Accessed, pp. 09-18, 2023.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a. URL hthhtps:
7/ //axiiv.org////c...a.aaoo.....aa wwwpii.org/abb//eee
03326
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-999-1330,, June 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 19730-19742. PMLR, 23-29 Jul 2023a. URL https: petts::/proceediinee http///ooceeengsge///e/ wttp:/g..........//2//22./22
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024c. URL bttps://arxiv..
org/ 22055 06355
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22195-22206, June 2024d.
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal 11m, 2024e. URL https:: //axxvv.. orq/ 2407 02392
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023b. URL https::/ http:is:///avvvv../ wwwplvv..rrggab///3/3333 17043
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large linion-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023c. URL https: http:ss///penreeeee..eeeo
x07JJ0kZZXF
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection, 2023a. URL https://arxxiv.
hog/abs/22 20ab/2311... 10122
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal org/ abs//2111 2bs/2231.07775
large language models, 2023b. URL htth::t::///atr
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34892-34916. Curran Associates, Inc., 2023a. URL htttps::/ hwwpseeedddd........e.eeeeeeeeee neurips cc paper files/paper/2023) Conferencee.cccco pdff
File/ htte:///.22///e222222243421112d0 henpreeeeeeeeeepppeee----eeeeeee
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34892-3444 34916. Curran Associates, Inc., 2023b. URL ott.a//eee///rreeeei.a.ooooooooa wtw.aaarrrr.eeeeepeeeeeeee
fi 1/ Fal 64efr7eeeeee3e1111100eeeerr0
14

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16296-26306 June 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, oct, and world knowledge, January 2024b. URL https://
1lava-vl. www.giithuu.... httpa//www..tttbb..oooooooooo/oe
Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H Li, and Ge Li. One for all: Video conver- sation is feasible without video instruction tuning. arXiv preprint arXiv:2309.1578, 2023c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chum Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answering. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 2507-2521. Curran Associates, Inc., 2022. URL https://proceedings wtwp:/inn..n.nuissse.eeemm.. cc/paper http://aarrffles/eaaaeaeee files/paper/2022. pdf
File/
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024. URL https:
05424
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnos- tic benchmark for very long-form video language understanding. In Thirty-seventh Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
net wttp:///ooooond.dde............ http:://.ppoooooeeeeeeeee///otls
OpenAl. Introducing chatgpt, 2022. URLhhtts.///pena..co///b//////g
OpenAl. Gpt-4 technical report, 2023.
OpenAl. Hello
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference Learning Representations, 2022. URL
net/ wttp://wwwwweeeteeteee/eee/m 2tte://fooomm0o000PpPPP00
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, URL https: httpss:/ppocceeedeeooo mlr wttpp///see/.////oreeeooaoo otgpoeeeeeoa.nlll.epeeess/////ra
18 24 Jul 2021.
html
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. ArXiv, abs/2312.0205/ 2023.
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models, 2024. URL httts:///axiv..rr
2403. 5388
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
15

---

Published as a conference paper at ICLR 2025
Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18221- 18232, June 2024a.
Zhende Song, Chenchen Wang. Jiamu Sheng. Chi Zhang, Gang Yu, Jiayuan Fan, and Tao Chen. Moviellm: Enhancing long video understanding with ai-generated movies, 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and tnn-ttnnd chat models, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural
2017. URL Information Processing Systems, volume 30. Curran Associates, Inc., fil 3fle/335ee243541 7dee
An-Lan Wang, Bin Shan, Wei Shi, Kun- Yu Lin, Xiang Fei, Guozhi Tang, Lei Liao, Jingqun Tang, Can Huang, and Wei-Shi Zheng. Pargo: Bridging lisoun-language with partial and global views, 2024. URL/tttss:// http/rrivv..// wrw./oorg/aooo abs wtlbb///4... 2408. 12928
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning, 2022. URL //tp://aavvvv.oo org/ abs 2212 03191
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks, 2024.
wttp:/twwwwwoooo//o/oaraaaoaaaap 2209. 17453
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024a. URLL:ttpss//axxx...rr///////////
04840
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality, 2024b. URI URI.https:://
org/abs 2304 14178
16

---

Published as a conference paper at ICLR 2025
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owe ow12: Revolutionizing multi-modal large language model with modality collabo- ration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13040-13051, June 2024c.
Xiaoju Ye. calfiops: a flops and params calculate tool for neural networks in pytorch framework, 2023. /tttps://github... ctmp
pytorch
Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. Voco-Ilama:
Towards vision compression with large language models, 2024d. WwRLhttpss//arxiv.coo
12275
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023.
URL /tap:s://arxvv.ooo abs 2200 02490
Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, and Lichao Sun. TinyGPT-v: Ef- ficient multimodal large language model via small backbones. In 2nd Workshop on Ad- vancing Neural Network Training: Computational Efficiency, Scalability, and Resource Opti- mization (WANT@ICM) 2024), 2024a. URL https: https:s//ppenrview..ro net/ forum?id=
Lvm wm20TZhRK
Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Lim inference unveiled: Survey and roofline model insights, 2024b. URL httpstsss////ittorrrr
abs/2402, 2402. 16363
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tumed audio-visual language model for video understanding. 2023. UKL/ 2306. 02858
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li. and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero- init attention, 2024.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models, 2024a. URL ftttps://arxiv..r
2t5//2402.1422
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, arXiv preprint arXiv:2406.0422
2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li. and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL
Forum? Corum?ld==itzzq88f22
17

---

Published as a conference paper at ICLR 2025
A DETAILED SETTING OF PRELIMINARY ANALYSES
In Sec 3.2 we analyze the importance of visual tokens in LMMs from an attention- perspective inform strategies for compressing vision tokens. Here, we give detailed introduction of the experimental setup for the attention analysis.
We focus the LLaVA series architecture, where the input tokens the LLM composed of instruction tokens, vision tokens, and response tokens, as shown in (T. We compute the average attention received by each type of token reveal how the importance of different token categories changes across layers.
Calculation of Attention Weights Formally, we denote the attention of the 8th token h; to the jth token h as Gij, where day is the average attention across all attention heads. All tokens fed to the LLM are divided into instruction tokens, vision tokens, and response tokens according inputs type, denoted sets Tisstrcctions Trision. and Tresponse respectively. Finally, denoted the target and source token types as igl_bype, svc_lype E (instruction, vision, response). the average attention weights from type type tokens to src_type type tokens in our analyses calculated as:
En.eT Ehiittin EnJET aij EheETer laps Inn, Foraaaop azy20
Attu(tgt type + sre type)
(4)
S1
1 if EnjeTm type Oij > 0
where 1 En 0g20
(5)
0 otherwise
Specifically, EhyeTor top EnyETon tym aij calculates the sum of attention weights from all igt type type tokens all sre_type type tokens, Eni "I Enn ST typ- a,>0 counts the number of tot-type type tokens, thus Attn(tgt (tgt_type - sre_type) represents the average attention weight from tgt type type tokens to sre type type tokens. Attn tgt type + sre type) is consistent with the legend in Figure 2
Calculation of Attention Entropy The calculation of attention entropy is similar to that of atten- tion weights, with the key difference being the addition of a normalization step. When computing the entropy of a specific type of token (e.g., vision tokens), the sum of attention weights for this token type may not equal 1. Thus, we perform normalization on the attention of these tokens (e.g.. vision tokens) to ensure the definition of entropy is satisfied.
In practice, for LLaVA-v1.5 (pad) (Liu et al. 2023b) and LLaVA-Ne.X (anyres) (Liu et al. 2024b), which may involve different resolution vision inputs, we use their original settings. In our analysis, we do not further distinguish between different types of vision tokens (e.g., global or local), but treat them collectively as vision tokens.
B TRAINING DETAILSS
The compression method of LLaVA-Mini can be easily plugged into ex- isting multi-modal pipelines, as it only requires the addition of two extra modules (the compression module and the modality pre-fusion module) before the LLM, while the other components (such as the vision encoder, the L.M,, and the training loss) remain unchanged. The pre-fusion module applies the same decoder-only architecture the LLM, including both the structure and hyper- parameters. The motivation behind this setting is to ensure flexible adaptation to existing LLM frameworks and other acceleration techniques.
Implementation Details
Training The overall training process follows a two-stage paradigm similar to LLaVA, consisting of rision-language pretraining followed by instruction tuning. Table reports the two-stage training details of LLaVA-Min
C BENCHMARKS
We conduct a comprehensive evaluation of LLaVA-Mini, including both image and video under- standing benchmarks.
18

---

Published as a conference paper at ICLR 2025
Table 9; Training details of LLa
Stagel
Stage2
Settings
Vision-Language Pretraining
Instruction Turning
Vision Encoder
Frozen
Frozen
Projection
Trainable
Trainable
Modules
Large Language Model
Frozen
Trainable
Compression
N/A
Trainable
Modality Pre-fusion
N/A
Trainable
Batch Size
256
256
Learning Rate
lc-4
MM Learning Rate
le-3
le-5
Hyperparameters
Schedule
Cosine decay
Warmup Ratio
0.03
Optimizer
AdamW
Epoch
C.I MAAGE-BASED BENCHMARKS
Following the LLaVA framework (Liu et al. benchmarks, including VQA-v2 (VQA") Goyal et al. 2017). GQA (Hudson & Manning 2019)
2023b) we conduct experiments on 11 widely adopted
VisWiz (Gurari et al. 2018). Science A-MMMG (SciQA) (Lu et al. 2022). TextVQA (VQA (Singh
al. 2019), POPE (Li et 2023c MME (Fu al. 2024). MMBench (MMB) (Liu et
2024c).
(SEED) Li et al. 2024b), ALaVA--Benhh-i inntch-i--the-Wee Wild (LLaVA W (Liu et al.
2023a
and MM- Vet (Yu al., 2023), which cover diverse range of visual tasks. The evaluation pipelines for all benchmarks are consistent with those used in LLaVA.
C.2 VIDEO-BASED BENCHMARKS
Video-based Generative Performance Benchmark For video-based evaluation, we conduct ex- periments on video open-ended guestion-answering benchmarks, including MSVD-QA (Chen & Dolan 201T), MSRVTT-QA (Xu et al.] 2016), and ActivityNet-QA (Caba Heilbron et al. (0155). Furthermore, we use the video-based generative performance benchmark (Maaz al. (024) to assess the performance of VA-Mini across five dimensions: correctness, detail orientation, con- textual understanding, temporal understanding, and consistency. The evaluation pipelines for both the open-ended poestion-answeriig
and the generative performance benchmarks adhere to Video- ChatGPT (Maaz al. 2024), employing the GPT model (gpt-3.5-turbo version) to evaluate the accuracy of responses (True or False) and to assign a score ranging from to for response, where higher scores indicate superior performance.
MV Bench (Li et al. 2024d) MVBench is a comprehensive benchmark for multimodal video un- derstanding that encompasses 20 challenging tasks. The evaluation aspects of MVBench include Ac- tion (such as Action Sequence, Action Prediction, Action Antonym, Fine-grained Action, and Un- expected Action), Object (Object Existence. Object Interaction, Object Shuffle), Position (Moving Direction, Action Localization), Scene (Scene Transition), Count (Action Count, Moving Count), Attribute (Moving Attribute, State Change), Pose (Fine- seinnegraiied Pose). Character (Character Order), and Cognition (Egocentric Navigation, Episodic Reasoning, Counterfactual Inference). The evalua- tion of MVBench employs multiple-choice format, using accuracy as metric.
MLVU Zhou et al. 2024b) MIVU is comprehensive benchmark for multi-task long video un- derstanding. The evaluation aspects of MIVU include Topic Reasoning (TR), Anomaly Recognition (AR), Needle QA (NQA), Ego Reasoning (ER), Plot QA (PQA), Action Order (AO), and Action Count (AC). The evaluation of MLVU also employs multiple-choice format, using accuracy as the metric.
EgoSchema is long-form video question- ceistioonannswrrngg dataset, which serves as benchmark for assessing the long video understanding capabilities of first-person videos. The evaluation EgoSchema also employs multiple-choice format, using accuracy as the metric.
EgoSchema (Mangalam et al. 2023)
19

---

Published as a conference paper at ICLR 2025
D INTRODUCTION TO BASELINES
LL- is image and video LMM, so we compare it with several advanced image-based and video-based LMMs.
D.1 IMAGE-BASED LMMs
We compare LLaVA- with LLaVA-v1.5 (Liu et al.
2023b) and other advanced LMMs of sim-
ilar data and model scales, including BLIP-2 (Li et al.
2023a), InstructBIIP (Liu et al. 2024a),
IDEFICS (Laurençon et al. 2023), Qwen-VL. VL (Bai et al. 2023), Qwen-VI--Che (Bai et al. 2023)
SPHINX (Lin et al. 2023b). mPLUG-Ow12 (Ye et al.) 2024c).
LMMs with Fewer Vision Tokens Additionally, assess LLaVA- against various efficient LMMs that utilize fewer vision tokens, showing advantages compression rate and performance. Most of these models share the same architecture and training data LLaVA, primarily focusing the merging of vision tokens in the vision encoder. These efficient LMMs are introduced as follows.
MQT-LLaVA (Hu et al. 2024) introduces flexible query transformer that allows encoding an image into a variable number of visual tokens (up to a predefined maximum) to adapt to different tasks and computational resources.
PruMerge Shang et al. 2024) reduces visual tokens in LMMs by identifying and merging im- portant tokens based the attention sparsity in vision encoder. PruMerge has variant, named PruMerge++, which enhances the original PruMerge method by evenly adding more vision tokens (about 144 vision tokens) further improve performance.
LLaMA-VID (Li et al.
2023b) LLaMA-VIL compresses the instruction and image into one token respectively, with total of two tokens representing each image, thus facilitating the understanding longer videos.
2024d) compresses all vision tokens using language models, significantly improving computational efficiency.
VoCo-LLaMA eY al.
TokenPacker (Li al. 2024e) visual projector that efficiently reduces visual tokens by 80%
using a coarse-to-fine approach.
Previous methods have often focused on reducing the number of vision tokens output by the vision encoder. LLaVA-Mini takes this a step further by shifting attention to how vision tokens and text tokens interact within the LLM backbone. Based on this insight, we propose modality pre-fusion, which enables better performance even under the extreme compression of reducing vision tokens to just one token.
D.2 VIDEO- BASED LMMS
LLa can also perform high-quality video understanding, so we compare VLaVA-Mini with
the current advanced video LMMs, including LLaMAoddaptor (Zhang et al., 2024). Intern Video
Wang et al. 2022), VideoChat (Li et al. (024c)) Video-LLaMi (Zhang et al. 2023), mPLUG-OWl
Ye et al. 2024b). Video-ChatGPT (Maaz et al. 2024). BT-AAdporr (Liu et al. 2023c LLaMA-VID
(i et al. 2023b), and Video-LLaVA (Lin et al.
2023a).
We also compare VLaVA-Mini with several video LMMs specifically designed for long videos, including MovieChat (Song al. 2024a), Movie-I.LM (Song et al. (2024b), TimeChat (Ren et al.) 2023), MA-LMM (He et al.) 2024). Note that among these video LMMs, LLaVA-Min and Video- LLa VA can complete image and video understanding with a unified model.
E EXTENDED EXPERIMENTAL RESULTS
E EFFECT OF COMPRESSION MODULE
To verify the effectiveness of the compression module, we compared the compression module in LLaVA-Mini with previous advanced token merging methods. To ensure fair comparison of
20

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of LLaVA-Mini with previous token merging methods.
Performance GQA
Methods
#Vision Tokens
VQA
MMB
MQT-LLAVA
2
61.0
50.8
54.4
MQT-LLaVA
36
73.7
58.8
63.4
MQT-LLLVA
256
76.8
61.6
64.3
PruMerge
32
72.0
60.9
PruMerget+
144
76.8
64.9
LLaVA-Mini
72.4
54.2
57.7
LLaVA-Mini
16
74.1
59.2
LLaAA-Mini
64
75.3
56.7
62.1
LLaVA-Mini
144
76.9
58.9
64.9
token compression performance, we remove the modality pre-fusion module from LLaVA-Mini for the comparison with SOTA token merging methods, including PruMerge (Shang et al. 2024). PruMerge++ (Shang et al. 2024), and MQT-LLaVA (Hu et al., 2024). Specifically, PruMerge ap- plies the widely-used token merge (ToMe) technique (Bolya et al.] (l.33) on ViT, PruMerge++ im- proves upon PruMerge by uniformly sampling additional vision tokens, and MQT-LLaVA employs Matryoshka representation learning to compress vision tokens.
As shown in Table 10 LLaVA-Mini's compression module outperforms PruMerge, PruMerget+, and MQT-LLaV. VA at the same compression rate, showing the advantages query-based compres- sion.
E.2
EFFECT OF MODALITY PRE-FUSION
Table 11: Performance of LLaVA-Mini when using only pre-fusion module without compression.
Performance GQA
Methods
#Vision Tokens
VQA 2
MMB
VLaVA-
576
78.5
62.0
64.3
V.AVAAMinii (w/o compression)
576
80.0
62.9
66.2
To validate the effect of the pre-fusion module, we remove the compression module and retained only the modality pre-fusion module, thereby comparing with LLaVA-v1.5 while both using 576 vision tokens. As shown in Table, when using only the pre-fusion module without compression, LLa achieves superior performance compared to L.LVA-v1.5 with both using 576 vision tokens, demonstrating the effectiveness of the pre-fusion module.
E.3 WIIY PREFORMING COMPRESSION AND PRE-FUSION OUTSIDE LLM BACKBONE?
LLaVA- -iii performs compression and modality pre-fusion before the LLM backbone. The moti- vation for conducting these processes outside the LLM backbone, rather than conducting at the 1hh layer within the LLM, stems from two key considerations:
Vision representations after the 1tth layers contain contextual information, which hinders the compression module: After the vision tokens fed into the LIM,, the early layers cause the visual representations to carry contextual information. Applying query-based compression on top of these representations makes difficult for the compression module distinguish between different vision tokens.
The inter-layer operations within the LLM may not be compatible with existing accelcration frameworks: One of the main motivations for placing the compression and pre-fusion modules outside the LLM backbone in LLaVA-Miní sign allows for compatibility with nearly all existing LLM acceleration technologies and frame- works, further enhancing efficiency.
keep the LLM backbone unchanged. This de-
21

---

Published as a conference paper at ICLR 2025
Table 12: Comparison of performing compression and pre-fusion outside within LLM backbone.
Performance
Methods
#Vision Tokens
FLOPs (T)
VQA 92
GQA
MMB
MivV--Mini
1.96
77.6
60.9
65.6
LLaVi-Mini (perform compression and pre-fusion within LLM)
1
1.84
76.3
60.1
64.5
We also conduct a comparison between LLaVA-Mini and LLa (compression and pre-fusion within LLM) in Table [2] The results demonstrate that the configuration of LLaVA-Mini is more advantageous. We will incorporate this result and the architectural motivation into the manuscript as per your recommendation.
E,4 EFFICIENCY ACROSS VARIOUS HARDWARE
Table 13: Inference latency (millisecond) of LLaVA-Mini on various hardware platforms.
Methods
#Vision Tokens
RTX 3090 (24G)
A100 (40G)
A800 (80G)
LLaVA-v1.5
576
198.75
113.04
87.43
64.52
38.64
27.43
4
65.52
38.84
27.71
LLaVA-Mini
68.97
39.28
28.92
64
80.10
46.23
34.65
The efficiency improvements brought by LLaVA-Mini stem from reduced computational load (FLOPs), which is consistent across different hardware platforms. To demonstrate the scalabil- ity of model efficiency across different hardware platforms, we compute the inference latency of LLaVA-Mini on three hardware platforms: RTX 3090, A A100, and A800. As shown Table [3] the efficiency improvements brought by LLaVA-Mini scalable across these hardware platforms.
E.5
COMPUTATIONAL OVERHEAD OF EACH COMPONENT
Table 14: Computational overhead (FLOPs) of each component in LLaVA-Mini.
FLOPs (T) Compression
Methods
Res.
Vision Encoder
Projection
Pre-fusion
LLM
Total
LLaVA-v1.5
336
0.349
0.024
8.177
8.55
VA-Mini
336
0.349
0.024
0.001
0.125
1.460
1.96
LLaVA-v1.5
672
1.745
0.121
38.623
40.49
VAAVA-iini
672
1.745
0.121
0.009
1.183
4.131
7.19
LLaVA-Mini significantly reduces the computational load of LMMs by decreasing the number of vision tokens. To further study the proportion of computational load contributed by each component LLaVA-Mini, we compute the FLOPs of each module, as shown in Table [4 The proposed com- pression module and pre-fusion module incur minimal computational cost, while the computation required by the L.LM backbone significantly reduced.
F VISUALIZATION OF COMPRESSION
LLaVA-Mini introduces query-based compression to adaptively compress vision tokens while pre- serving essential information. The learnable queries in compression module interact with all vi- sion tokens through cross-attentie attention to capture key visual information. To verify the effectiveness of the proposed compression, Figure [2] visualizes the cross-attention during the compression pro- Across various image types and styles (e.g., photographs, text, screenshots, and cartoons),
22

---

Published as a conference paper at ICLR 2025
MOHOAK
HONDRY
JUST MONDAA
JOSTLWONDRY
(d)
(e)
De ooth hore socrrdiannie
My Joka Valsote
le V- V-betta
[iuryy Joke] (pus to nuesi puctlinj
Drowy Malee foush col cibllee
(h)
DERTES
SERTiGE
(k)
Compression (cross-attention in LLaVA -Mini
(m)
Figure 12:
Visualization of the cross-attention in the compression module introduced in LLaVA- Mini. The left side the original image, and the right side is the cross-altention distribution heat map, where brighter areas are more heavily weighted during compression. The example images are Wild benchmark.
all from the
VA-Mn''s compression exhibits strong interpretability, effectively extracting key visual infor- mation from images. In cases where critical information is concentrated (such (b), (d), (h), in Figure 12p, LLaVA-Mini focuses on these key locations. Conversely, in cases where the main object unclear. (such as (f), ()), (i), (m) in Figure 12). LLaVA-Mini exhibits a more dispersed attention pattern during the compression process, thereby preserving a broader range of visual information.
In particular, for complex image like Figure [2]k), which contain multiple sub-figures with logi- cal relationships, the proposed compression module adaptively pay attention to the VGA-shaped charger, the product name on charger packaging, and the charging port of the charger, demon- strating the superiority of the proposed compression. Overall, compared to compression based on average pooling, query-based compression allows LLaVA-Mini to adaptively assign greater weight to key information, effectively retaining important visual details after compression.
G MORE CASES
Image Understanding Figure 13 illustrates an example of LLaVA-Mini's capabilities in more complex image reasoning. The image in Figure 13 incorporates features such as metaphor and counterfactual reasoning, requiring LMMs to accurately interpret the visual information and reason about the humorous aspects based on the entities present in the scene. The results demonstrate that only LLaVA-Mini and GPT-4o successfully capture the phrases "the musicians are performing a giant violin" " and "The violin, being a large and buoyant object, placed on the water", while both Video-LLaVA and LLaVA-v1.5 fail understand this image. In terms of the perception of
23

---

Published as a conference paper at ICLR 2025
Image:
Instruction:
What is funny about this image? Describe it panel by panel.
The image is creative and humorous take traditional concert setting, where the musicians are performing on a giant violin, which a whimsical and unexpected sight. The violin, being large and buoyant object, is placed on the water, and the musicians are seated on it it were stage. The presence of the gondolas and the water setting adds to the humor, it's not a typical environment for a concert. The musicians' serious expressions and the formal attire of the performers contrast with the absurdity of the situation, which likely intended to be playful and entertaining spectacle, The image plays on the concept of concert in a unique and imaginative way, using the visual pun of the violin's shape and the water setting to create humorous and memorable scene.
LLaVA-Mini:
This image shows humorous and creative scene of a boal shuped like giant violin floating on a canal, surrounded by traditional gondolas. Here's a panel-by-panel description: Central Focus: The Violin Boat - The main object of humor is the boat designed to look like a massive violin stringed instrument. On top of there are people standing or possibly playing music. This design turns an ordinary boat into whimsical, oversized musical instrument. Surrounding Gondolas Around the violin-shuped boat, regular gondolas can be seen with people rowing and sitting in them. These gondolas look tiny compared to the giant violin boat, adding to the comedic contrast. The gondoliers and passengers in these smaller boats seem to be watching or escorting the violin boat, enhancing the spectacle.
GPT-4o:
Water and Context gondolas. The idca of transforming a boat into musical instrument on such canal adds to the absurdity and charm of the scene.
canal is typical places like Venice, known for its waterways and
The humor comes from the umusual sight of a giant floating violin on a canal, combining the elegance of music with quirky, oversized version of everyday object in whimsical, real- world setting.
The image is panel of four pictures that showcase humorous scene of boat with man playing musical instrument on it. The boat floating in body water, and man is playing guitar. The funny aspect of the scene is that the boat is shaped like guitar, making it a unique and amusing sight. The four pictures capture different moments of the man playing the guitar on the boat, showcasing the creativity and humor of the scene.
Video-LLaVA:
LLaVA-v1.5:
The image a panel of a comic strip featuring a man a boat, paddling with a paddle. The boat is surrounded by other boats, and the man is wearing tie. The funny aspect of image
man is wearing a tie while paddling, which an unusual combination of attire for a boat ride. The scene is set in river, and there are several people in the boats around him. The man's attire and the presence of multiple boats create humorous and unexpected visual.
Example demonstrating LLaVA-Minn image understanding capability on more complex image reasoning. Output marked in red indicates factual errors.
Figure 13:
entities in the picture, both Video-LLaVA and LLaVA-v1.5 exhibit hallucinations in their descrip- tions. Specifically, Video-LLaVA erroneously interprets the image as "The image is a panel of four pictures" and "the man is playing guitar", while LLaVA-v1.5 fails to recognize the presence of the violin entirely.
Figure 14 iillustrates example of LLaVA-Ming capabilities in more challenging OCR task. The text in the image is presented an unusual cursive handwriting style, which can significantly hinder the recognition quality of LMMs. For this challenging OCR case, both L.L VAV-MMin and GPT-
24

---

Published as a conference paper at ICLR 2025
Congrabulations Hate duke on your upcominng triival
Image:
Instruction:
What's the text the cake?
LLaVA-Mini: Congratulations Kate Luke on your upcomnin arrival
GPT-4o:
The text on the cake says: "Congratulations Kate & Luke on your upcoming arrival"
Video-L.LaVA: The text on the cake reads "Congratulations Kate & Duke on your upcoming arrival!"
LLaVA-v1.5: The text on cake reads "Congratulations Kate & Luke on your upcoming wedding!"
Example demonstrating LLLVA-Mini's image understanding capability on more chal- lenging OCR task. Output marked in red indicates recognition errors.
Figure 14:
Instruction: Describe what happen this first person video?
3min video
The first-person video captures a moment of solitude and relaxation, likely in a private space such as a bedroom or a hotel room. The individual seated comfortably on chair, engrossed in reading a book, which suggests moment leisure or study, The presence of the yoga mat indicates that the person may have been plans to engage in physical activity, possibly as part of routine or for relaxation. The room's minimalistic decor and person's casual attire suggest a setting that is informal and personal. The video does not provide any explicit narrative context beyond these observations, any further interpretation would be speculative.
LLaVA-Mini:
Video-LLavee In this video, stands up and walks towards the bathroom where he takes shower. After that, he is seen sitting on the floor and reading the book again. The vidco seems to be simple yet peaceful depiction of man's daily routine. The man appears be calm and relaxed throughout the video, and the setting is minimalistic, with other characters objects present. The video's focus is solely on the man and his actions, and the viewer is left to interpret the meaning behind the scenes. Overall, the video a simple yet intriguing portrayal of man's daily routine, and it is sure leave the viewer with sense of calm and relaxation.
is sitting on chair and reading book. He then
the video, a man is seen sitting on a chair and reading a book. He then puts the book down and stands up. The video then shows him walking down hallway and entering room.
Video-ChatGPT:
Figure 15: Example demonstrating LLaVA-Mini's video understanding capability on first-person view video. Output marked in red indicates factual errors.
25

---

Published as a conference paper at ICLR 2025
4o accurately identify the text in the image, particularly with LLaVA-Mini using only vision token. In contrast, Vidco-LLaVA and LLaV incorrectly recognize "Duke" and "wedding" and erroneously add exclamation mark "!" at the end. Overall, LLaVA-Mini demonstrates superior performance in perceiving and reasoning about visual information.
Video Understanding Figure 15 illustrates example of LLaVA-Mini's capabilities in process- ing longer first-person video. The results show that LLaVA-Mini exhibits more comprehensive and detailed understanding of the video, effectively capturing entities in the room, such as the yoga mat. In contrast, Video-LLaVa VA mistakenly imagines "he takes a shower" due to its limitation of extracting only frames from the video. some detailed information. Overall, -L exhibits superior understanding of the video.
provides much shorter responses, lacking
H DETAILED RESULTS ON MVBENCH
Table 15 reports the detailed results on each subset of MV MVBench, corresponding to Table
Table 15: Detailed results on 20 subsets of MVBench.
Spatial
Temporal
mPLUG. Owl
Video- ChatGPT
VideoChat
LLaMA-
LLaMA
VID
LLaVA
Mini
Average
29.7
32.7
34,1
35.5
41.4
43.1
44.5
Action Sequence
22.0
23.5
63.5
44.5
44.5
Action Prediction
28.0
26.0
25.5
26.5
42.0
50.0
44.5
Action
Antanym
Fine-grained Action
29.0
22.5
29.0
33.5
43.0
42,0
37.0
Unexpected Action
29.0
26.5
39.0
40.5
42.0
54.5
58.5
Object Existence
40.5
54.0
48.0
53.0
39.0
$2.5
50.0
Object
Object Interaction
Object Shuffle
31.5
40.0
38.0
30.0
36.5
40.5
29.5
Position
Moving Direction
27.0
23.0
22.5
25.5
44.0
27.0
31.0
Action Localization
23.0
20.0
22.5
27.0
35.5
28.5
32.5
Scene
Scene Transition
29.0
31.0
43.0
48.5
22.0
84.5
85.5
Count
Action Count
31.5
30.5
34.0
35.0
44.5
44.5
35.0
Moving Count
27.0
25.5
22.5
20.5
28.5
26.5
40.0
Atteibute
Moving Attribute
40.0
39.5
32.5
42.5
19.0
53.0
48.0
State Change
44.0
48.5
45.5
46.0
55.6
38.5
41.0
Pose
Fine- grained Pose
24.0
29.0
32.5
26.5
37.5
34.0
29.5
Character
Character Orler
31.0
33.0
40.0
41.0
34.0
42.5
52.0
Egocentric Navigation
26.0
29.5
30.0
23.5
84.5
32.5
31.0
Cugnilion
Episodic Reasoning
20.5
26.0
21.0
23.5
40.5
38.0
38.0
Counterfactual Inference
29.5
35.5
37.0
36.0
56.5
32.0
36.0
26


Reviews:
[{"review_id": "m8JGlpJgFt", "reviewer": "Reviewer_rV2Y", "summary": "LLaVA-Mini proposes a compute and memory efficient Multi-modal by compressing the vision tokens that is passed to the LLM. The work achieves this by introducing a technique of modality pre-fusion where visual representation is fused into the text tokens and compression of vision tokens using cross-attention with learnable compression queries. LLaVA-Mini achieves comparable performance to existing models like LLaVA-v1.5 across several benchmarks, while significantly improving efficiency in terms of FLOPs, latency, and memory usage.", "questions": "- Have the authors done any ablation on using SOTA token pooling methods for the vision token pooling and compare against them (other than average pooling).\n- Can the authors highlight results on more vision centric benchmarks and show the impact of different number of vision tokens on these e.g., CV-Bench [1]\n\n\n\n\n\n[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The finding that vision tokens are primarily crucial in early layers is key to the design of the efficient early fusion architecture, a significant contribution to reducing computational overhead in LLMs.\n- Overall, the paper is well written and easy to follow. The methods, concepts are well explained and tables/ figures convey key findings. The qualitative visualization examples such as the cross-attention in compression module helps provide intuition. \n-   LLaVA-Mini achieves significant performance, reducing FLOPs by 77% and VRAM usage by more than 99% per image compared to LLaVA-v1.5, without compromising much in terms of model accuracy. This efficiency could make LLaVA-Mini highly suitable for edge devices, making it as a practical solution for real-time multimodal interactions.\n- The authors provide exhaustive ablations experiments on number of pre-fusion layers/ vision tokens etc.", "weaknesses": "- The paper does not explain the complexity of plugging this compression method to existing multi-modal pipelines. The lack of open source implementation of the code might hinder reproducibility.\n- While the paper highlights successful qualitative examples, it would be valuable if it could also discuss any failure cases encountered with the compression technique, along with potential solutions for addressing them.\n- It would be great to understand how model efficiency scales across different hardware platforms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3X9YJy7Gx6", "reviewer": "Reviewer_WP7G", "summary": "This paper presents LLaVA-Mini, an efficient large multimodal model that significantly reduces computational overhead while maintaining competitive performance. The key innovation lies in its ability to represent visual information with as few as one vision token, achieved through two main components: a modality pre-fusion module and a vision token compression module. The authors first conduct a layer-wise analysis of how LLMs process vision tokens, revealing that vision tokens are most crucial in early layers. Based on this insight, they design the pre-fusion module to process visual information before it enters the LLM, and employ a compression mechanism to minimize the number of vision tokens. Experimental results across 11 image and 7 video benchmarks demonstrate that LLaVA-Mini achieves comparable performance to LLaVA-v1.5 while reducing vision tokens from 576 to 1, resulting in 77% FLOPs reduction and significant memory savings (from 360MB to 0.6MB per image). This enables processing of long videos exceeding 10,000 frames and reduces inference latency from 100ms to 40ms.", "questions": "See \"Weakness\". \n\nAdditional question:\n\n1. Entropy Visualization Clarification:\n- Could the authors explain the apparent contradiction between the claim of decreased attention to vision tokens in the latter layers and the high entropy values shown in the visualization?\n- What causes high entropy in vision-related attention weights in the latter layers despite lower attention weights?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Quality:\n- Comprehensive empirical validation across 18 benchmarks (11 image-based and 7 video-based) demonstrates the robustness of the approach.\n- The method achieves significant efficiency improvements (77% FLOPs reduction, 600x memory reduction) while maintaining competitive performance with the baseline.\n\n2. Clarity:\n- The motivation and problem formulation are well-articulated, clearly identifying the computational challenges in current LMMs.\n- The paper presents a logical progression from empirical observations to proposed solutions.\n\n3. Significance:\n- The dramatic reduction in computational resources makes multimodal processing more accessible.\n- The approach addresses a critical limitation in current LMMs, potentially enabling real-time applications and processing of high-resolution inputs.", "weaknesses": "1. Writing:\n- The attention weight analysis lacks crucial technical details. The authors do not specify how they handle multiple attention heads (whether through averaging, individual analysis, or selective visualization), making the results difficult to reproduce.\n- The paper fails to clarify how attention scores are averaged among instruction, vision, and response tokens, which is essential for understanding the analysis methodology.\n- Despite using LLaVA 1.5 and LLaVA NeXT as base models, which both incorporate high and low-resolution vision inputs, the analysis does not differentiate between these different types of vision tokens.\n\n2. Technical Inconsistencies:\n- There's a discrepancy between the visualization results and claims: while the authors argue that attention to vision tokens decreases in latter layers, the entropy visualization shows persistently high values for vision tokens in these layers, creating confusion about the actual attention distribution.\n- The paper lacks sufficient details about the pre-fusion transformer architecture and implementation, which is crucial for reproduction and understanding the method's effectiveness.\n\n3. Architectural Design and Claims:\n- The paper's title and main claim about \"one vision token\" is potentially misleading. The implementation still requires processing full projected vision tokens in the pre-fusion transformer, suggesting that the computational efficiency gains are not solely from token reduction.\n- The ablation studies (lines 446-457) emphasize the importance of the pre-fusion module, raising questions about the actual contribution of the token compressor. The paper should explore the effectiveness of using only the pre-fusion module without compression.\n\n4. Computational Analysis:\n- The paper lacks a detailed analysis of the computational overhead introduced by the fusion module, particularly regarding the processing of N^2×5 vision tokens plus l_q text tokens.\n- The training costs during the instruction tuning stage, including memory and FLOPs comparisons with LLaVA, are not adequately addressed, making it difficult to assess the true efficiency gains during training.\n\n5. Logic:\n- The presentation of \"Modality Pre-fusion\" and \"Vision Token Compression\" would benefit from a chronological reorganization to better reflect the processing pipeline (image encoder -> vision token compression -> modality pre-fusion).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HLHQGV4xGZ", "reviewer": "Reviewer_eQGe", "summary": "a. The paper proposes LLaVA-Mini, an MLLM with improved efficiency via token compression. Specifically, the authors first unveil that early layers of MLLM are important for visual understanding. On top of this finding,  the efficient MLLM is then designed with 2 strategies: 1) compressing input visual tokens using the query-based resampler and 2) fusing visual tokens with text instruction tokens in early layers of LLM.The paper is well written, and delivered with clear motivation and simple yet effective methods.\n\nb. The authors conducted comprehensive experiments to demonstrate the effectiveness of the methods. On the other hand, LLava-mini manages to achieve competitive performance with much-reduced tokens.", "questions": "The authors claim that for video inputs, they train and infer via sampling frames at 1fps. In other words, the number of compressed visual tokens for videos of different lengths varies. I’d like to know if there’s any strategy (e.g, temporal embedding) to solve the discrepancy between training and inference (i.e., much longer videos).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A. The paper is well written, and delivered with clear motivation and simple yet effective methods.\n\nB. The authors conducted comprehensive experiments to demonstrate the effectiveness of the methods. On the other hand, LLava-mini manages to achieve competitive performance with much-reduced tokens.", "weaknesses": "A. Since the token numbers are substantially reduced. LLaVA-mini may achieve inferior performance under scenarios demanding fine-grained visual details, e.g., text-oriented benchmarks including Text VAQ, especially when compared with models with longer visual sequence input （e.g., LLaVA-Next)\n\nB. The paper lacks an important baseline, namely, keeping the visual tokens in early layers of LLMs and then compressing the visual token output on the L-th (4 in the paper) layer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5W5uAaTJQs", "reviewer": "Reviewer_CwER", "summary": "This paper proposes an efficient LMM with minimal vision tokens. 1/ To achieve a high compression ratio of vision tokens while preserving visual information, the authors first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers, where they fuse visual information into text tokens. 2/ LLaVA-Mini introduces a novel modality pre-fusion to fuse visual information into text tokens in advance before feeding into LLM and a compression module to reduce #vision tokens into minimal ones. 3/ Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on GPU hardware with 24GB of memory.", "questions": "Kindly refer to the weakness sec above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1/ The problem of how to improve efficiency of MLLM is important. \n\n2/ The authors conducted a very interesting and detailed analysis of how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers, where they fuse visual information into text tokens. \n\n3/ LLaVA-Mini introduces a modality pre-fusion module to fuse visual information into text tokens in advance before feeding into LLM and a compression module to reduce #vision tokens into minimal ones. \n\n4/ Experimental results are strong and convincing.", "weaknesses": "1/ According to Sec 3, if I understand correctly, for each downstream task, this paper trains a new, independent compression and modality pre-fusion modules. In other words, for another task, we need to train another model. I wonder, how does this approach differ from task-specific distillation? It seems the reason why the efficiency can be improved significantly is because of removing unnecessary tokens for the target downstream task. Instead, would it be possible to train generic compression and modality pre-fusion modules in stage 1 that can be generalised to various downstream tasks?\n\n2/ For the proposed approach, a new modality pre-fusion module has been introduced. I wonder if this is necessary. Instead, can we re-use the first few layers, which have high activation for vision tokens, as the fusion module. To further include the compression module, in the first few layers of LLM, the part digesting vision tokens can have two pathways, one pathway is to output the compressed tokens while the other pathway is to be gradually fused together with the text tokens.  \n\n3/ When it comes to video, there are some recent streaming video LLM works such as \"VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024\" which also advocates the idea of representing each frame with minimal vision tokens to improve efficiency. It might be worthwhile to compare or just discuss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)