Matryoshka Multimodal Models
Authors: Mu Cai
Jianwei Yang
Jianfeng Gao
Yong Jae Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Model
matryoshka
efficiency
flexible
Primary area: foundation or frontier models, including LLMs
TL;DR: Use matryoshka representation learning for the visual tokens in large multimodal models

Abstract: Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a Large Language Model (LLM). However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning/merging methods do exist, they produce a single length output for each image and do not afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose : Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2)  provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around 9 visual tokens to obtain accuracy similar to that of using all 576 tokens; (3) Our approach provides a foundation to explore the best trade-off between performance and visual token length at sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.

Published as a conference paper at ICLR 2025
MATRYOSHKA MULTIMODAL MODELS
Mu Cai'
Jianwei Yang"
Jianfeng Gao?
Yong Jae Lee'
University of Wisconsin-Madison
Microsoft Research, Redmond
https://matryoshkk- github.io.
ABSTRACT
Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into fixed large number of visual tokens and then feed them into Large Language Model (LLM). However, this design causes excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency.
While token pruning and merging methods exist, they produce a single-length output for each image and cannot afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M": Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to- tt-fiie granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g., adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M provides a framework for analyzing the granularity needed for existing datasets, where find that COCO-style benchmarks only need around 9 visual tokens to obtain an accuracy similar to that of using all 576 tokens; (3) Our approach provides foundation to explore the best trade-off between performance and visual token length at the sample level, where our investigation reveals that large gap exists between the oracle upper bound and current fixed-scale representations.
INTRODUCTION
Large Multimodal models (LMMs) (OpenAI, 2023a; Liu et al., 2023a; Zhu et al., 2024; Liu et al., 2024b;a; Wang et al., 2023; Bai et al., 2023) have shown strong performance in visual-linguistic understanding and reasoning. Models such as LLaVA (Liu et al., 2023a; 2024a;b) first embed the input image with fixed number of visual tokens, and then feed them prefix tokens to a Large Language Model (LLM) (Vicuna, 2023; Meta, 2024) to reason about the input image. Similar model designs are borrowed in video LMMs (Lin et al., 2023b; Zhang et al., 2023a), where each frame contributes a fixed number of tokens to form the final video representation.
In reality, the number of visual tokens can be prohibitively large in the case of high-resolution images, and even more so for long videos. Existing works (Lin et al., 2023b; Liu et al., 2024b; Zhang et al., 2024b; Team, 2024) mainly tackle this issue by increasing the input context length and consequently, feeding a large number e.g., 3-8k of visual tokens into the LLM. This approach has a couple of significant drawbacks: (1) the extremely long context makes both training and inference inefficient; (2) an excessive number of visual tokens can actually harm the LMM's performance, distracting it from attending to the relevant information, as we show Sec. 4.3. Several recent works (Bolya et al., 2023; Chen et al., 2024; Shang et al., 2024) use heuristics to prune and merge visual tokens to reduce the sequence length. However, they produce a single-length output and do not afford control over the final sequence length, which could be useful to trade information density versus efficiency while accounting for resource constraints in the deployment phase.
Images and videos naturally exhibit hierarchical structure from coarse to fine details, and our human visual system has evolved to recognize visual information in this coarse to fine manner, as shown by biologists and psychologists decades ago (Harris & Giachritsis, 2000; Hegdé, 2008). Can create a similar structure for LMMs, where within one suite of model weights, the visual content tokens are organized into different scales of granularities? Conceptually, our goal is to learn the visual

---

Published as a conference paper at ICLR 2025
M3 3
Describe image for me.
X5
In the heart of bustling restaurant, young airi finds solace at table...
In the heart of bustling restaurant, young girl with vibrant hair seated at wooden table, her attention captivated by the camera...
In the heart bustling restaurant, young with long. dark hair the center of attention. She's dressed blue and white striped sweater, The table is adorned with a white paper bag. perhaps holding her meal. blue Pepsi cup rests on the table ...
X5
Figure 1: Matryoshka Multimodal Models. We enforce the coarser set of visual tokens Xs,., be derived from the finer level of visual tokens Xs,- As result, the granularity of Matryoshka visual tokens gradually changes in a controllable manner. The image is from MSCOCO (Lin et al., 2014) validation set and the captions are generated given 1, 9, and 576 tokens, respectively.
tokens have a nested structure, similar to the Matryoshka Doll (Kusupati al., 2022). Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) builds the Matryoshka mechanism over neural network's 's representation vector, where each of the segments with various feature dimensions retrieval. However, for LMMs, the inefficiency
is capable of handling tasks like classification mainly comes from the number of tokens. Thus, inspired by, but different from MRL, our work is motivated to build Matryoshka Multimodal Models upon the token length dimension, that can flexibly adjust it.
Specifically, propose M Matryoshka Multi- modal Models, which enforces an LMM to learn hierarchy of visual representation granulari- ties at the token sequence level, instead of the feature dimension level in MRL (Kusupati et al., 2022). With this representation, at infer- ence time, the visual granularity can be flexibly controlled based on specific requirements, e.g.. to account for the input image's information den- sity and efficiency constraints. Our training pro- cess simple and straightforward. During train- ing, we encode the image into M sets of visual tokens from coarse to fine, Xs, 1,... M, where the number of visual tokens gradually increases, i.e., [Xs_ < | And impor- tantly, the visual tokens in coarser level derived from the visual tokens in a finer level, i.e., Xs.-, Xs., Vi. In this way, the visual information in [Xs,, X "....... Xs gradually includes more fine-grained details. For exam- ple, given natural image as shown in Figure 1, X.s, includes high-level semanties such as the restaurant and girl, while Xs includes more details such as the Pepsi cup and white paper bag. All other training settings, such the loss function and model architecture, are kept the same as LLaVA (Liu et al., 2023a; 2024a;b).
70
60-
HRSund Peffrmancs
50-
40-
M
LLaVA 1.5 Specific Scale
Ower Vt. Chat
CIBLP-78
100
200
300
400
500
600
Number Visual Tokons
Figure 2: MMBench evaluation results under 3 oracle under LLaVA-1.5-M*, LLLVA-1155 with av- erage pooling at inference time, LLaVA-1 sep- arately trained for each specific scale, and other methods. M shows least good performance as LLaVA trained for each specific scale. A large gap exists between the oracle upperbound and model's 's actual performance on a specific scale.
Our approach, M introduces several novel properties and benefits for LMMs. First, our approach can efficiently represent visual content where users can flexibly choose the visual token scale at inference time. Under one suite of weights, it generates multiple nested sets of visual tokens with different granualarities in information density. This enables flexibility in the number visual tokens used for any image during inference, enabling control over the best tradeoff between cost and performance
2

---

Published as a conference paper at ICLR 2025
based on the image video content. For example, one can use all visual tokens for images with dense details and use just a few tokens for simpler images. This flexibility can be particularly significant when handling very long visual sequences, such as vidcos. For instance, given fixed budget of 2880 visual tokens, a user could represent a video of 2880 frames each with one token or represent the same video by sampling 5 frames each with 576 tokens.
Second, our method can be used as a general framework to evaluate the visual complexity of vision- language benchmarks, i.e., which level of granularity is needed in order to perform the given task correctly. Surprisingly, we find that most benchmarks, especially those mainly crafted from natural scenes (such as COCO) (Goyal et al., 2017; Li al., 2023c; Liu et al., 2023b), can be handled well with only ~ 9 tokens per image. In contrast, dense visual perception tasks such as document understanding or OCR (Singh et al., 2019; Masry et al., 2022) require a greater amount of tokens (144 - 576 tokens) per image to handle the task well. The detailed findings are presented in Sec. 4.2.
Finally, our approach provides foundation to tackle critical task in LMMs: How to use the least amount of visual tokens while answering the visual questions correctly?. Based on the model's predictions on the test set, we find that compared to full visual tokens, the oracle can use far fewer tokens while performing much better. For example, under six common LMM benchmarks used in LLa VA-NeTT (Liu et al., 2024b), the oracle with the trained M model can use as few as 8.9 visual tokens average to achieve performance that is 8% points better than LLaVA-NeXT which uses 576 tokens per image grid. This indicates that there is a large room for improvement compared to the oracle upperbound,
show in Sec. 4.2.
To enable further research on adaptive LMMs that learn diverse information granularities, we publicly release our code and models.
2 RELATED WORK
Large Multimodal Models. Large Language Models (LLMs) like ChatGPT (OpenAI, 2023b), GPT-4 (OpenAI, 2023c), and LLaMA (Touvron et al., 2023) have demonstrated impressive reasoning and generalization capabilities for text. The LLM landscape has been significantly transformed by the introduction of models that also incorporate visual information e.g., GPT-4V (OpenAI, 2023a). Building upon open-source LLMs (Touvron et al., 2023; Vicuna, 2023), a plethora of multimodal models have made significant strides, spearheaded by models like LLaVA (Liu et al., 2023a; 2024a) and MiniGPT-4 (Zhu et al., 2024), which combine LLaMA's (Touvron et al., 2023) language capabilities with a CLIP (Radford et al., 2021) image encoder. Recently, LMMs on more tasks and modalities have emerged, such as region level LMMs (Cai et al., 2024; Zhang et al., 2023c; Chen et al., 2023; Peng et al., 2023; Zhang et al., 2023b), 3D LMMs (Hong et al., 2023), and video LMMs (Lin et al., 2023b; Zhang et al., 2023a; 2024b). However, existing LMMs typically represent the visual content with large, fixed number of tokens, making it challenging to scale to very long visual sequences such as high-resolution images or long-form videos. In this work, we propose to efficiently represent the visual content by learning multiple nested sets of visual tokens, providing flexibility in the number of visual tokens used for any image during inference.
Matryoshka Representation Learning. Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) addresses the need for flexible representations that can adapt to multiple downstream tasks with varying computational resources. This approach, inspired by the nested nature of Matryoshka dolls, encodes information at different granularities within the same high-dimensional feature vector produced by neural network. The adaptability of MRL extends across different modalities, including vision (ResNet (He et al., 2016), ViT (Dosovitskiy et al., 2021)), vision + language (ALIGN (Jia et al., 2021)), and language (BERT (Devlin al., 2018)), demonstrating its versatility and efficiency. Recent work (Li et al., 2024) extends MRL to both the text embedding space and the Transformer layers space. Our approach is inspired by MRI., but instead of learning multiple nested embeddings for high- nigh-dimensional feature vector, we learn nested visual tokens along the token length dimension for the visual input. We are the first to show that the idea of Matryosha learning can enable explicit control over the visual granularity of the visual content that an LMM processes.
Token Reduction. One of the main causes of inefficiency in recent LMMs is their large number of prefix visual tokens that are fed into the LLM (Liu et al., 2023a; Zhu al., 2024). The quadratic complexity in Transformers (Vaswani et al., 2017) is the key issue in scaling the input sequence length for Transformers. Token reduction serves as effective technique to reduce computational costs in
3

---

Published as a conference paper at ICLR 2025
Text Prompt
: Describe the scene for me.
CLIP Image
Encoder
Xis
Granularity Controller
Large Language Model
There are a group of people standing in the ski facility, some of them are holding green flag while other are...
Figure 3: Architecture of our proposed Matryoshka Multimodal Models. The visual features from CLIP are represented as several groups of coarse-to-fine visual tokens. At test time, users can explicitly control the granularity of the visual features.
Transformers. Sparse attention methods such as Linformer (Wang al., 2020) and ReFormer (Kitaev et al., 2020) conduct attention operations within local windows rather than the full context, thereby reducing the quadratic complexity of the vanilla attention operation. Another notable method is Token Merging (ToMe) (Bolya et al., 2023), which utilizes full attention but gradually reduces the number of tokens in each transformer block by selecting the most representative tokens through bipartite matching for the Vision Transformer (ViT). A recent work (Haurum et al., 2023) further studies different families of token reduction methods for ViT. However, prior approaches produce a single length output per input image and do not offer multiple granularities over the reduced token sequence. Our M approach instead learns a multi-grann cualli-granllarityy. coarse-to-fine fine token representation within the same model architecture and weights, enabling to easily be adjusted to various computational or memory constraints.
concurrent work (Hu et al., 2024) shares similar spirit with our approach, representing an image with varying numbers of visual tokens using a single set of model weights. While their method reformats the visual tokens into a sequential list via transformation layers, we use average pooling to preserve the spatial structure of the visual tokens, demonstrating effectiveness in our experiments.
3 Mr MATRYOSHKA MULTIMODAL MODELS
Our goal is to learn a Large Multimodal Model (LMM) that represents visual content as nested sets of visual tokens capturing information across multiple coarse-to-fine granularities, so that one can explicitly control the visual granularity per test instance during inference. Here we introduce how we learn a Matryoshka doll-like token sequence.
LMMs such LLaVA (Liu et al., 2023a) typically input a sequence of visual tokens as prefix tokens to the LLM for -inguliing reasoning. The visual encoder from pretrained vision-language models, such CLIP (Radford et al., 2021) and SigL.P (Zhai al., 2023), is typically utilized to project the images into the set of visual tokens. In particular, the CLIP visual encoder represents an input image I an H x W grid of visual tokens XHxw. where each X; E EC is a C dimensional feature vector. Our goal is to learn nested sets of visual tokens [Xs, Xs2 visual information we do not introduce any new learnable parameters to the LMM. We instead optimize the CLIP visual encoder learn the nested visual representation directly, and train the ensuing LLM to adapt the learned nested set of tokens.
.Ssæl which encode the coarse-to-fine manner. To this end, we enforce Xs, C X S Vi. Importantly,
For ease of exposition, we consider CLIP- ViT-L-336 (Radford et al., 2021) as the visual encoder, where an image is encoded as 24 x 24 visual tokens (576 total). We create M sets of tokens e.g., |S,] E { [1, 9, 36, 144, 576). in which the visual tokens at the coarser level are derived directly from those at the finer level. Specifically, given the initial 24 24 visual tokens, We sequentially apply 2 2 pooling with a stride 2, resulting in 12 x 12, 6 x 6, and 3 x visual tokens. Finally, we apply 3 x 3 pooling and get the most condensed single visual token. In this way, the sets of Matryoshka visual tokens can gradually preserve the spatial information in the original tokens while simultancously forming coarse-to-fine nested representation.
4

---

Published as a conference paper at ICLR 2025
We train M by averaging the autoregressive next token prediction loss for each scale S; for each image L. Specifically, given Matryoshka visual representation Xs, for scale St we maximize the likelihood of the predicted tokens matching the ground-truth answer X,:
L II Polay Xs. X4. Xa,es),
P(X. Xs. X
(3.1)
where 0 is the trainable parameters of the model, which includes both the CLIP visual encoder and the ensuing LLM. X, denotes the question in text format, L denotes the token length of the ground truth answer X,, and Xa,<, denotes all the ground truth answer tokens before the current prediction token J,,, where j denotes the token index during text token generation. We omit system messages for clarity, though they are part of the conditioning. Figure shows our model architecture.
The final objective averages over all M visual token scales:
min M Σ log P(X, Xg))
(3.2)
With this objective function, M learns nested sets of visual tokens that gradually include more details with increasing scale. For example, in Figure I, smaller set of visual tokens describes the whole scene high level while the larger set of visual tokens includes more details such as the Pepsi cup. Our training objective affords our model to conduct visual question answering under any granularity during inference. This can be particularly useful in resource constrained applications; e.g., the visual granularity can be flexibly adjusted based on the anticipated simplicity or complexity of the visual content while taking into account compute and memory constraints.
Discussion. Given the diverse snecstion-answere training data across multiple domains and sources, the intermediate visual representation in 3 will be forced to represent the visual input to make the prediction be as accurate as possible. Given a small budget such as one token, the model will automatically represent the most salient image attributes in an unsupervised manner.
At a high level, the aforementioned learning mechanism is analogous to that in autoencoder (Hinton & Salakhutdinov, 2006; Coates et al., 2011), which reconstructs the input from the compressed latent code in an unsupervised manner. If the autoencoder has small/large dimensional latent space, then will produce a blurrier/sharper reconstruction. But in either case, the learned bottleneck will still try to compress the important information in the input, just to varying degrees.
This is also demonstrated the prior work Matryoshka Representation Learning (MRL) (Kusupati et al., 2022), where the model can learn the correct granularity even when the supervision signals come from the same single ground-trute -trut target. As mentioned in MRL, the coarse-to to-fiee granularity is achieved by explicitly optimizing (log(d)) lower-dimensional vectors in nested manner, which analogous to our nested token learning. Empirically, this is further validated by our hallucination analysis on POPE and RefCOCO in Sec. 4.4.
Finally, we argue that it would be fundamentally ill-posed to curate different ground-truth targets for different granularities, as we do not know the criterion of how the different targets should differ. Therefore, we choose to let the multimodal model learn the granularity in an "unsupervised" manner.
EXPERIMENTS
In this section, we first detail the experiment settings in Sec 4.1. Then we show the performance of 3 on both image-level benchmarks 4.2 and video-level benchmarks 4.3. Finally, we analyze the behavior of Matryoshka Multimodal Models and provide ablations in Sec 4.4 and 4.5.
4.1 EXPERIMENT SETTINGS
Model We use LLaVA-1.5 (Liu et al., 2024a) and LLaVA-NeXT (Liu et al., 2024b) as the base LMMs, both with Vicuna 7B as the language model backbone. We finetune the whole model using the exact visual instruction data from LLaVA-1.5 and LLaVA-NeXT, respectively. The learning rate The learning
of L.M is 2 10-5 and 1 10-5 respectively for LL- and
5

---

Published as a conference paper at ICLR 2025
Table Comparison between
across various benchmarks under image understanding benchmarks. LLaVA-1.5-MF maintains the performance of LLaVA-1.5 while outperforming Qwen- and InstructBLIP with fewer tokens.
Approach
Tokens
MMBench
GQA
POPE
VizWiz
SEEDBench
Qwen- VL (Bai al., 2023)
256
38.2
59.3
35.2
56.3
Qwen- Qwen-VL-Chie Ch- (Bai et al., 2023)
256
60.6
57.5
38.9
58.2
InstructBIID IP-7B (Dai et al., 2023)
32
36.0
49.2
34.5
53.4
ParructBLIP-1331 (Dai et al., 2023)
32
49.5
78.9
33.4
LLaVA-1.5-7B
576
64.8
62.0
85.9
54.4
60.5
576
65.9
61.9
87.4
54.9
60.6
144
66.4
61.3
87.0
53.1
59.7
LLaVA-1.5-M
36
64.8
60.3
85.5
52.8
58.0
63.1
58.0
83.4
51.9
55.4
59.5
52.6
78.4
49.4
50.1
rate for the visual encoder 10-5 for both models. We train both models for epoch using NVIDIA H100 GPUs.
Instead of training the language model from scratch, we initialize the language model weights from praited LLaVA-1.5 and LLaVA-NeXT, which we empirically works better. We name our Matryoshka Multimodal Models LLaVA-1.5-MP and LLaVA-NeXT-M
Visual Token Scales We design 5 scales for the visual tokens. LLaVA-1.5 (Liu et al., 2024a) and LLaVA-NeXT (Liu et al., 2024b) both leverage CLIP-VTT-L-33 (Radford et al., 2021) as the visual encoder, where an image is embedded into 24 x 24 visual tokens. We gradually apply 2 x 2 pooling with stride 2, resulting in 12 12, 6 x 6, and 3 visual tokens, where we finally apply 3 x pooling get the final single visual token. Therefore, the size of Matryoshka visual token sets are S E {1, 9, 36, 144, 576). following a nested manner. The efficiency anlaysis on the system level is shown in Appendix B, where M boosts the speed of the LMM prefill process through diminished floating- operations (FLOPs) and lessens computational memory requirements.
evaluate LLa and LLaVA-NeXT on (a) diverse multimodal benchmarks: POPE (Li et al., 2023c), GQA (Hudson & Manning, 2019), MMBench (Liu al., 2023b), VizWiz (Gurari al., 2018), SEEDBench (Li al., 2023a), ScienceQA (Lu al., 2022), MMMU (Yue et al., 2024), and (b) document understanding/Opue benchmarks: VQA (Mathew et al., 2021), ChartQA (Masry al., 2022), A12D (Kembhavi et al., 2016) and TextVQA (Singh et al., 2019).
Evaluations. For image understanding.
character recognition (OCR)
For video understanding, we use both (a) open ended video question answering benchmarks evaluated by GPT-3.5: MSVD-QA (Xu et al., 2017), MSRVTT-QA (Xu et al., 2017) and ActivityNet- QA (Yu al., 2019); and (b) multi-choice video question answering benchmarks: NExT-QA (Xiao al., 2021), IntentQA (Li et al., 2023b), and EgoSchema (Mangalam et al., 2024).
4.2
IMAGE UNDERSTANDING
LLAVA-1.5-M3 We evaluate LLaVA-1.5-MS on the common multimodal understanding and rea- soning benchmarks. Results are shown in Table V.. with full tokens maintains the performance of LLaVA-1.5 1.5 across diverse benchmarks. More importantly, our approach shows strong performance even with 1 or 9 tokens. Specifically, in MMBench, a comprehensive multimodal under- standing benchmark, VALaVA-1555 1.5-M3 with 9 tokens surpasses Swen-VV-CChee with 256 tokens, and achieves similar performance as Qwen-VL. VL-Chat with even token. Compared with InstructBLIP (Dai al., 2023), LLa 5 with 9 tokens surpasses DnstructBLIP-71 and MittuuttBLIP-1311 across all benchmarks. This demonstrates that our model has both flexibility and strong empirical performance under diverse number of visual tokens.
LLaVA-NeXT-M We use the proposed Matryoshka Multimodal Models to finetune LLaVA- NeXT, and compare LLaVA-NeXT-MW with SS, which denotes the setting where the LLaVA-NeXT trained under a Specific Scale of visual tokens also for I epoch. We also include the oracle upperhound performance: 'Oracle' denotes the case where the best tradeoff between visual tokens

---

Published as a conference paper at ICLR 2025
Table Comparison of approaches with the SS baseline and M across various benchmarks under LLa VA-NeXT (Liu et al., 2024b). Here # Tokens denotes the number of visual tokens per image grid LLaVA-NeXT SS denotes the baseline model trained with a Specific Scale of visual tokens. M at least good as SS, while performing better tasks such TextVQA, ChartQA, and MMBench. Oracle denotes the case where the best tradeoff between visual tokens and performance is picked.
Tokens Per
Approach
TextVQA
A12D
ChariQA
DocVQA
MMBench
POPE
ScienceQA
MMMU
Grid
SS
64.53
64.83
59.28
75.40
66.58
87.02
72.29
34.3
576
Ma
63.13
66,71
58.96
72.61
67.96
87.20
72.46
34,0
SS
62.16
65.77
55.28
67.69
67.78
87.66
72.15
36.4
144
A13
62.61
68.07
57.04
66.48
69.50
87.67
72.32
36.1
SS
58.15
65.90
45.4 40
56.89
67.01
86.75
71.87
36.2
36
58.71
67.36
50.24
55.94
68.56
87.29
72.11
36.8
50.95
65.06
37.76
44.21
65.29
85.62
72.37
36.8
9
51.97
66.77
42.00
43.52
67.35
86.17
71.85
35.2
38.39
63.76
28.96
33.11
61.43
82.83
72.32
35.3
M/3
38.92
64.57
31.04
31.63
62.97
83.38
71.19
34.8
# Tooeens
31.39
11,54
41.78
64.09
8.90
6.08
7.43
22.85
Oracle
Performance 70.51
76.36
70.76
81.73
74.35
94.29
76.07
50.44
Table 3: Overall accuracy of LLaVA-NeXT-M* and recent video LMMs on various video understand- ing benchmarks. Here Tokens denotes the overall number of visual tokens across all frames.
Approach
# Tokens
MSVD
MSRVTT
Activity Net
NexIQA
IntentQA
EgoSchema
Video- LMM (Zhang et al., 2023a)
32
$1.6
29.6
12.4
LLaMA-Adapter (Zhang et al., 2024a)
54.9
43.8
34,2
Video-ChatGPT (Mazz
2023)
264+ 2048
49.3
35.2
Video-LLLaV (I et al., 2023a)
70.7
59.2
45.3
InterVideoo (Wang et al., 2022)
59.1
32.1
LLaVA-NeXT-7B (Liu et al., 2024b)
2880
78.8
63.7
54.3
63.1
60.3
35.8
64.5
63.1
36.8
720
79.0
64.5
55.0
62.6
59.6
37.2
LLaVA-NeXT-7B-M
180
77.9
63.7
$5.0
61.4
59.3
37.6
59.5
38.8
73.5
62.7
50.8
56.5
56.7
36.2
and performance is picked for each test instance; i.e., for each test instance, we select the scale with the fewest amount of tokens but can answer the question correctly. Table 2 shows that our approach, M3, at least as good as SS, while performing better on tasks such as document understanding (TextVQA and CharlQA) and common benchmarks such as MMBench (Liu et al., 2023b).
Our results also show that dataset level biases towards the visual token scales do exist. For example, ScienceQA maintains consistent performance across all visual token scales. A12D and MMBench only encounter a small performance drop for even as few as 9 to tokens. On the other hand, dense visual perception tasks such as TextVQA and Doc show significant performance drop with fewer tokens. This analysis shows that 3 could serve as framework to analyze the granularity that benchmark needs.
Furthermore, there is a large gap between the model's actual performance under full tokens and the upper-bound oracle. This indicates that using full tokens cannot always result in the optimal performance for all samples; i.e., there is a large room of improvement towards the oracle point.
4.3 VIDEO UNDERSTANDING
Following (G-VLM (Kim al., 2024), we directly conduct zero-shot inference diverse video benchmarks using LLa VA-NeXT-MM Specifically, 6 frames are uniformly sampled over the entire video, then arranged collage, which is fed into LLaVA-NeXT along with the question get the response. Results under LLaVA-NeXT-MM and recent video LMMs are show in Table 3.
LLaVA-NeXT-MA with full visual tokens again shows comparable performance with LLaVA-NeXT. More interestingly. results indicate that full visual tokens usually do not lead to the best performance in video understanding tasks. Specifically, 4 out of 6 benchmarks, full visual tokens show less
7

---

Published as a conference paper at ICLR 2025
Table Comparison between and heuristics based sampling baselines (average pooling, spatial sampling, sequential sampling) at inference time on MMBench with the LLaVA-NeXT architecture.
Tokens
M
Average Pooling
Spatial Sampling
Sequential Sampling
576
67.96
67.18
67.18
67.18
144
69.50
61.68
60.14
36
68.56
50.77
60.05
44.76
67.35
45,45
45.45
31.96
62.97
19.33
26.29
22.42
aaeau dratie dan
via sumber Do
whuttsaa sead seint
that campuny spencor
via kkples
Figure 5: TextVQA test samples with correct and incorrect predictions upon different scales. Answers vary with different number of visual tokens. In addition, M can serve as a framework evaluate the complexity of images.
desirable performance compared 720 or 180 visual tokens. We suspect that very long visual context could bring distraction (e.g., too much focus potentially irrelevant background) to the model's prediction, where a compact representation of the video focusing on the more relevant information may be more advantageous.
Finally, for most video understanding tasks such as ActivityNet, IntentQA and EgoSchema, with tokens per image grid (45 tokens in total), the accuracy difference compared to full tokens (2880 in total) is less than 1%. This demonstrates that the video questions in these benchmarks usually require very sparse visual information, the source of such video understanding benchmarks mostly comes from natural scenes, which matches our observation in image understanding benchmarks.
4.4 IN-DEPTH ANALYSIS
M shows much stronger performance compared to heuristics based sampling at test time. A simple way to reduce the number of visual tokens via training-fres way is to conduct heuristic token merging or reduction, which is utilized in works such as Zhang et (2024b) and Rippel et al. (2014). In Table 4, we compare M with three training-frea approaches: average pooling, spatial sampling, and sequential sampling. M is much more re- silient when the number tokens decreases, while the heuristic based sampling approaches show dramatic perfor- mance drop. A visualization of the spatial and sequential sampling shown in Figure 4.
(a) Sequential
(b) Spatial
Figure 4: Visualization of sequential vs spatial sampling. Given 24 24 grids, visualized cells denote sampled tokens.
S serves as good metric for image complexity. We extract the response from -LaVA-NeeT-MM in the TextVQA benchmark, and show samples where using visual tokens across different scales can answer the question correctly and incorrectly. Shown in Fig. 5, the OCR performance aligns with the complexity of images, indicating that M can be utilized metric towards sample level complexity.
Large gap between oracle and actual performance. As shown in Table 2, the oracle upper-bound very few (6 64) tokens yet achieve least 10% better performance compared to full visual tokens. This suggests that a visual token scale predictor, where the model learns to automatically select the best visual token scale given the input images or both input images and questions, has potential to achieve a better tradeoff. This would be interesting future work.
Zero-shot generalization to longer visual sequences. Here extend the length of the visual tokens at inference time to study the model's zero-shot generalization behavior. Results under LLaVA- NeXT are shown in Table 5. Here L.aVA-NeXT-My$ is trained 2 image grids but evaluated
8

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison of different image grid configurations with LLaVA-NeXT-MP.
Grids
| Tokens per grid
Overall Tokens
TexiVQA
A12D
ChariQA
DocVQA
MMBench
POPE
ScienceQA
x
62.61
68.07
57.04
66.48
87.67
72.32
3 3
144
1440
64.73
67.75
58.84
70.59
69.50
87.67
72.22
2
576
2880
63.13
66.71
58.96
72.61
67.96
87.20
72.46
Table 6: Accuracy, Precision, and Recall on the POPE dataset with M
Score for Adversarial Type and Overall Hallucination Rate.
Table 7:
# Tokens
|Accrracy Precision Recall
# Tokens
|Score A.dversariall Rate
S76
88.08
94.13
81.22
S76
2.00
69%
144
88.43
93.84
82.27
144
2.08
56%
36
88.02
88.02
82.24
36
1.92
77%
87.03
92.31
80.80
2.00
64%
83.58
83.58
82.38
2.08
69%
on 3 x 3 grids. We set the number of visual tokens to 144 in each image during evaluation. The model obtains significant improvement in document understanding by 2.12, 1.80, and 4.1 1 on TexIVQA, ChartQA, and DocVQA, respectively, while maintaining the same performance on benchmarks mainly composed of natural scene images. 3 image grids with 144 tokens per grid own 1440 tokens, yet achieve similar performance with the default LLaVA-NeXX 2 x 2 image grids with 2880 total tokens (576 tokens per grid). This indicates it promising to feed more subimages while making the number of visual tokens within each subimage much smaller.
Detailed investigation into potential hallucinations. We next investigate potential hallucinations of M on two popular benchmarks: POPE (Li et al., 2023c) and MMHal-Bench (Sun et al., 2023). Table 6 shows results POPE. Accuracy reflects the proportion of correctly answered questions, Precision and Recall reflect the ratios of correctly answering questions whose answers are "Yes" or "No", respectively. M maintains the same level of recall while showing gradually increasing precision as the number of tokens is increased. Results for HMMHal-Bench (Table 7) reveal similar trends. Different numbers of visual tokens show similar performance on the "adversarial" type. Similar to the "No" question type in POPE, here "adversarial" questions ask models about the objects that do not exist the image. Thus "adversarial" questions expects models to say "No". Different numbers of visual tokens show similar overall hallucination rate. Overall, these results demonstrate that M maintains a similar level of hallucination across different seales of visual tokens.
Effect of object size. For the objects appearing in POPE images, we categorize the objects into three groups based on the area occupied by the object. With more visual tokens, M shows significant performance boost in small objects compared to the large objects; see Table 8. Visual grounding results on RefCOCO testA set (ACC@0.5) (Yu et al., 2016) corroborate this finding, where more visual tokens leads to significant performance boost for understanding small objects compared to large objects; see Table 9, These results indicate that M tends to preserve more coarse-grained information when the number of tokens are limited. Importantly, both the hallucination experiments from the previous section and the results here demonstrate that coarse-tee to- to-fine granularities can be learned in unsupervised manner, using one ground-truth target for all token lengths.
4.5 ABLATION STUDIES
Matryoshka visual token sampling. Here we compare three different ways to select the visual tokens for Matryoshka Multimodal Models, including average pooling, spatial sampling, and sequen- tial sampling, which is illustrated in Figure 4. Shown in Table 10, averaging pooling shows better performance than the alternatives across diverse benchmarks. In general, sequential sampling performs the worst. We hypothesize that this due to the visual tokens having spatial information, while sequential sampling does naturally align with the spatial distribution of the visual tokens.
Training the entire LMM vs only training CLIP. Since the nested behavior of Matryoshka visual tokens is learned within the CLIP visual encoder, we next evaluate whether it is necessary to also finetune the L.LM. Shown in Table 11, training the whole LLM achieves better performance. This demonstrates that by also training the LLM, the model can better adapt the patterns of the visual tokens distributed in the Matryoshka manner.

---

Published as a conference paper at ICLR 2025
Table 8: M Performance on different scales with varying numbers of visual tokens POPE.
Table 9: RefCOCO visual grounding accuracy for different object sizes.
# Tokens
Small
Medium
Large
# Tokens
Small
Medium
Large
Average
576
85,20
84.05
88.98
576
76.2
86.4
88.5
83.7
144
86.31
86.11
88.98
144
70.3
84.5
87.3
80.7
36
78.19
82.50
87.96
36
54.8
75,4
85.5
71.9
76.39
80.53
85.40
9
30.5
55.9
75.2
53.9
78.53
80.02
85.31
7.8
14.2
28.8
16.9
Ablation on Matryoshka visual token sampling including average pooling, sequential sampling, and spatial sampling with LLaVA-NeXT-M*.
Table 10:
TeaVQA
MMBench
Num of Vis Tokens
Avg Pooling
Sequential
Spatial
Avg Pooling
Sequential
Spatial
Avg Pooling
Sequential
Spatial
576
63.13
59.37
60.45
67.96
64.60
64.43
66.71
65.61
64.96
144
62.61
55.80
58.33
69.50
64.18
64.52
68.07
64.90
64.96
36
$8.71
$2.79
$2.39
68.56
63.92
64.69
67.36
64.51
64.02
51.97
44.05
44.19
67.35
63.14
62.11
66.77
63,70
63.92
38.92
28.03
29.91
62.97
59.36
57.47
64.57
63.21
63.08
Table 11: Performance comparison of training LLaVA-NeXT-M* with and without training the LLM across diverse benchmarks. We clear drop when freezing the LLM.
Num of Vis Tokens
TexiVQA
MMBench
A12D
DocVQA
w/ LLM
w/o LLM
w/ LLM
w/o LLM
w/ LLM
w/o LLM
w/ LLM
w/o LLM
576
63.13
61.16
67.96
63.66
66.71
63.92
72.61
69.15
144
62.61
57.79
69.50
65.21
68.07
63.73
66.48
59.77
58.71
49.75
68.56
63.92
67.36
62.89
55.94
44.08
51.97
36.15
67.35
61.08
66.77
62.05
43.52
28.36
38.92
19.72
62.97
51.80
64.57
60.59
31.63
17.37
Table 12: Comparison of performance between average pooling (parameter-free) and learning-based approaches including Resampler (Nlayrac et al., 2022), C-Abstractor (Cha et al., 2024), and D- Abstractor (Cha et al., 2024) MMBench and TextVQA with L.5VA-L.5-M3.
MMBench
TeatVQA
Tokens
Avg Pooling
Resampler
C-Absuactor
D-Abstractor
Avg Pooling
Resampler
C-Abstractor
D-Abstractor
576
65.9
65.8
65.4
58.8
57.75
53.50
$5.47
48.22
144
66.4
66.0
66.3
56.7
56.76
51.95
54.32
46.69
63.9
64.9
48.67
9
63.1
59.4
62.8
53.5
53,50
49,83
46.53
45.24
59.5
53.2
58.4
49.6
49.49
44.64
46.05
44.12
Comparison with training-based visual token reduction. in Table 12 show that average pooling used in M overall produces the best performance compared to three learning based approaches: (a) resampler (Cha et al., 2024), (b) C-Abstractor (Cha et al., 2024), and (c) D-Abstractor (Cha et al., 2024). This is likely because averaging pooling preserves spatial details better. Furthermore, average pooling does not require any new learned layers or parameters.
Results on MMBench and TextVQA
5 CONCLUSION
We introduced M Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens, capturing information across multiple coarse-to-fing granularities. LMMs equipped with M afford explicit control over the visual granularity per test instance during inference. We also showed that 3 can serve needed for existing datasets, where we discovered that a large number multimodal benchmarks only need as few as visual tokens to obtain accuracy similar to that of using all visual tokens, especially for video understanding. Furthermore, we disclosed a large performance- -effunciceeffincccyycc gap between the oracle opper-bount and the model's performance. Our work to other domains. For example, the long context in a text-only LLM or vision tokens dense vision tasks can also be represented nested sets of tokens in Matryoshka manner.
analysis framework to investigate the visual granularity
be naturally extended
10

---

Published as a conference paper at ICLR 2025
6 REPRODUCIBILITY STATEMENT
We have publicly released our code, data, and pretrained models, so that the community can fully reproduce, and build-upon, our work.
ACKNOWLEDGEMENT
This work was supported in part by NSF CAREER IIS2150012, NSF IIS-2404180, and Institute of Information & communications Technology Planning & Evaluation(IITPP) grants funded by the Korea governmen((MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration) and (No. RS2022-00187233 Development of Large Korean Language Model Technology for Efficient Pre-training), and Microsoft Accelerate Foundation Models Research Program.
REFERENCES
Jean- nan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35:23716-23758 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Represen- tations, 2023.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee:
projector for multimodal 1lm. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), 2024.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint 230..22300111995. 2023.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. arXiv preprint MeXXi:224030011 2024.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort Lauderdale, FL, USA, 11-13 Apr coates html.
2011. PMLR. URL. https:ttttt... http:://ooooceoeeooeoooo ngs ml prwss//s///e...... /otts//s/s//s/////c//caaaaoossee
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purgae purpose vision-language models with instruction tuning, 2023.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 180.v:1810.048805 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICI.R, 2021.
11

---

Published as a conference paper at ICLR 2025
Yash Goyal, Tejas Khot, Douglas Summers-Ster Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5904-6913, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3608-3617, 2018.
Mike G Harris and Christos D Giachritsis. Coarse-grained information dominates fine-grained information in judgments of timetto-contact from retinal flow. Vision research, 40(6):601-611 2000.
Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, and Thomas B. Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) Workshops, October 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Jay Hegdé. Time course of visual perception: coarse-to- processing and beyond. Progress in neurobiology, 84(4):405-44 439, 2008.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 31 31365786))504--07, 2006.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurlPS, 2023.
Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. arXiv preprint arXhv:2405.19315 2024.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904 PMLR, 2021.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 235-251, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0.
Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth video: Zero-shot video question answering using a vlm. arXiv preprini arXiv:2403. 144403.88406 18406, 2024.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL URRphtst:///peeeeiiii n net/Eorumt Eorum? w id=rkgNKkHtv...
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning, Advances in Neural Information Processing Systems, 35:30233-30249, 2022.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal 11ms with generative comprehension. arXiv preprint arXiv:2307.1673 2023a.
Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In Int. Conf. Comput. Vis., pp. 11963-11974, 2023b.
12

---

Published as a conference paper at ICLR 2025
Xianming Li, Zongxi Li, Jing Li, Haoran Xie, and Qing Li. 2d matryoshka sentence embeddings. arXiv preprint arXiv:2402.14776 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023c.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. ArXiv abs/2311.10122, 2023a.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024b. URL https:
// //llava-vl,
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?
arXiv preprint arXiv:2307.0628 2023b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022.
Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv abs/2306.05444 055224 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In Ady. Neural Inform. Process. Syst. 2024.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics, doi: 10. 1/211853/v1/022.. hin.:..iinninnaac..11. URL https: /aw.aaaaanntoooyyy.oooooo org/ 2022. findings- aat.nngsaaal.17.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Meta. Llama-3, https:///// http:/aa.aaaaa...e.mm mi..mmta..cmm///o//eeett.. 1llmma 3/, 2024.
OpenAl. Gpt-4v(ision) system card. Card.pdf, 2023a.
OpenAl. Chatgpt. https: /ttp//ppppaaaaaeeeee//e.e...a..a wwtpeenai.ccmmbioeem htt.a/aocc/obobooocccct//////// 2023b.
OpenAl. Gpt-4 technical report. 2023c.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp. 1746-1754. PMLR, 2014.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint WrXiv:2403.15385 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 1309, 2023.
Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Vicuna. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. //vicuna. Imsys org/, 2023.
https:
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. 2020.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogylm: Visual expert for pretrained language models, 2023.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. ArXiv abs/2212.03191, 2022.
Junbin Xiao, Xindi Shang. Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering explaining temporal actions. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference Multimedia, pp. 1645-1653, 2017.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part Il 14, pp. 69-85. Springer, 2016.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: dataset for understanding complex web videos via question answering. In AAAI, volume 33, pp. 9127-9134, 2019.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05822 2023.
14

---

Published as a conference paper at ICLR 2025
Zhihang Yuan, Yuzhang Shang. Yang Zhou, Zhen Dong. Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee. Yan Yan, et al. Llm inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng. Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Conf. Empirical Methods in Natural Language Processing, pp. 543-553, 2023a.
Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, and Jianwei Yang. Llava-grounding: Grounded visual chat with large multimodal models, 2023b.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ICLR, 2024a.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-offinterest arXiv preprint arXiv:2307.03661, 2023c.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL wal.ivll.iiiiiiii wiw.ub...........gg.g.ggg....... io /bll//llo/////oooooooooo/ /aloo//2000 20040002220000 04- 30-130vee llava next video/.
https:// //wllvalaaaaaaaeee -vl.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ICLR, 2024.
APPENDIX
A BROADER IMPACT
The broader impact of 3 framework with nested visual representations, has potential benefits and risks associated with its deployment and release. Our model is trained using the exact same architecture and data of LL. (Liu et al., 2024a) and LLaVA-NeXT (Liu et al., 2024b). All the concerns same LLaVA. Specifically, GPT-4 and GPT-4V generated data. The bias from GPT-4 and GPT-4V would still exist in VA..A.
example, LLaVA conducts instruction tuning using
B EFFICIENCY ANALYSIS
To illuminate the computational benefits conferred by M we employ the roofline-based Viewer analysis as detailed in (Yuan et al., 2024). Our analysis is set within a hypothetical context designed to emphasize the effects of M on processing efficiency in LMMs. We study the LLLVA--1.5 case where a 336 x 336 resolution image is processed using a CLIP- image encoder, resulting 576 visual tokens. Accompanied by text prompt with an assumed number of 30 tokens, the nested visual tokens in M substantially lowers the visual token count. The consequences of this reduction are substantial outlined in Table 13, detailing the computational costs involved in the LMM prefill process. Notably, 3 not only boosts the speed of the LMM prefill process through diminished floating-point operations (FL.Pss) but also lessens computational memory requirements.
15

---

Published as a conference paper at ICLR 2025
It is erucial highlight that the advantages of M are not limited to just efficiency improvements. The token reduction approach of M can also enhance other LMM acceleration methods, such as quantization and factorization, referenced in (Yuan al.. 2023). This complementary relationship accentuates the broad potential of M to contribute to a wider array of efficiency-boosting strategies.
Table 13:
Computation Cost Analysis. The development device is Tesla V100 GPU, and time estimated by the roofline model represents the theoretical performance that the hardware can achieve.
# Tokens
FLOPs (T)
Prefill Time (ms)
Total Memory (GB)
Storing Activation (GB)
576
8.0
58.1
21.6
3.8
144
2.2
19.5
15.0
0.7
36
0,9
18.0
13.8
0.3
9
0.5
17.7
13.6
0.2
1
0.4
17.6
13.5
0.1
C MORE ABLATION STUDIES: INITIALIZE WEIGHTS FROM LLAVA AND AVERAGE LOSS UPON ALL TOKEN SCALES
As explained in Sec. 3 and 1, we (a) initialize the LLM weights from LLaVA and (b) minimize the loss averaged upon all visual token scales for each sample during training. An alternative choice is to randomly sample a visual token scale. Shown in Table 14, initializing the LLM weights from LLaVA and minimizing the losses over all scales shows consistent performance boost compared to using the vanilla text-only pre-trained LLM weights (Vicuna, 2023) and randomly selecting visual token scale. Initializing the LLM weights from LLaVA VA makes the training process of M more stable. By learning all scales at once, the model is forced to learn the nested behavior for each sample, which leads to better performance.
D MORE VISUALIZATIONS ON NESTED VISUAL REPRESENTATION
Shown in Figure 6, with more visual tokens, LMMs can discover more details, such as furniture and human attributes. Besides. LMMs generate higher quality descriptions with more visual tokens, as demonstrated by the OCR capability in Figure 6 (b).
E THE BLIND LOWER BOUND PERFORMANCE WITH O VISUAL TOKENS
Table 15 shows that 0 visual tokens (i.e., text-only model) align with the trend shown in our paper under different numbers of visual tokens. Removing all visual tokens results significant performance drop most benchmarks especially for document understanding such as ChatVQA and VOA. On the other hand, ScienceQA, MMMU, and AI2D show very similar performance when dropping all visual tokens, demonstrating that language prior plays much more significant role in those benchmarks.
F PERFORMANCE OF TOKEN CONCATENATION UNDER M
An intriguing question is: Can token concatenation under M further improve performance? As shown in Table 16, (1) concatenating all token scales shows slightly better performance but requires many more visual tokens. (2) concatenating all but the last token scale shows comparable performance the largest visual token scale (576 tokens).
G GENERALIZATION TO MORE TOKEN SCALES
We conduct experiments on a set new scales. As shown in Table 17, on new scales, M can still represent images in nested manner with strong performance.
16

---

Published as a conference paper at ICLR 2025
Table 14: Impact of (a) initializing the LLM weights from LLaVA, and (b) averaging the loss from all scales vs randomly selecting scale for each sample during training with LLaVA-NeXT-M
Technique
TexiVQA
A12D
Init LLM weights from LLaVA
Average losses over all scales
V
576
60.36
62.25
61.01
63.13
62.40
65.06
65.84
66.71
144
59.61
61.02
59.80
62.61
63.67
65.61
65.77
68.07
36
54.86
55.91
55.32
58.71
63.67
65.32
66.68
67.36
46.84
47.04
48.80
51.97
63.02
64.83
65.38
66.77
33.78
33.68
36.05
38.92
61.53
63.21
63.37
64.57
Table 15: Performance 3 across various benchmarks under LLaVA-NeXT (Liu et al., 2024b).
Here Tokens denotes the number of visual tokens per image grid in LLaVA-NeXT, including visual token.
# visual tokens
TextVQA
A12D
CharQQ
DocVQA
MMBench
POPE
ScienceQA
MMMU
63.13
66.71
58.96
72.61
67.96
87.20
72.46
34.00
62.61
68.07
57.04
66.48
87.67
72.32
36.10
58.71
67.36
50.24
55.94
68.56
87.29
72.11
36.80
51.97
66.77
42.00
43.52
67.35
86.17
71.85
35.20
38.92
64.57
31.04
31.63
62.97
83.38
71.19
34.80
10.03
60.43
14.24
11.95
21.48
56.65
70.79
31.20
Besides, other Matryoshka sampling strategies can also support a diverse number of visual tokens at inference time, such as sequential sampling. However, in this paper, we choose average pooling due to its superior performance, demonstrated in the ablations in Table 10.
H STUDY ON MORE STRICT ORACLE PERFORMANCE WITH MULTIPLE RUNS
We evaluations for 5 times using different seeds with temperature 0.2 get more strict oracle. Then, we take a majority vote for the prediction for each sample in each scale. Shown Table 18, with 5 runs, the upper bound performance is similar to that of a single run.
I STUDY A MORE STRICT ORACLE PERFORMANCE WITH MULTIPLE RUNS
Our proposed 3 is plug-and-play methodology that can be generally applied to multimodal models including both LLaVA-style (treating visual tokens as prompts) and Flamingo- (Alayrac et al., 2022) (using cross-ates aotenttntntion to fuse visual features into text features). Specifically, Flamingo-style MLLMS (i) first extract visual features using the Perceiver Resampler Alayrac et al. (2022), and (ii) attention for multimodal fusion. M3 can be naturally applied between these two stages to produce the multi-granin guutuigrranulrrity visual representation before the cross attention stage, resulting the Flamingo-M Therefore, the computation, reflected by measures such as FLOPS, can also be reduced if a coarse-graine gransed visual representation scale is used.
then apply
Specifically, consider ny as the number of visual tokens, nhy as the number of visual tokens after passing the perceiver resampler in Flamingo, n. as the number of text tokens, M as the visual token reduction scale for our proposed M3, dndand,,, as the hidden dimension in the vision encoder and language model, respectively. For simplicity, we omit the multimodal connector (MIPP in LLaVA and Perceiver Resampler Flamingo), as well as the number of layers in vision encoder and language decoder since they in the same scale (24 v.s. 32 layers in LLa VA-1.--BBB. Then the computational complexity can expressed Table
17

---

Published as a conference paper at ICLR 2025
M3
Describe this image for me.
X55
The image shows interior space that appears to be living room or a combined living and dining area...
The image shows an interior space that appears to be a living room or lobby. The room has warm color scheme with beige walls and darker
XSc
large, L-shaped
The image shows an interior space that appears to be a living room or combined living dining area... large, L-shaped sofa light-colored upholstery, positioned the center of the room. In front of the sofa, there is glass-top coffee table with various...
Xsx
(a)
M3
3
Describe this image for
This white photograph capturing baseball game. the foreground, there three individusls...
This is black and white photograph capturing moment from baseball game. the foreground, three baseball players are standing field. player
X.22
KIIN
wearing baseball uniform the "KIMBLE" the front, cap, and glove..
ERLLY
This black and white photograph capturing moment from a baseball game. left section, team. He is dressed in white baseball uniform with the word "KIMBERLY" emblazoned across the chest. He is holding beseball glove, ready action.
player from Kimberly
Xey
(b)
Figure 6: More visualization examples. With more visual tokens, LMMs can discover more details, and generate higher quality descriptions. The images are from MSCOCO (Lin et al., 2014) validation set.
Table 16: Performance of M across various benchmarks under different numbers of visual tokens and when (1) all token scales are concatenated (2) all but the last token scale are concatenated.
# visual tokens
MMBench
GQA
POPE
VizWiz
SEEDBench
576
65.9
61.9
87.4
54.9
60.6
144
66.4
61.3
87.0
53.1
59.7
36
64.8
60.3
85.5
52.8
58.0
9
63.1
58.0
83.4
51.9
55.4
I
59.5
52.6
78.4
50.1
49.4
14443649+1=190
63.8
63.5
86.7
56.1
61.2
576+ 376+1444434941-766
62.5
63.7
85.0
56.7
61.4
18

---

Published as a conference paper at ICLR 2025
Table 17: Performance of M across various benchmarks under of visual token scales. On new scales, M can still represent the images in a nested manner with strong performance.
# visual tokens
MMBench
GQA
POPE
VizWiz
SEEDBench
576
67.4
61.5
87.2
49.5
60.2
144
65.2
59.3
85.3
46.5
57.1
64
64.7
58.4
84.3
46.0
55.1
36
63.2
57.0
83.6
45.9
53.1
16
61.8
54.7
80.9
44.7
50.2
9
58.7
53.3
78.4
42.7
49.1
4
54.0
51.3
77.5
41.4
45.2
1
47.9
47.7
74.6
40.3
43.1
Table 18: Token counts and performance across multiple datasets for run and runs for the oracle performance of M
TextVQA
A12D
ChartQA
DocVQA
MMBench
GQA
ScienceQA
MMMU
Tokens
31.39
11.54
41.78
64.09
8.90
6.08
7.43
22.85
Oracle Performance
70.51
76.36
70.76
81.73
74.35
94.29
76.07
50.44
runs # Tokens
33.24
13.42
44.05
67.54
9.52
6.09
8.05
21.03
Oracle Performance
71.00
75.94
69.44
76.48
94.27
76.94
50.11
Table 19: The computational complexity of the original model and M in LLaVA and Flamingo style models.
Model
Original Complexity
Matryoshka Complexity
LLaVA
Onn dr (nv n1.)22.. dr) On dy + nhy ne di + n dr)
O dv + nny v11))² dr. ny
Flamingo
O dy + M nn di + n², dr)
19


Reviews:
[{"review_id": "DjlBNIQbZq", "reviewer": "Reviewer_8DFt", "summary": "The paper proposes Matryoshka Multimodal Models (M^3), a MLLM framework supporting variable visual token length for flexible inference cost control. The multi-scale visual features are generated in a homogeneous manner by pooling visual encoder output with different kernel sizes, and the same set of weights are jointly trained on all scales, producing a single all-in-one model for easy deployment. Models are constructed in a continual finetuning setting on top of LLaVA and LLaVA-NeXT, and evaluated on a broad range of MLLM benchmarks with extensive ablation studies.", "questions": "* **Symbol clarification**: At Line 203, it's stated that $X_{S_i} \\subset X_{S_{i + 1}}$. However, if we consider $X_{\\cdot}$ as a set of $C$-dimensional features, then this seems self-contradictory because the average of a subset of elements $\\frac{1}{n} \\sum_j X_{S_{i + 1}, j}$ is not necessarily one of the original elements. Could the authors provide a more rigorous definition about symbols $X_{S_i}$ and, in turn, the definition of Matryoshka property studied in this case?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper tackles a meaningful problem in practice. The motivation is coherent and easy-to-follow.\n\n* The method description is mostly clear.\n\n* Extensive experiments with strong results and insightful analysis.", "weaknesses": "* **Comparison with Flamingo-style (i.e., cross-attention-based) methods**: Despite the popularity of LLaVa-style MLLMs which treat visual tokens as prompt, Flamingo-style MLLMs, which decode text-conditioned salient visual features with cross-attention modules, are also studied as an alternative paradigm in several previous works, e.g., [1, 2]. It's noteworthy that cross-attention alleviates most of the the performance penalty due to long visual sequences by nature, because the visual tokens do not go through the expensive MLP and quadratic-complexity self-attention in the language model part (it may still be quadratic in the visual encoder part though). Even if extensive experiments are infeasible within the rebuttal period, it might still make the argument stronger and benefit future readers if at least some discussions could be included (e.g., related works, theoretical analysis of both methods, FLOPS comparison, what if they are used together).\n\n* **Training cost**: From the description at around Line 226, it seems that *all* scales of *all* images are used for training, which means the training cost could be a few times of single-scale training. A comparison with specific numbers and some discussions on scalability would be very helpful as training data of state-of-the-art MLLMs are approaching billion-scale.\n\n* **Minor writing issues**: In Table 13, the unit of FLOPs should be in T instead of TB (TB=TeraBytes is for memory). Repeated reference item: Zhang et al 2023a / Zhang et al 2023b.\n\n[1] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" Advances in neural information processing systems 35 (2022): 23716-23736.\n\n[2] Zhang, Renrui, et al. \"LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention.\" The Twelfth International Conference on Learning Representations. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WdrUpqfHRp", "reviewer": "Reviewer_pHpw", "summary": "This paper proposed a multimodal LLM M3 supporting varing number of visual tokens, inspired by the Matryoshka Dolls. By changing the nbumber of visual tokens, M3 can understand the visual content at different granularity. The proposed method is simple and easy to follow, which has been verified by the experiments.", "questions": "- How to extend M3 to any number of tokens. Currently, it can only support a set of token numbers (576->144->xxx). If we can enlarge the set, then switching between them will be smoother.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation is clear. Current LMMs need more and more visual tokens to enhance their performance, the study of token reduction is important for efficient LMMs.\n- The method is simple and easy to implement. Instead of tuning LLM for accepting varing number of tokens, M3 shows that tuning CLIP also works.\n- The main evaluation and ablation analysis confirm M3's effectiveness.", "weaknesses": "- Comparisions with dynamic sampling methods like Token Merging and Chat-Univi [1]. The performance drop is significant when reducing the number of tokens, while dynamic sampling methods like Chat-Univi can even surpass its full token baseline. Besides, M3 can be regarded as a special case of dynamic sampling. I suggest a fair comparison with these methods.\n- High-resolution and long video evaluation and comparisons with other works (LLaVA-HD, SPHINX, LLaMA-VID etc.) . Since these tasks usually requires more tokens, M3 may lead to a better trade-off between performance and computation.\n- Suggest to add speed and computation cost comparisions on the main paper, instead of the appendix.\n\n[1] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AsYvPQlI0g", "reviewer": "Reviewer_rbWX", "summary": "This paper addresses the problem of visual representation in large multimodal models (LMMs) with a method called Matryoshka Multimodal Models (M3). M3 sequentially applies average pooling to the initial visual tokens extracted by the CLIP-ViT to obtain visual representations at different granularity levels. During training, the LMM learns to autoregressively generate the next tokens based on visual representations at each granularity level individually. In inference, the level of visual granularity can be adjusted to balance performance and efficiency. Experimental results indicate that, on a number of benchmarks, M3 achieves performance on par with LLaVA-1.5 and LLaVA-Next, while requiring significantly fewer visual tokens. Additionally, M3 provides a tool for analyzing the visual complexity of vision-language benchmarks by assessing the granularity required to arrive at correct answers. The results reveal that dense visual perception benchmarks, such as TextVQA and DocVQA, indeed require a higher number of visual tokens compared to other benchmarks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-motivated, addressing the important capability of representing visual information at varying levels of granularity. This flexibility enables adjusting the number of visual tokens based on both computational budget and task complexity.\n- The proposed method is effective, demonstrating comparable performance with the baseline LMMs (LLaVA-1.5 and LLaVA-Next) while using significantly fewer visual tokens, across benchmarks that do not demand dense visual perception.\n- The empirical study offers several valuable insights:\n  - A substantial gap exists between the naive use of all visual tokens and the upper-bound performance achieved by selecting the optimal number of tokens for each test instance.\n  - Different vision-language tasks indeed require varying numbers of visual tokens to be addressed.\n  - Reducing the number of visual tokens does not increase the level of hallucination for M3. \n  - The ablation study compressively compares different token reduction methods and suggests the advantage of average pooling.", "weaknesses": "- Although M3 can produce visual representations at multiple granularity levels, the number of visual tokens used at inference must be predefined. In other words, the method cannot adaptively adjust the number of visual tokens for different instances.\n- The baseline methods used for video understanding are relatively weak. For example, recent 7B-scale VLMs have achieved over 60% accuracy on EgoSchema, while the best baseline in this work only reaches 35.8%. M3 would likely benefit from integration with more advanced video LMMs. It would also be better to explore alternative video encoding methods other than \"arranging video frames into a collage\".\n- The method for obtaining Oracle performance is not clearly explained. In line 323, the authors state, \"for each test instance, we select the scale with the fewest tokens that can answer the question correctly.\" However, it is unclear what happens if the model cannot produce the correct answer at any scale.\n- The capability of the LMMs is not taken into account when using M3 as a tool to analyze the visual complexity of vision-language tasks. If the LMMs cannot effectively consume additional visual information, increasing the number of visual tokens may negatively impact performance, even if the task itself requires more visual information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qpfxIaWRMq", "reviewer": "Reviewer_swzi", "summary": "This paper focuses on the VLLM domain and proposes to use average pooling to downsample the visual tokens to a hierarchy of scales for efficient computation.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Presentation is clear and concise\n2. The method is simple but effective\n3. Experiments are well-designed and can support the value of the proposed method\n4. The large number of visual tokens in multi-modal LLMs is a significant and urgent problem", "weaknesses": "1. Whereas the paper claims that the method can ''**adaptively** and efficiently represent visual content\", I think the word **adaptively** is kind of misleading here as it makes readers feel that the method per se includes some dynamic inference strategy. I can understand that the authors hope to express that the users can tradeoff computation for accuracy, but it is important to make it more clear.\n\n\n(**This is actually a weakness of mine instead of the paper**) I am not able to precisely evaluate the novelty of the paper because the VLLM domain is altering from day to day and I have not been closely tracing the frontier papers in the token reduction topic, and I hope other reviews can give better novelty judgments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)