Matryoshka Query Transformer for Large Vision-Language Models
Authors: Wenbo Hu
Zi-Yi Dou
Liunian Harold Li
Amita Kamath
Nanyun Peng
Kai-Wei Chang
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal
vision-language
visual tokens
efficiency
llm
Primary area: machine_vision

Abstract: Large Vision-Language Models (LVLMs) typically encode an image into a fixed number of visual tokens (e.g., 576) and process these tokens with a language model. Despite their strong performance, LVLMs face challenges in adapting to varying computational constraints. This raises the question: can we achieve flexibility in the number of visual tokens to suit different tasks and computational resources? We answer this with an emphatic yes. Inspired by Matryoshka Representation Learning, we introduce the Matryoshka Query Transformer (MQT), capable of encoding an image into $m$ visual tokens during inference, where $m$ can be any number up to a predefined maximum. This is achieved by employing a query transformer with $M$ latent query tokens to compress the visual embeddings. During each training step, we randomly select $m \leq M$ latent query tokens and train the model using only these first $m$ tokens, discarding the rest.
Combining MQT with LLaVA, we train a single model once, and flexibly and drastically reduce the number of inference-time visual tokens while maintaining similar or better performance compared to training independent models for each number of tokens. 
Our model, MQT-LLaVA, matches LLaVA-1.5 performance across 11 benchmarks using a maximum of 256 tokens instead of LLaVA’s fixed 576. Reducing to 16 tokens (8x less TFLOPs) only sacrifices the performance by 2.4 points on MMBench. On certain tasks such as ScienceQA and MMMU, we can even go down to only 2 visual tokens with performance drops of just 3\% and 6\% each.
Our exploration of the trade-off between the accuracy and computational cost brought about by the number of visual tokens facilitates future research to achieve the best of both worlds.

Matryoshka Query Transformer for Large Vision-Language Models
Wenbo Hu Zi-Yi Dou Liunian Harold Li Amita Kamath
Nanyun Peng Kai-Wei Chang University of California, Los Angeles
{whu, zdou, liumian www.ma.aaaoood....mm hww.odl.oo..... 1i, 8ssa www..lll....oo......eeeeee...eee edu
http://gttubb.omm/Mq1LLLaaa
Abstract
Large Aision-Language Models (LVLMs) typically encode image into fixed number of visual tokens (e.g.. 576) and process these tokens with a language model. Despite their strong performance, LVLMs face challenges in adapting varying computational constraints. This raises the question: can we achieve flexibility the number of visual tokens suit different tasks and computational resources? We answer this with an emphatic yes. Inspired by Matryoshka Representation Learning, we introduce the Matryoshka Query Transformer (MQT), capable of encoding an image into m visual tokens during inference, where m can be any number up to a predefined maximum. This achieved by employing query transformer with M latent query tokens to compress the visual embeddings. During each training step, we randomly select m S .M latent query tokens and train the model using only these first m tokens, discarding the rest. Combining MQT with LLaVA, we train a single model once, and flexibly and drastically reduce the number of inference- time visual tokens while maintaining similar or better performance compared to training independent models for each number of tokens. Our model, MQT-LLAVA, matches LLaVA-1.5 performance across 11 benchmarks using maximum of 256 tokens instead of LLa VA's fixed 576. Reducing to tokens (8x less TFLOPs) MMBench. On certain tasks
only sacrifices the performance by 2.4 points such as ScienceQA and MMMU, we can even go down to only 2 visual tokens with performance drops of just 3% and 6% each. Our exploration of the trade-off between the accuracy and computational cost brought about by the number visual tokens facilitates future research to achieve the best of both worlds.
Introduction
Recent work in Large Vision-Language Models (LVLMs) OpenAl 2023 Liu ct al. 2023b Bai et al.
2023) has shown remarkable performance across a broad range of languae tasks (Huang
2024a Li al. 2023b These LVLMs typically consist of a vision encoder to embed images into grid features, which are fed into a Large Language Model
al. 2023 Chen et al.
2023 Cai et al.
(LLM) (Touvron al.
2023 Chiang et al. 2023) for processing and reasoning alongside text input.
key research question how to transform these raw visual embeddings into the visual tokens that are fed into the L.M.. Prior work either directly projects the grid features with multi-layer perceptron (MI.) (Liu et al. 2023b) or compresses the grid features into fewer tokens with query transformer resampler (Li et al. 2023a; Dai et al., 2023 Bai et al. 2023 Ye et al. 2023 Alayrac ctt al. 2022). However, these models all need to pre-determine how many tokens an image is worth, and set fixed number for all images. Finding flexible number that adaptively strikes a balance
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Average Over 11 Benchmarks
2X Speed-up
500
Average Score (%)
MMBench Score (%)
Number of Visual Tokens
64
400
63
BX Speed-up
LLavA-1.5
62
2.4 Accuracy Drop
-300
200
MOT-LLaVA (Ours)
60
#Visual Tokens (Baseline)
100
LLavA.1.5
59
144
576
1100
4 1
10e
10
Number of Visual Tokens
TFLOPs
Figure I: Our model, MQT-LLAVA, matches LLaVA-1.5 performance on 11 benchmarks using only 256 visual tokens instead of 576. We achieve a 2x speed-up with 256 tokens and 8X speed-up in TFLOPs using 16 tokens with only a 2.4 performance drop compared to LLaVA-1.5 on MMBench.
between efficiency and performance is difficult. More visual tokens encode more information, but come at a higher inference cost, as the complexity of the transformers used in these LVLMs scales quadratically with the number of input tokens. Additionally, not all applications require or allow the same token budget: some applications have limited computational resources, necessitating a lower token budget to ensure real-time processing. In practice, most best-performing LVLMs choose a fixed, large number of visual tokens per image (e.g., 576 for LLaVA-1.5) without the ability to adaptively adjust the visual token allocation at deployment time.
In this work, inspired by Matryoshka Representation Learning (MRL) (Ksuupati et al. 2022 Kudugunta al. 2023), we introduce Matryoshka Query Transformer (MQT), simple way train a single LVLM that supports adaptively changing the number of visual tokens at inference time. We query transformer (Li et al. 2022. |Alayrac et al. 2022) with M latent query tokens to transform grid features into visual tokens. Crucially, during each training step, we train the model using only the first m latent query tokens while dropping the rest, where is randomly selected within the range of M. With such a tail-token dropping strategy, the query tokens form a Matryoshka structure. Intuitively, the significance of each token correlates with its placement within this nested structure. During inference, we have the flexibility to selectively utilize solely the initial m visual tokens.
We combine MQT with VA-1:::: the resulting model, MQT-LLAVA, is able match LLaVA-1.5 performance across 11 benchmarks using only a maximum of 256 tokens, instead of LLaVA's fixed 576. When the maximum number of tokens is dropped drastically to only 2 tokens, MQT-LLAVA performance drops by only 3% on ScienceQA and 6% on MMMU. Finally, we study the performance of 2, 4, 8, 16, 36, 64, 144, and 256 visual tokens during inference across 11 benchmarks, and offer a trade-off in the selection of visual tokens that balances achieving the highest accuracy with minimizing computational costs different tasks. Interestingly, find that changing the number of visual tokens impacts different tasks very differently. For instance, tasks involving language- based reasoning and subject-level scientific knowledge achieve excellent performance with only few tokens, whereas complex open-ended visual question tasks that involve rich local information details require a larger number of tokens.
In summary, we make the following key contributions:
We introduce Matryoshka Query Transformer (MQT), which allows for a flexible choice of the number of visual tokens and accommodates varying computational constraints in different tasks.
Leveraging MQT, build MQT-LLAVA, vision-language model that matches the perfor- mance of LLaVA-1.5 using less than half the number of visual tokens, and outperforms it in 6 out of 11 benchmarks.
We further explore the performance and computation trade-offs across 11 tasks and demonstrate that a significant speed-up up can be achieved with minimal performance drop by reducing the number of visual tokens (e.g., 8X fewer TFLOPs with 2.4 points drop MMBench).

---

Pre-trained LLM
Elastic Visual Tokens
Query Transformer
Grid Features
Vision Encoder
Image
Our model employs query transformer to encode images visual tokens. We randomly select the first m tokens during training, and enable flexible choice of any m number under M during inference, where M is the maximum number of initialized tokens.
Figure 2:
2 Matryoshka Query Transformer
Preliminary: Matryoshka Representation Learning (MRL). MRL (Kusupati al. 2022
Kudugunta et al. 2023) involves training models with nested dimensions learn representations at multiple granularities, larities, enabling adaptive deployment per computational constraints. MRL defines series of models f1, f2...., fAs with the same input and output space but growing hidden dimensions.
The name "Matryoshka" comes from the fact that the parameters of fm are contained by fm+1- For example, in Kudugunta et al. (2023). {fm} are series of Transformers with the same depth but different widths. Consider specific Feed Forward Network (FFN) block in JM that has d neurons in the hidden layer. Then, the FFN block in Sm will contain the first dm neurons, and d I d2 < ... S dy. MRL then trains these models jointly with the following loss:
Σ Com L (fm(())) y).
(I)
where L is the loss function and y is the ground truth label. Note that for each training step, MRL performs forward and backward passes for all M models, inducing significant training overhead compared to training one model. After training, MRI. can perform inference with any hidden dimension digM, enabling flexible deployment based on specific needs. MRL is our motivation to train LVLMs that can perform inference with a flexibly selected number visual tokens.
2.1 MQT-LLAVA
We first explain how encode images with query transformer, then discuss training paradigm.
Encoding images with Query Transformer. We employ a query transformer-bass architecture to extract visual tokens from images following previous work (Li et al.] 2022; Bai et al. 2023). first processed by image encoder and then flattened into H x W grid features G = [gu, ... gIw, gin. ... gnw]. Then, query transformer Q is applied compress the grid features to M visual tokens. Specifically, Q assumes set of latent query tokens Z = [Zi,.... ZMr as input, where M is usually smaller than / x W. The query tokens cross-attend to the grid features and compress the information into the query tokens. The final-layer query tokens become the visual tokens that are fed to large language model together with the input text tokens. I.e., V QZ. G). linear projection layer is added in the end to match the hidden size of the language model]
Specifically, input image
Matryoshka Query Transformer. To enable elastic inference, given the M latent query tokens 2i...., 2... each training step, feed only the first query tokens the query transformer Q. Subsequently, we obtain only m visual tokens from the query transformer. m can be any number equal smaller than the maximal token number M. In practice, choose m from linear set of maximum dimensions, in increments of 2, e.g. m can be any number in {2, 4, 6, ..,, 252, 254, 256}
Ye et al. 2023 that first applies projection followed by attention. empirically find that "attention then projection architecture performs better (c.f. (4.3)
Unlike previous work Bai et al. 2023

---

when M = 256. From a training efficiency perspective, our approach uses, on average, half of the visual tokens compared to the original query transformer-bast based models.
Formally, given an input image with its corresponding text question and answer y, at each training step, randomly select and feed the first latent tokens Zi:m and the text question y the model. We compare the model output and y and minimize
Cm L LV q)i y), where V = Q(= G),
(2)
LM is the language model, L is the language modeling loss function, and Gm is a constant coefficient to control the weight of different numbers of visual tokens, which is always set to 1 in our setting.
Discussion. Here we discuss several interesting properties of MQT. (1) Unlike the original ma- tryoshka representation learning that maintains nested structure in the parameter space. we specif- ically target LVLMs and make the visual tokens Matryoshka-liks (2) Despite discarding the tail M - m tokens during each training step, models trained with this token-dropping strategy perform comparably to those trained consistently with all M tokens, as long we utilize the entire M tokens during inference for both models. (3) Unlike the original MRL, which performs forward and backward passes for all M configurations in each step, we now select just one model configuration per training step, significantly cutting training costs. (4) Our cost reduction enables training across a broader spectrum of m values, facilitating the training of models with more diverse range of choices compared to the original MRL's limited scope.
3 Experiments
We first introduce the implementation details of our query transformer architecture (3.1)) We then show the empirical performance of our approach compared to state-of-the-art models across 11 benchmarks (322 Finally, we further study the performance--effccinncy trade-off ($3.).
3.1 Experimental Setup
MQT-LLAVA Implementation Details. We implement our models based LLaVA-1.5 (Liu et al. 2023a), except that we use our Matryoshka Query Transformer instead of an MLP to obtain the visual tokens. The MQT is single-layer Transformer with cross-attention. Following Liu et al. (2023a), we select CLIP ViT-L/14 (Radford et al. (021) as our vision encoder, supporting 336x336
image resolution, and Vicuna-v1. (Chiang et al. 2023) as our LLM. As studied in Hu al. (2023); Zhu et al. 2023); : Liu et al. (2023b), we adopt a two-stage training approach. We train only the query transformer in the first-stage alignment, using VA-558K for 1 epoch with a batch size of 256 and a learning rate of 1e-3, We then fine-tune both the query transformer and LLM using VLa for 2 epochs with a batch size of 128 and a learning rate of 2c-5. All training is on 8xA6000s, with and 30 hours per stage, respectively. We apply MQT during the second stage (c.f. $4.3)
Baselines. As shown Table D we compare our model with LLaVA-1.5 (Liu et al. 2023a) and our model's baseline LLaVA query transformer (QT-LLaVA) which is trained with a fixed number of 256 visual tokens across all training stages. We also list other models' results for comparison, including BLIP-2 (Li et al. 2023a), InstructBLIP (Dai et al. 2023), Shikra (Chen et al. (023). and Qwen-VI VL (Bai et al. 2023).
IDEFICS (DEFICC) 2023)
Evaluation Benchmarks. We evaluate our model across 11 mainstream benchmarks, including VizWiz (Guari et al.] 2018)) ScienceQA-IMG (Lu et al. 2022), VQA-v2 (Goyal et al. 2017).
GQA (Hudson and Manning 2019). POPE (Li al.) 2023c). MME Perception (Fu et al. 2023).
MME Cognition (Fu et al. 2002). MMBench (Liu et al. 2023c). LLaVA -A-Becchhhhh An-the-Wiild (Liu
al.
2023b), and MM-Vet Yu et al. 2024).
3.2 Main Results
Table I presents the results of MQT-LLAVA with inference visual token budgets of 2, 4, 8, 16, 36, 64, 144, and 256. We refer to the baseline approach, where the model trained with a fixed number of visual tokens across all training stages, as LLaVA Query Transformer (QT-LLaVA), MQT-LLAVA

---

Method
LLM
Res.
#Tokens
VizWiz SQA'
VQA
GQA
|POPE
MME"
MME
MMMU MMB LLAVAW
MM-V Avg
BLIP-2
Vicuna-133 13B
224
61
41.0
85.3
1293.8
38.1
22.4
InstructBLIP
Vicuna-7B
224
32
34,5
60.5
49.2
1084
229
30.6
60.9
26.2
InstractBLIP
Vieuna-13B
224
32
33.4
63.1
49.5
78.9
1212.8
243
33.8
58.2
25.6
Shikra
Vicuna-131
224
256
77.4*
58.8
IDEFICS-9B
LLaMA-7B
224
64
35.5
50.9
38.4
48.2
IDFFICS-80R
MaMA-65B
224
60.0
45.2
Owen VL
Qwen-7BB 7B
448
256
35.2
67.1
78.8"
59.3'
38.2
Owen- VL-Chat
Qwen-7B
68.2
78.2"
1487.5
LLaVA-1.5
Micuna-1.5-72
336
576
50.0
66.8
78.5
62.0"
85,9
1510.7
316.1
34.7
64.3
63.4
30.5
59.2
QT-LLaVA
Vicuna-1.5 1.5-B
336
256
51.1
68.1
76.8'
61.5'
84.1
1431 /2 348. 22
34.3
64.0
63.9
27.9
58.8
MOT-LLAVA
Vicuna-1..5 1.5-7B 336
256
53.1
67.6
76.8"
61.6"
84.4
1434,5
353.6
34.8
61.33
64.6
29.8
59.4
MOT-LLAVA
Neu-a-11.--BB 336
144
67.5
76.4"
83.9
1446.4
351.8
34.4
29.9
MQT-LLAVA
Viuuna-I.5-78
336
64
51.5
67.0
75.3'
60.0"
83.6
1464.3
352.9
34.4
63.5
59.4
28.9
58.3
MOT-LLAVA
Vicune-1.5-7B 336
36
51.0
66.8
73.7*
58.8*
81.9
1416.3
349.3
34,4
63.4
59.6
27.8
57.4
MOT-LLAVA
Vicuna-I 1,5-7 336
16
49.8
67.5
71.1"
57.6'
80.8
1408.5 5 349.3
33,6
55.2
25.3
MQT-LLAVV
Vicuna-1.55 1.5.7B 336
8
49.4
66.2
67.2"
55.5'
79.4
1282.2
323.6
33.1
58.6
51.4
21.3
53.3
MOT-LLAVA
Vicuna-1. 1.5-7B 336
65.1
77.6
1176.1
296.8
32.8
20.2
MQT-LLAVA
Vicuna-1.5-78 336
2
48.5
65.0
61.0"
50.8"
74.5
1144.0
268.9
32,5
$4.4
41.7
19.5
49.0
Table 1: Comparison with offe-of-the-art methods on 11 vision-language benchmarks. Our model (MQT- lLAVA) with up to 256 tokens achieves on par or better than LLaVA-1.5 performance across 11 benchmarks, outperforming on 6 of 11 benehmarks. We mark the best performance in bold and the sccond-bess underlined. #Tokens the number of visual tokens used during inference. Avg is the normalized average across 11 MannnceQAIMGGGG MME": MME Perception, MME MME Cognition, MMB: MMBench, LLaVA W: LLaVA-Bench (In-the-Wild) "The training images of the datasets are observed during training.
benchmarks, out of 100, Benchmark names are abbreviated for brevity: SQA':
outperforms the baseline QT-LLaVA with 256 tokens in 9 out of 11 benchmarks. One possible explanation is that by enforeing our model to only see fewer tokens during training, the stricter constraint helps the model generalize better to unseen tasks. This is especially evident in the higher performance on VizWiz. When compared to open-source state-of-the-ar models, our model with 256 tokens achieves par or better than LLaVA-1.5 performance with 576 tokens across 11 benchmarks, outperforming it in 6 out of 11 benchmarks. Even with 64 tokens, our model falls short of LLa by only 0.9 points on average. When drastically drop to only 2 tokens, our score falls by only 3% on ScienceQA and 6% on MMMU. While directly adding a query transformer to LLaVA degrades the performance, strategy achieve comparable better performance than LLa VA.1.5.
We explore performing inference using a variety of numbers of visual tokens, including 1) an ex- tremely low number of tokens; 2) a number of visual tokens unseen during training, As shown in Figure 3 MQT-LLAVA with only visual tokens outperforms InstructBLIP (Vicuna-7B). which is based on Q-Former (Li et al. 2023a)
Average Over 8 Benchmarks
MOT-LLaVA InstructBUP-78
Average Score (%)
48
using 32 visual tokens. This demonstrates the effectiveness of our model in compressing vi- sual information, pointing to its potential use for applications in computation-heat heavy tasks. For an unseen number of visual tokens, we pick a ran- dom number of visual tokens: 77, and include its results in Appendix C Despite never being explicitly trained for this number of tokens, our model can generalize any number within 256 during inference, demonstrating a further benefit of our elastic approach.
42
V2
Figure With only visual tokens, MQT-LLAVA outperforms InstructBLIP (which uses 32 visual tokens) on all benchmarks is evaluated on.
3.3 Computational Efficiency
To demonstrate our computational efficiency, we compute TFLOPs when running MQT-LLAVA on MMBench with 8, 16, 36, 64, 144, and 256 visual tokens, compared to LLaVA with 576 tokens. As shown in Figure [ we are able to achieve significant speed-ups with little-to-no performance loss: our model with 256 and 144 tokens respectively achieve 2x and 3x speed-up compared to
5

---

Input Image
"Is there a man ironing behind the car? Answer: Yes"
16
258
Input Image
"How many cats there the image? Answer: 2"
16
64
256
Input Image
"List all the objects on the desk
256
Figure 4: Grad-CAM visualization of randomly picked token from using 8, 16, 64, 256 visual tokens, respectively, encode image. The model effectively concentrates on high-level concepts using fewer tokens and delves into low-level details with more tokens. The complete input to the third image is "List all the objects the desk. The objects on the desk include computer monitor, keyboard, a mouse, a cell phone, and pair of headphones".
LLaVA-1.5 while maintaining the same or even better performance; and when using 16 tokens, we achieve an speed-up with performance drop of only 2.4 points.
4 Analyses
To better understand the meaning of visual tokens and to systematically study the number of tokens required by different langua-a tasks, we investigate two key questions: (1) How does the focus of the model change with varying numbers of visual tokens? (4..)) and (2) How do different numbers of visual tokens impact various tasks? (34.2)
4.1 How does the focus of the model change with varying numbers of visual tokens?
To explore what visual information each token encodes, we utilize Grad-CAM (Selvaraju al. 2017) to visualize the focus of visual tokens. As illustrated in Figure 4 we qualitatively analyze the results of using 8, 16, 64, and 256 tokens.
We observe that the model's focus changes with the number of tokens used. When using a few tokens (c.g.. 8), the model accurately concentrates on global visual concepts related the question. As the number of tokens increases (c.g., 256), the model not only attends to the relevant objects but also delves into localized details. For example, in the third image, with tokens, the model focuses on the monitor. With 16 tokens, includes both the monitor and the mouse. With 64 tokens, it highlights the monitor and keyboard. Finally, with 256 tokens, the model encompasses several objects, including the monitor, keyboard, and cell phone. In the examples from the first and second images, our model effectively focuses on the man ironing behind the car and the two cats, even with only tokens. The impressive qualitative results, especially those using only few tokens, demonstrate the effectiveness of our approach and the strong capabilities obtained despite using minimal number tokens.
4.2 How do different numbers of visual tokens impact different tasks?
When using varying numbers visual tokens during inference, we observe that the model's perfor- mance change varies across different tasks.

---

Score (80
Scare (%1
Score (161
120
300
MOTILLIAA
MOTALAAS
200
Visual
Number Visual
Number Visual
Visual
Figure 5: The number of visual tokens impact different tasks differently (x-axis is in log-scale). Our model's 's performance on ScienceQA, MME-Cope Cog-Cognit and MMMU is remarkably robust to token reduction. For full visualization all 11 benchmarks, see Figure and Figure Appendix.
if 20:
if 20:
print print("Hello")
print
else: print "Korld")
else: print ("World")
The image shows python code. the output of the code 'Hello"? Answer: Yes
Question:
7 X 20
Question: All apples are shown in the picture. My brother and divide the apples equally. May have two apples? Answer: No.
Question: Is the answer to the arithmetic question in the image 340?
Answer: Yes
Figure 6: Examples from MME Cognition. Grad-CAse results are from using 16 tokens which answered all the questions correctly.
Tasks requiring a large number of visual tokens. Tasks that require fine mne-grained visual under- standing and deep reasoning across multiple areas of the image naturally demand a higher number of visual tokens for optimal performance. When the number of visual tokens decreases, the encoded image information is reduced, leading to performance degradation. This trend is evident tasks such VQAv2, GQA, VizWiz, MMBench, LLL and MM-Vet. As illustrated in Appendix Figure 8 the performance on these tasks gradually declines as the number of visual tokens decreases from 256, with more rapid decline observed when the tokens are further reduced.
Tasks robust to visual token reduction. contrast, for several benchmarks primarily targeting the visual perception skills of models, performance remains consistent when gradually reducing the number of visual tokens until threshold is reached. Beyond this threshold, performance drops significantly (see Figure and Appendix Figure 8). This "turning point" is observed in benchmarks such as MME Cognition, MME Perception, POPE, and MMMU.
For instance, in UME-Cognition (see Figure 6 tasks involving commonsense reasoning, code reasoning, and numerical calculation can be performed effectively with few 16 visual tokens, allowing the model to focus on the relevant image sections. Similar results are seen in other tasks, like the hallucination question "Is there car in the image?" from POPE. However, once the "turning point" is reached, further reducing the number of visual tokens prevents the model from attending to the correct objects, leading to a sharp decline performance.
Another notable observation comes from ScienceQA and MMMU, which contain subject-specific questions from school curricula. The model's performance these tasks remains robust despite decrease in visual tokens, achieving scores of 65.0 and 32.5, respectively, with only 2 tokens. This suggests that the reasoning required for academic questions is primarily conducted by the language model (LLM)); even with minimal visual hints, the LLM can interpret the image content and perform the reasoning tasks effectively.
When are fewer visual tokens better? As shown above, MQT-LLAVA with 16 tokens can achieve better performance ScienceQA compared MQT-LLAVA with 144 tokens. To understand

---

Input Image
#Visual Tokens: 16 (Correctv/)
#Visual Tokens: 144 (Wrong x)
taraes
Quastion: Which property do these three objects have in common? A.shiny, B.slippery, C.opaque. Anawer C
Input Image
#Visual Tokens: 16 (Wrongxx)
#Visual Tokens: 144 (Correct/)
Question: Which is highlighted? A.Texas, B.Oklahoma, C.Virginia, D.Georfia. Answer:
Figure 7: Comparison of correct and failure cases in 16 vs 144 visual tokens on Science QA (test-set).
Method
VisWiz SQA'
VOA
GQA
POPE
MME"
MME
MMMU
MMB
LLaVA"
MM-Vet
Avg
QT- (Baseline)
S11.1
61.5"
5888
MQT-LLAVA (Ours)
53.1
67.6
76.8"
61.6"
84.4
1434,5
353.6
34.8
64.3
64.6
29.8
59.4
Log-hased hased Mairyushka Tokens
51.2
67.4
75.6'
60.3
83.2
1418.9
314.1
32.8
62.6
59.2
27.3
57.3
w/ Project Attention
50.5
66.8
73.4
57.1
82.3
1382.8
317.5
32.7
60.0
29.5
w/ Fust-stage training with Query Transforec
51.6
67.2
75.9"
60.5"
82.6
1378.6
295.4
33.2
63.1
56.5
26.8
56.7
Table For simplicity ablation studies, we evaluate all the models with 256 visual tokens. All models trained with the same hyperparameters.
why fewer tokens may benefit this task, we qualitatively analyze instances where MQT-LLAVA succeeded with visual tokens, but failed with 144. We show representative example in Figure 7 MQT-LLAVA with 16 visual tokens attends to all three objects, allowing it to understand their mutual relationship and answer question correctly. On the other hand, with 144 visual tokens, MQT-LLAVA focuses various portions of the image and attend each object independently. This discourages the model from reasoning with the common attributes among the three objects, thus predicting the wrong answer. In summary, fewer visual tokens seems to be preferable when fine-grained visual understanding is not required.
However, it should be noted that using fewer tokens is not always better in this case. As shown in Figure 7 MQT-LLAVA with 144 tokens precisely identified state of Virginia on the map and answered the question correctly. Whereas 16 tokens concentrated on another region which potentially confused its final prediction, lacking the abilities of distinguishing local details of the geographic shape on the map.
4.3 Ablation Studies
We ablate several design choices across 11 benchmarks Table 2 Each ablation independently modifies our best variant, MOT-LLAVA, create new variants. (i) linear vs. log-based token number selection. We replace our linear growth elastic tokens, i.e., m E {2, 4, 6..., 252, 254, 256} to the log-based approach of MRL, i.e., m (2, 4, 8, 16,..., ... 128, 256}. This results in average accuracy of 57.3%, 2. % lower than MQT-LLAVA, validating our hypothesis that gradually compressing the visual tokens helps the model perform better than loseb choices. (ii) query transformer architecture. As mentioned in 2 we choose to first perform cross-attention between query tokens and visual features, then project the learned visual tokens to the LLM. We call this technique "attention then projection". The alternative variant is "projection then attention", which achieves lowest average performance, with a score of 56.6%. This suggests that directly applying the attention mechanism helps preserve the rich grid features, making them better projected to the LLM. (iii) first- ftatt-stage pretraining with query transformer. As mentioned 13.1 we choose to apply our elastic training paradigm only during the second stage. Experimental results demonstrate that adopting elastic training during the first stage leads average performance dropped by 2.7%. We hypothesize that the first stage aims to align the randomly initialized query tokens with lasion-language awareness.

---

Therefore, it is important to train all 256 tokens with this prior knowledge before reducing the number of tokens in the second stage.
5 Related Work
5.1 Large Vision Language Model
Large Vision Language Models (LVLMs), pioneered by Liu et al. (2023b); Zhu et al. (2023); Dai
al. (2023): Yin et al. (2023) have successfully showcased promising results on a wide variety of
laigualangu perception and reasoning tasks. Recent works further expand the capabilities of
LVLMs region-level image understanding (Huang et al. 2023 Peng et al. 2023 Chen et al. 2023
Cai et al., 2024a). video understanding (Zhang et al. 2023 Li et al. 2023b Jin et al. 2024) and
3D understanding (Hong al.) [003] Szot et al. (024). These models mostly consist of vision
encoder and an LLM aligned by a vision-language connector module, which can be an MLP (Liu
et al. 2023b Hu al. 2023), Q-Former or queries through cross attentions (Dai al. 2023 Bai
et al. 2023 Ye et al. 2023). or Resampler (Alayrac et al.] 2022). The number of visual tokens output by these modules can be very large, especially for higher image resolutions, multiple frames in video tasks, and multiple images for in-context learning. In this paper, we propose a training paradigm
that can dynamically choose a number of visual tokens that adapts variable computation costs at inference time.
5.2 Efficient Vision Transformers
Reducing the computational cost of LVLMs at deployment an active area of research. Several works, e.g.. TinyLLaVA (Zhou et al. 2024) and LLa (Zhu et al.) 2024). reduce the size of the LLM backbone by replacing with smaller one, e.g., Phi-2 (Javaheripi al. 2023). MobileVLM, (Chu et al.) (023) and Mobile VLM-22 (Chu et al., 2024) focus on compact architecture design and training paradigms specifically for mobile usage. In these cases, the computation reductions come from reducing the size of either the LLM or vision encoder backhones, whereas our method focuses on increasing LVLM efficiency by dynamically reducing the number of visual tokens.
long-standing issue with vision transformers is that the attention mechanism introduces computa- tional complexity that scales quadratically with the input tokens. Vision Longformer (Zhang et al.] 2021) adopts sparse attention (Kitaev et al. 2020) to speed up vision transformers for larger inputs. Other works design various strategies retain the most informative tokens and reduce the number
of visual tokens at the inference stage (Fayyaz et al., 2021 Liang et al./ 2022 Rao et al. 202
Bolya al.] 2023] Yin al.. 2022). Most similar to our work is SparseFormer (Gao et 2024)
which employs gross-attention to learn sparse representations of both latent tokens and Rol descriptor
tokens. In this work, we use the most simple query transformer architecture to actively control the number of visual tokens and explore the impact of reducing the number of visual tokens in LVLMs.
2024b) also studies matryoshka-style visual tokens, and designs 5 scales of pooling layers to control the granularity of images. Different from their work, we introduce a query transformer to extract visual tokens, enabling more flexible choice of any number of visual tokens under predefined maximum. Their work corroborates our findings by demonstrating robust performance and efficient use of a minimal number of visual tokens.
Concurrent work (Cai et al.
6 Conclusion
In this work, we present MQT-LLAVA, single vision-language model that enables elastic inference on various downstream tasks and computation resources. We demonstrate that our model achieves performance comparable to or better than training with a fixed number tokens. MQT-LLAVA matches the performance of LLa LLaVA-1. 1.5 across 11 benchmarks using less than half the number of visual tokens, and outperforms LLa in 6 out of 11 benchmarks. We achieve an 8x less TFLOPs when reducing to 16 tokens while only sacrificing the performance on MMBench by 2.4 points. We hope our exploration of the trade-off between the accuracy and computational cost caused by the number of visual tokens will facilitate future research to achieve the best of both worlds.

---

Acknowledgments
We would like to thank members of UCLA NLP and PLUS Lab for their helpful comments. We also thank the reviewers for their valuable reviews. This work was supported by Amazon gift grant, ONR grant N00014-23- 1--27800 U.S. DARPA ECOLE Program No. #HR001 12390060, and an SRA from Optum Labs. The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkkwski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. 2022. Flamingo: visual language model for few-shot learning In Advances in Neural Information Processing Systems, volume 35, pages 23716-23736. Curran Associates, Inc.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, 2023. Qwen-vl: A versatile vision-language model for understanding localization, text reading, and beyond ArXiv preprint.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token merging: Your vit but faster In The Eleventh International Conference on Learning Representations.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. 2024a. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition.
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. 2024b. Matryoshka multimodal models)
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023, Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. 15195.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna An opensource chatbot impressing gpt-4 with 90% chatgpt quality, ArXiv preprint.
Xiangxiang Chu, Limeng Qiao, Xinyang I Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. 2023. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv: 16886.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, al. 2024. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision. language models with instruction tuning
Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Juergen Gall. 2021. Adaptive token sampling for efficient vision transformers) In European Conference on Computer Vision.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394
10

---

Ziteng Gao, Zhan Tong, Limin Wang, and Mike Zheng Shou. 2024. Sparseformer: Sparse visual recognition via limited latent tokens In The Twelfth International Conference on Learning Representations.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR).
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3d-llm: Injecting the 3d world into large language models. NeurIPS.
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. 2023. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023. Language is not all you need: Aligning perception with language models
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR.
IDEFICS, 2023. Introducing idefics: An open reproduction of state-of-the the- visual language model. hoo/blog/idefics
https://huggingfacce httpg//fa...acooo//aoa/aa///e/ee
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog.
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. 2024. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXivv2402.03111.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, Prateek Jain, et al. 2023. Matformer: Nested transformer for elastic inference. arXiv preprint arXiv:2310.07707.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Hlward--Syyder Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning In Advances in Neural Information Processing Systems, volume 35, pages 30233-30249,
Curran Associates, Inc.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models In Proc. of ICML.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified lasion-language understanding and generation. In ICML.
Yanwei Li, Chengyao Wang, and Jiaya Jia. 2023b. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:231 1.11.117743
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Rong Wen. 2023c. Evaluating object hallucination in large vision-language models In Proc. of EMNLP.

---

Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. 2022. EVit Expediting vision transtormers via token reorgamizations: In International Conference on Learning Representations.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual
instruction tuning ArXiv preprint.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. 2023b. Visual instruction tuning In
Advances in Neural Information Processing Systems, volume 36, pages 14892-34916 Curran
Associates, Inc.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023c. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 0rXXiv22307.06281.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn explain: Multimodal reasoning via thought chains for science question answering. NearIPS.
OpenAl. 2023. Gpt-4 technical report
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.1443 14824.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0 000000000
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dy- namicvit: Efficient vision transformers with dynamic token sparsification In Advances in Neural Information Processing Systems, volume 34, pages 13937-13949, Curran Associates, Inc.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via muuieent-based localization. In ICCV, pages 618 626.
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Met- calf, Natalie Mackraz, Devon Hjelm, and Alexander Toshev. 2024. Large language models as generalizable policies for embodied tasks. ICLR.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. /mplug-owl: Modularization empowers large language models with multimodality. ArXiv preprint.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. 2022. A- VIT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306. 2304.199..
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR.
Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLaMas An instruction-tunee aadio-visua language model for video understanding In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543-553, Singapore. Association for Computational Linguistics.
12

---

Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. 2021. Multi-scale vision longformer: new vision transformer for high-resolution image encoding In 2027 IEEF/CVF International Conference on Computer Vision (ICCV), pages 2978-2988.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. 2024. Tinyllava: A framework of small-scale large multimodal models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint urXiv:2304.1059/
Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. 2024. Llava-phi Efficient multi-modal assistant with small language model
13

---

A Broader Impact
The deployment and release of MQT-LLAVA carry both potential benefits and risks. These consid- erations include visual aspects well common issues found in existing LLMs like Alpaca and Vicuna. Since MQT-LLAVA is built on LLaMA, Vicuna, and CLIP, inherits certain challenges associated with LLMs and vision encoders.
Hallucination
Similar to other LLMs, MQT-LLAVA might produce outputs that are not based on factual information or input data. This raises concerns about the accuracy of inferences, particularly critical applications such as medical fields.
Biases Biases present in the base models can be brought to MQT-LLAVA, stemming from both the vision encoder (CLIP) and the language decoder (LLaMA/Vicuna). This may result in biased outcomes or unfair representations of diverse content.
Energy Consumption We followed LLaVA-1.5's datasets (smaller datasets compared to other methods) in training model, which makes energy consumption is not primary concern.
B Limitation
Despite our remarkable performance, one limitation of our work is that the maximum number of tokens MQT-LLAVA can accommodate inference time is 256. We leave the exploration of using larger numbers in inference than training time to future work.
C Additional Results
We include how number of visual tokens impact the different tasks differently across all 11 bench- marks in Figure and Figure
We present the results of choosing a random number of visual tokens, 77 as shown in Table 3 to demonstrate our flexibility in selecting any number of tokens during inference.
To demonstrate that the visual tokens used for visualization in Figure 4 are not cherry-picked, present all the first eight tokens in Figure 10
Avcruy over 11 bendhmaks
VOA x2
68)
Scisnn GK IGG
64-
604
Aneegge (%6)
(46)
66
Score
VOT
Myn mi HLAGEE
14:056
36 64
14056 576
16 36
16456 576
16 36 14036 576
Number Visual
Numbar Visual
Number Visual
Number of Visual Tokens
MVV-Cogsstion
1500
240
Skare 19a)
1600
120
20
70
POTLLANK
1200
WC
200
MOTALLAN
76-
MORLLAM
74-
164250
1100
260
72
Numbber Visusi Tokens
Number Visual
Numbbr Visual
Number Visual Tokens
IVNU
vitberch
LLankkdench
32
0%
20
Score (N4)
3/
Score 12
70
18
55
Jute 6
LLAN 15
ULN 13
4 16. Mi 14.0 576
16
36 66 TRDis 576
à 16
36
1é0ts 536
16 36 64 TROSE 576
Number of Visual Tokons
Numbar Visual Takons
Number of Visual Takens
Numbar of Visual Tikans
Figure 8: The number of visual tokens impact the different tasks differently across 11 benchmarks. We log scaled x-axis for readability.
14

---

Avouge one bondhmalo
Anergge Score (15)
55-
VOTLLEAN
HOT
Mgn 6n
50
Number
Numbar Visual
Number
Venkte
MV AVE-Serrcptioo
NME WVE-Coggition
POFC
1500
12-
1002
170
80-
1300
200
76-
Lawh22
Link-33
1300
260
1004 144 256
2061 264
216
575
2004
14c
250
376
2064 1-4
576
Number Visusi Tokens
Number of Visual Tokens
Number Visual Tokens
Number of Visual Tokeess
HONN
50-
viiberch
70-
LLatk-derch
32-
VeN-Vt
Score (36)
26-
23
26
50
22-
POTLLAR LLAA333
JUWK
MOTALOR
20-
Mon Lank (LAW
3964 144
256
2661 144 236
2064
146
516
M661 164
576
Number Visual
Number Visual
Number Visual
Numbar Visual Tokens
Figure 9: The number of visual tokens impact the different tasks differently across 11 benchmarks. No log scaled on the x-axis is applied.
Method
LLM
Res. #Tokens VizWiz SQA!
VQA
GQA
[QOPE MME?
MME
MMMU
MMB
LLaVA W
MM Vei Aug
QT-LLaVA
Vicuna-1.5-7B 336 256
51.1
76.8*
61.5"
61.5"|84.1
1431.2
348.2
34.3
64.0
63.9
27.9
MQT-LLAVA Vicuna
Soanaa-1.5:7P 1.5:7B 336 256
53.1
67.6
76.8"
61.6"
84.4
1434.5
353.6
34.8
64.3
64.6
29.8
59.4
MOT-LLAVA Vicuma-I_5-7B
76.4"
61.4"
351.8
34.4
MQT-LLAVA Vicuna-1 5-78 336
77
51.6
67.1
75.8'
60.4"
83.6
1457.0
336.1
34.0
64.0
59.9
29.3
58.3
MQT-LLAVA Vicuna- 1.5.7 336
51.5
75.3'
60.0"
83.6
1464.3
352.9
34.4
63.5
59.4
28.9
58.3
Table Results MQT-LLAVA with different numbers visual tokens. To demonstrate our flexibility in selecting any number of tokens up to 256, we chose random number of visual tokens during inference, 77,
which was not seen during training.
Daa N0
DUN ScAN P2
Ganyy Stevet
Sonny
Ganry Sham
Duty Sievan
Figure 10: Grad-CAM visualization from all the tokens in our model when inference with 8 tokens. Input: "How many cats are there in the image? Answer: 2".
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our main claims in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Limitation Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper doesn't 't introduce new theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: Yes, we fully disclose all the information see Section and our experimental
setup 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We provide our code along with scripts to supplemental material. All the code and models will be open sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
iocy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer our implementation details in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We benchmarked on large-scale data with LVLM models that includes a 7B LLM, making it expensive run things multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to our implementation details in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarrrs.cccp......caaaaaa
Answer: [Yes]
Justification:
We discuss ethics concern and broader impact in Appendix A
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impact in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We require users to follow the guidelines such as LLM (Llama and Vicuna)'s guidelines when release the model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly credited the original owners and followed their license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "TGFCQVGkEe", "reviewer": "Reviewer_gY5g", "summary": "This paper addresses the challenge of achieving flexibility in the number of visual tokens to suit different tasks and computational resources. Inspired by Matryoshka Representation Learning (MRL), the authors propose the Matryoshka Query Transformer (MQT), which allows for any number of visual tokens during inference. Experimental results demonstrate considerable performance across varied visual token lengths and show promising results even with extreme token numbers.", "questions": "As in LLaVA-1.5, the 576 visual tokens do not introduce significant computational overhead. It would be more meaningful to evaluate this method on a more extensive case like LLaVA-NEXT.", "limitations": "Please consider answer the questions above.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of handling an arbitrary number of visual tokens is highly valuable. The authors propose a novel MRL-based method to tackle this issue.\n\n2. The experimental results are promising, highlighting the trade-off between performance and computational resources.\n\n3. The analysis of the impact of visual tokens is valuable, providing meaningful conclusions that deepen our understanding.", "weaknesses": "1. Can you explain the choice of the MQT transformer design? Does it need to maintain sufficient capability for extracting information from varied tokens? For example, Q-Former has multiple layers to achieve effective information extraction.\n\n2. The extraction process V=Q(Z,G) lacks correlation with textual information. In Fig 4, different text prompts should result in the same attention map due to this. Therefore, this design might be limited when dealing with complex images, where visual representations should maintain different semantics.\n\n3. How does the model perform in text-rich scenarios, such as TextVQA? The sophisticated recognition ability might be significantly compromised in these cases.\n\n4. Since the number of visual tokens affects performance, how does the model perform when the number of visual tokens exceeds the original 576 tokens?\n\n5. It would be better to provide a comparison with the LLaVA baseline, using adaptive pooling or vanilla Q-former to adjust token numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n1. How many layers does MQT transformer need as compared to Q-Former?  \n\nWe studied the layers for MQT transformers at the early stage of our research. We didn’t find improved performance with multiple layers. This similar one layer design is also employed by Qwen-VL (Bai et al., 2023) which is a strong LVLM in the open-source community. Our extensive experiments in **Table 1** and comparison to Q-Former in Figure 3 (''With only 2 visual tokens, MQT-LLAVA outperforms InstructBLIP on all 8 tasks'') have demonstrated the sufficient capabilities of MQT design.\n\n2.  Should text prompts be involved in the visual feature extraction process? \n\nThanks for the detailed observation of our model. Most current LVLMs (except BLIP2 has an extra text encoder) do not use text prompts as input for extracting visual information. All the visual features are provided to LLM, together with the input question. LLM can then employ the textual information and identify useful visual features in answering the text question. We acknowledge this is a good suggestion in the lower number of visual tokens scenarios, whereas the model can pre-select the useful visual feature with textual information before extracting them. But an extra text understanding model needs to be properly designed for this purpose. We will leave this suggestion to future study of our work, but we believe this is orthogonal to the main claims of our paper. \n\n3. TextVQA Performance\n\nPlease refer to the general response’s Table 2 for added TextVQA results. We observe a very steady performance trend even when reducing the number of visual tokens to 16 tokens. In our paper’s Table 1, we have tested on 4 comprehensive evaluation benchmarks for Large Multimodal Models including MME,  MMMU, MMBench, and MM-Vet, where our model demonstrated robust performance on these tasks. All 4 benchmarks involving evaluation on OCR and text related tasks, we report the specific scores for these sub categories in our general response’s Table 3, please refer to it as well.  In summary, our model performs the best in these text related tasks with 3.2 points higher than LLaVA-1.5 on average. \n\n4. How does the model perform when the number of visual tokens exceeds the original 576 tokens? \n\nTechnically, the number of visual tokens is bounded by the original number since that’s the maximum visual information that a vision encoder can represent. But we did experiment with this interesting idea at an early stage. When we initialize the query tokens as sinusoidal embeddings similar to positional embeddings, then we can extrapolate the number of query tokens beyond the original limit. However, we didn’t find significant improvement in performance and the extrapolated visual tokens are also non intuitive in their meanings. \n\n5. Adaptive Pooling or Vanilla Q-Former baseline.\n\nOur main idea is to train a model once and enable elastic inference of **any** number of visual tokens (under the maximum) for various computation resources and downstream tasks (line 7-9 and 12-14 in our Abstract). Adaptive pooling based approach will limit the choice of visual tokens to only several options. Whereas for Vanilla Q-Former, we conducted preliminary experiments at the starting point of our project, please refer to the detailed scores in our general response’s Table 2. In summary, vanilla Q-Former doesn’t demonstrate strong capabilities for our tasks. \n\n6. Evaluation on more extensive heavy computation cases like LLaVA-Next. \n\nOur MQT design can be served as a plug-and-play module to other models (as mentioned by reviewer jmsV, too), we adopted LLaVA-1.5 as the base model to validate our methods and have demonstrated strong capabilities. We plan to extend our MQT method to videos and Interleaved image text tasks like LLaVA-Next as well. Recent work such as SlowFast-LLaVA (Xu et al., 2024)  from Apple have demonstrated promising results in this direction. However, these are out of the scope of this paper and can be considered as a future work direction. We have released source code and models for the community to try out the ideas for other models as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ndVsR147BE", "reviewer": "Reviewer_jmsV", "summary": "The paper considers multimodal vision transformers, in which we have a stream of both visual and textual tokens. Current architecture typically assume a fixed number $m$ of visual tokens. In contrast, the proposed `MQT` aims to achieves a dynamic number of visual tokens. This is done by integrating a form of compression+dropout mechanism during training. Specifically, the image tokens are compressed to $M$ tokens using cross-attention. To vary the number of tokens, the model only needs to use only  $m < M$ queries from te cross-attention mechanism. At training time, this is done by randomly selecting $m$ and feeding the **first** $m$ queries.", "questions": "- How are the $m$ latent queries chosen at inference ? I am assuming $m$ is chosen as a hyperparameter, and the first $m$ queries are kept, however it is not very clear from the methods section\n\n- Addressing the **low FLOPs regime**:   From Figure 1, it seems that the benefits of the proposed `MQT` decrease in the low FLOPs regime, when increasing the sparsity ratio of $M / m$. I think it is not necessarily a surprising result as increasing sparsity too much will severely impact accuracy. However, I am wondering how would the trade-off curves would look like when starting with a base `QRT` model with a smaller $M$ and lower sparsity ratio, i.e. with a model that would achieves same number of tokens/FLOPs, but where the relative sparsity ratio to achieve is less aggressive.", "limitations": "The paper has a limitation section but in my opinion it does not fully address some limitations of the paper (e.g. no improvement in performance in the low FLOPs regime)", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed method is a simple idea that can be added as plug-and-play to any model\n* generally the idea of using some form of dropout also often has benefits for regularization so it may also benefits more than efficiency\n* The paper is well written and has comprehensive ablation experiments", "weaknesses": "- **Lack of comparison with dynamic baselines**: in the field of image-only ViT, there is a flourishing literature on dynamically reducing the number of tokens using merging or pruning or scale selection. I am not as familiar with this literature for multimodal models, but there does seem to be similar existing approaches: for instance, *LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models* is one, though it might be too recent for the authors to have considered it for the submission. However, more generally, having a simple off-the-shelf token merging/pruning approach for the ViT encoder would be interesting.\n\n- **No improvement over baselines in the low FLOPs regime**. From Figure 1, it seems that the benefits of the proposed `MQT` decrease in the low FLOPs regime, when increasing the sparsity ratio of $M / m$.  This limits the usefulness of the methods in the realm of model efficiency, though it seems to be a useful and simple training strategy for the high FLOPs regime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n1. Comparison with dynamic baselines. \n\nFor comparison with LLaVA-PruMerge, please refer to our Table 1 in general response. In summary, \nOur model significantly outperformed LLaVA-PruMerge in **5 out of 6 tasks**, and by a **2.3** absolute score on average. Moreover, LLaVA-PrugeMerge employs a interquartile range (IQR) method to identify import visual tokens, the identified number is “fixed” in IQR. Whereas our MQT can enable a dynamic number of **any** visual tokens during inference. \n\nAs for dynamic in ViT encoder, the following discussion is directly cited from LLaVA-PruMerge paper Section 3.5 which corresponds with our thoughts as well. ''Block by block, tokens are gradually reduced in number, correspondingly reducing the computation cost in the internal ViT'',  ''LMMs usually require a large stack of visual tokens… Thus, using previous token merging methods to obtain one refined class token as representation of visual input is not consistent with the literature of large multimodal models''. \n\nTo the best of our knowledge, we didn’t find a dynamic visual tokens baseline in ViT for us to compare. Besides, training a dynamic ViT for this purpose is very computationally intensive. This highlights the efficiency and lightweight nature of our MQT design.\n\n2. Improvement in Low FLOPs regime. \n\nOur main contribution is not to train a model which performs better than independently trained low FLOPs models. But rather to have a model that can train only once that can enable elastic inference of **any** number of visual tokens. Please refer to line 12-15 in our abstract and line 43-44 in our introduction section. The illustration of Figure 1 demonstrated that our model is at least as good as training (exhaustively 256) possible models by picking several choices of visual token numbers in this range for evaluation and comparison. And we are glad to find out that our model even achieved better performance in high FLOPs regime.  \n\n3. How is $m$ chosen at inference? from the method section? \n\nYes, your assumption is correct. Please also refer to line 50 in our introduction section: ''During inference, we have the flexibility to selectively utilize solely the initial $m$ visual tokens''. In the method section, we mentioned how $m$ is chosen for training time such as at line 99, therefore how $m$ is chosen at inference time is intuitive. We will incorporate this detail to the method section in our revised paper. Thank you. \n\n4. Addressing Low FLOPs regime and MQT model starting with a lower $M$\n\nPlease first see the response to (2) above. For your question about the base MQT starting with a smaller $M$ and lower sparsity ratio, we believe you are probably referring to the baseline model with ''Fixed #Visual Tokens'' as illustrated in our Figure 1. Its results are indicated by the orange curve. If we misunderstand or you have further questions, please respond to our rebuttal and we are very happy to address your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fN7kXczols", "reviewer": "Reviewer_k14Y", "summary": "Authors try to use Matryoshka mechanism to guide the learning process of LVLMs such as llava. \n\nAuthors show a pretty good scaling curve with different amount of visual tokens.", "questions": "1. What are the performance under other datasets like TextVQA and ChartQA? Those are dataset different from the natrual image distribution. \n2. The visualizations are good but we don't have quantitive numbers to support your claim. Since those pictures can be cherry picked.", "limitations": "See comments above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea is interesting. \n2. The presentation is clear. \n3. The attention visualization is interesting.", "weaknesses": "1. For a given image, which scale should I choose to achieve best trade off? Is there an answer?\n2. Authors use more compute (2 epoch) to get the similar performance as LLaVA, is there any reason for this?\n3. All designs are center upon the qformer. There is little study on this. How many layers do we need? Do we really need it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n1. What scale to choose for the best trade-off?\n\nThe trade-off visualization is from Figure 5 and the complete version is from Figure 8 in our paper’s appendix. For the best tradeoff, as we mentioned in line 215, we observed a ''turning point'' in many benchmarks. This point has a relatively lower computation but keeps a high performance. After this point, the performance will decrease more drastically although with a much lower computation. We think this point can be a good answer for the best trade-off. \n\n2. Training Computation \n\nOur main contribution is to achieve a single model for elastic inference of **any** visual tokens (line 7-9 in our Abstract). As mentioned in our methods (see line 102), we achieved this by choosing $m$ from {2, 4, 6, …, 252, 254, 256}, which means only the first 2 tokens are always supervised for the entire training data, whereas for the rest of visual tokens, less supervision is applied, especially for the visual tokens at the tailing positions. But our gradually reducing token numbers strategy helps stabilize this issue as we demonstrated robust performance in our experiments. Therefore, we apply 2 epochs of training data to help this issue. Gladly, LLaVA instruction tuning is very computational friendly. We only trained for 15 hours (one epoch) more on one A6000 machine, which is a cost with less concern. \n\n3. Study on Q-Former \n\nConcerning the Q-Former layers. We studied the layers for MQT transformers at the early stage of our research. We didn’t find improved performance with multiple layers. This similar design is also employed by Qwen-VL (Bai et al., 2023) which is a strong LVLM in the open-source community. Our extensive experiments in Table 1 and comparison to Q-Former in Figure 3 also demonstrated the sufficient capabilities of MQT design. We also incorporated LLaVA + vallina Q-Former results in the Table 1 of our general response. \n\nConcerning Q-Former at all. Our MQT design is to achieve a single model which is trained only once for elastic inference of **any** visual tokens (line 7-9 in our Abstract). Query Transformer design has a learned number of visual tokens, therefore, during inference, an elastic number of visual tokens can be choosed for various downstream tasks to achieve our goal. \n\n4. TextQA and ChartQA\n\nPlease refer to the general response’s Table 2 for added TextVQA results. We observe a very steady performance trend even when reducing the number of visual tokens to 16 tokens. In our paper’s Table 1, we have tested on 4 comprehensive evaluation benchmarks for Large Multimodal Models including MME,  MMMU, MMBench, and MM-Vet, where our model demonstrated robust performance on these tasks. All 4 benchmarks involving evaluation on OCR and text related tasks, we report the specific scores for these sub categories in our general response’s Table 3, please refer to it.  In summary, our model performs the best in these text related tasks with 3.2 points higher than LLaVA-1.5 on average. \n\nAs for ChartQA, since LLaVA-1.5 didn’t incorporate any training data for chart understanding, their paper didn’t evaluate on this dataset. When we downloaded their weight and conducted evaluation, it reveals LLaVA-1.5 performs very poorly on ChartQA. Therefore, this task may not be suitable for use as a reference for comparison. But we also provided these results here for your reference. LLaVA-1.5 with 576 tokens achieved 18.2 on ChartQA, and our model with only 256 tokens, reducing to  [144, 64, 36, 16, 8, 4] achieved 14.3,  14.1,  14.0 , 13.6 , 13.7 , 12.7 , and 12.0 separately. This also indicates the strong robustness of our model in reducing numbers of visual tokens. \n\n5. Attention map visualization and quantitative results. \n\nWe have conducted extensive quantitative evaluation in our paper’s Table 1 and Table 2 and demonstrated strong capabilities of our model. We conducted attention map visualization in order to understand what visual information is the model focusing on when using a lower number of visual tokens. The visualization on 1 visual token is **not cherry-picked** but randomly picked. For example, we provided the visualization when our model inference with 8 tokens, please refer to Figure 1 in our general response. We will also incorporate this visualization to our revised version. For other settings such as 16 tokens or 256 tokens, we fixed it to randomly pick one visual token for visualization purpose.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1ROPhqGFn", "reviewer": "Reviewer_L7Jp", "summary": "This paper introduces a new concept called Matryoshka Query Transformer (MQT) that brings in the concept of Matryoshka information packing to make visual tokens flexible and can be used in multimodal vision language models like LLaVA. The reduced tokens tend to reduce the quadratic complexity of the language model due to potential flexibility in the processed visual tokens. \n\nOwing to this, there are significant efficiency benefits all the while retaining accuracy across various benchmarks. The paper also showcases analysis and further discussion on the technique. \n\n\n\n----------------------------\nThe review will be short and does not reflect the time put in for the review or the quality of the paper. When the ideas are simple and clear -- I tend to write shorter reviews to the point.", "questions": "See above.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I will go sequentially\n\n1) The paper is extremely well-written and easy to follow. \n2) The core ideas, while not completely novel as mentioned by the authors, Matryoshka and the Query transformer, making them work together and showing incredible benefits is commendable and is a worthy contribution. \n3) The modelling very clear and the mechanism to achieve and the details are very well fleshed out. \n4) The experiments are extensive, with a good analysis. \n5) Great job with visualizations and nice to see some TFLOPs measurement. \n6) Good analysis beyond benchmark numbers. \n\nOverall, this is a solid work with practical utility. I have a few questions about the paper mentioned in the weaknesses and would appreciate answers for them in the rebuttal, however, I am happy with the paper and am willing to champion it unless there is something I am missing found by other reviewers.", "weaknesses": "Most of these are not weaknesses, but rather questions\n\n1) How are you measuring TFLOPs? What does it include, the cost of vision encoder? LLM processing and generation? I might have missed this in the paper and any pointer would be great.\n2) While I understand the when you are picking a fixed # token for MQT, you make that choice before in hand. What happens if you actually just obtain 256 tokens and take the first 8? What will be the performance in that case? I know softmax etc make huge differences, but would be good to see that. \n3) Any thoughts on sampling vs joint training?\n4) Why is log-based (actually exponential) spacing so poor (not 3.5% as mentioned in the paper but 2.1% from the table) so much worse than fine granularity? This is a bit surprising to me because 256 token performance should not be affected because of the lower token counts.\n5) In visualizations, what do you mean by one random token? \n\nI am looking forward to your answers on these things.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n1. Measure of TFLOPs\n\nWe measured the TFLOPs by incorporating everything, i.e, including the cost of vision encoder and LLMs. Thank you for pointing out, we would like to incorporate this detail in our revised version. \n\n2. Scenario of just obtain 256 tokens and take the first 8.  \n\nDuring the inference process, the inputs to MQT are queries and visual features from vit, then the cross attention mechanism will output the learned visual feature before feeding into LLM. So there is no softmax involved in this process. We also experimented to find out that indeed no performance changed in this scenario. \n\n3. Sampling vs joint training.\n\nFor joint training, we can choose multiple $m$ tokens jointly. We think this design can slightly increase the computation of each training step, but may optimize faster from a global steps view. It’s also interesting to consider the design of how and where to choose the multiple $m$ tokens, for example choose $m$ from a local region (i.e, each $m$ is close with each other) or from a more global region (i.e, joint the the first token and all the 256 tokens). Thanks for your suggestion, we will leave this exploration to our future work. \n\n4. Why log based is worse than fine-grained, not 3.5% as  mentioned in the paper but 2.1% from the table.\n\nWe apologize for this confusion. We think the 2.1% absolute performance drop which is observed from the table is the same as 3.5% mentioned in the paper, which represents the relative performance change compared to our method. 3.5% is (our score (59.4) - abalated model score (57.3)) / 59.4. We only employed this relative score comparison in our ablation study table. We will incorporate this explanation into our revised version. As we discussed in line 249, this validated our hypothesis that gradually compressing the visual tokens helps the model perform better than log-based choices. Although the tested scenario is 256 tokens, the first few tokens of 256 are still “log-based” trained in the ablated model (for example, still large gaps between 64 and 128 tokens or 128 and 256 tokens), which experimentally found to be not as good as our final model. \n\n5. How is a random token picked in the visualization? \n\nFor example, we provided the visualization when our model inference with 8 tokens, please refer to Figure 1 in our general response. We will also incorporate this visualization to our revised version. Here, say we pick the 6th token which is on the 0.75 percentile location. For other settings such as 16 tokens or 256 tokens, we fixed it to randomly pick one visual token for visualization purposes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and their constructive reviews and suggestions. We are encouraged that the reviewers find that:

(1) Our Matryoshka Query transformer is interesting (Reviewer k14Y) and showing **incredible benefits and is a worthy contribution** (Reviewer  L7Jp). Our model handles the problem of arbitrary number of visual tokens which is **highly valuable** (Reviewer gY5g) and can be added as **plug-and-play** to any model (Reviewer jmsV).

(2) Our experiments are comprehensive (Reviewer jmsV). The experimental results are extensive (Reviewer L7Jp) and promising (Reviewer gY5g), highlighting the trade-off between performance and computational resources (Reviewer gY5g). 

(3) The analysis of the impact of visual tokens is valuable (Reviewer gY5g). This good analysis goes beyond benchmark numbers (Reviewer L7Jp).

(4) Our paper is well-written and clear (Reviewers jmsV, k14Y, L7Jp) and presents a great job with interesting visualizations (Reviewers k14Y, L7Jp).

Thank all the reviewers for their help again! We believe the comments and revisions have made the paper stronger. Please find individual responses to your questions below.

Meta-review: This paper unanimously receives positive rates thanks to the novel use of visual tokens and convincing experiments. Although overall reviews are positive, the rebuttal should be reflected in the final draft.
Decision: Accept (poster)