Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
Authors: Kevin Li
Sachin Goyal
João D. Semedo
J Zico Kolter
Venue: ICLR 2025 Poster (2025)
Keywords: vision language model
inference scaling
visual token compression
Primary area: foundation or frontier models, including LLMs
TL;DR: We study inference-compute optimal scaling behavior of visual language models by varying the input token count and parameter size.

Abstract: Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks, driven by incorporating image representations into the token inputs of Large Language Models (LLMs). However, their real-world deployment is often constrained by high latency during inference due to the substantial compute required by the LLM to process the large number of input tokens, predominantly arising from the image. To reduce inference costs, one can either downsize the LLM or reduce the number of input tokens needed to represent the image, the latter of which has been the focus of many recent efforts around token compression. However, it is unclear what the optimal trade-off is given a fixed inference budget. 
We first characterize this optimal trade-off between the number of visual tokens and LLM parameters by establishing scaling laws that capture variations in performance with these two factors. Our results reveal a surprising trend: for visual reasoning tasks, the inference-optimal behavior in VLMs is achieved by using the largest LLM that fits within the inference budget while minimizing visual token count - often to a single token. While the token reduction literature has mainly focused on maintaining base model performance by modestly reducing the token count (e.g., $5-10\times$), our results indicate that the compute-optimal inference regime requires operating under even higher token compression ratios. Based on these insights, we take the first steps toward designing token compression algorithms tailored for high-compression settings, utilizing prompt-based compression of tokens. Our work underscores the performance and efficiency benefits of operating in low visual token regimes and the importance of developing tailored token reduction algorithms for such conditions.

Published as a conference paper at ICLR 2025
INFERENCE OPTIMAL VLMs NEED FEWER VISUAL TOKENS AND MORE PARAMETERS
Kevin Y. Li Sachin Goyal"! João D. Semedo J. Zico Kolter'
'Carnegie Mellon University, Bosch Center for Artificial Intelligence (ky12, sachingo, zkolter) emu.edu
joao semedo@us wwwch.
ABSTRACT
Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks, driven by incorporating image representations into the token inputs of Large Language Models (LLMSs. How- ever, their real-world deployment is often constrained by high latency during in- ference due to the substantial compute required by the LLM to process the large number of input tokens, predominantly arising from the image. To reduce infer- ence costs, one can either downsize the LLM or reduce the number of input tokens needed to represent the image, the latter of which has been the focus of many re- cent efforts around token compression. However, it is unclear what the optimal trade-off is given fixed inference budget. We first characterize this optimal trade- off between the number of visual tokens and L.LM parameters by establishing scal- ing laws that capture variations in performance with these two factors. Our results reveal a surprising trend: for visual reasoning tasks, the inference-optimg behav- ior in VLMs is achieved by using the largest LLM that fits within the inference budget while minimizing visual token count often to single token. While the token reduction literature has mainly focused on maintaining base model perfor- mance by modestly reducing the token count (e.g., 5 10x), our results indicate that the compute-optimal inference regime requires operating under even higher token compression ratios. Based on these insights, take the first steps toward designing token compression algorithms tailored for high-compression settings, utilizing prompt-based compression of tokens. Our work underscores the perfor- mance and efficiency benefits of operating in low visual token regimes and the importance of developing tailored token reduction algorithms for such conditions.
INTRODUCTION
Recent advancements in Large Language Models (LLMs) have enabled Vision Language Models (VLMs) perceive, reason, and respond through both text and image inputs (Liu al.) 2023 Alayrac et al. [022] Dai et al. 2023). Many VLMs are built on top of pretrained vision encoders, such CLIP, and pass the patch-based tokens from the visual encoder into the pretrained LLM backbone at a one-to-one ratio for visual context. This results in the LLM processing hundreds of tokens per image, overshadowing those from the user prompt and accounting for most of in- ference time compute. Consequently, deploying VLMs in real-world applications, particularly on consumer-sided edge devices, c.g., monitoring systems, driving assistants, etc., often limited by the significant inference cost and resulting latency.
To reduce the inference cost of VLMs, many recent works have focused on decreasing, via merging or pruning, the number of visual tokens passed to the LLM without significant performance degra- dation (Li et al., 2024c [Shang et al.) 2024). Alternatively, inference FLOPs, proportional to the number of parameters and number of tokens processed, can be reduced by using smaller LLM. Since both the LLM size and number of visual input tokens directly affect the VLM's performance, it becomes unclear what the optimal trade-off between the two is. For example, a 4B LLM process- ing all visual input tokens results similar inference costs to a 8B LLM processing half the number of original visual tokens currently, the ideal choice is unknown.
Equal contribution, work partially done Bosch Research.

---

Published as a conference paper at ICLR 2025
Scaling #Params #Visual Tokens
Scaling #Params vs #Visual Tokens
Avg. Downstream Error
0.650
78
Avg. Downstream Error
78
0.625
6.07
LLLM Params (N)
0.600
0.077
LLM Params (N)
0.600
Optinal
0.575
Optiral
0.575
-48
0.550
-18
0.550
0.525
0.525
0.500
0.500
35
0.475-
144
0.475
64
144
576
576
0.58
0.450
0.58
10
10
102
103
10*
102
10?
10*
Inference FLOP (O(NV)))
Inference FLOP (e(N(50 + VI)))
Figure 1: Inference optimal scaling laws for VLMs. The number of visual tokens (V) passed to the LLM (after token compression, $ 2.2 along with the LLM parameter count (N), directly determine the inference cost of VLMs (O [O( + V)), where Q is the text input tokens. Since the downstream performance of VLMs is directly affected by both these factors, it makes it unclear what the optimal trade-off is for a fixed inference compute. In this work, we try to answer this question with scaling laws. Left: We plot the fitted scaling curves, assuming cached text input tokens (Q=0). We observe a surprising trend: for visual reasoning tasks, the compute optimal behavior (dotted black curve) requires using a single visual token with the largest possible language model that can fit under the inference budget. Right: Inference optimal behavior under Q 50 requires slightly higher number of visual tokens as the L.M already incurs a fixed cost due to the text tokens.
This observation raises an important question: given a fixed inference budget, what is the optimal trade-off between brateenffbetween I.I size and the number of visual tokens processed for downstream performance? In this work, we answer this question by building the first inference-ting compute-optimal scaling laws for VLMs, modeling performance LLM size and the number of visual tokens processed. Our scaling laws reveal a striking observation: for visual reasoning tasks, the compute-optimal inference regime entails using the largest feasible LLM with very small number of visual input tokens usually less than 3% the original number of visual tokens. However, for certain use cases that require detailed image analysis, like Optical Character Recognition (OCR) or document understanding tasks, the optimal approach is quite the opposite, requiring as many visual tokens as possible, as token compression proves ineffective for capturing the dense and diverse information present in such tasks.
function of both key factors affecting inference cost:
Most existing work on token compression has focused on reducing visual tokens by modest factor (e.g., from 576 to 144 tokens 64 tokens). In contrast, our results underscore the critical im- portance of pursuing much higher compression rates (e.g., reducing tokens to I or 4) for visual reasoning tasks where such compression is not only feasible, but also compute-optimal. Our work identifies the compute-optimal inference regime for VI.Ms, emphasizing the importance of high to- ken compression for visual reasoning tasks. We hope these findings will serve motivation and foundation for shifting token reduction techniques towards more effective and higher compression ratios.
We first introduce some preliminaries around inference costs and visual token compression for VLMs in Section 2 In Section 3 we formulate and analyze our inference-con compute scaling laws, in- cluding its generalization across compression techniques and trends across different tasks. Section covers the related work, and we conclude with Section
2 PRELIMINARIES
2.1 ESTIMATING INFERENCE COST FOR VLMS
The language model in VLMs processes the visual input tokens along with the user text query tokens. As language models become larger, the FLOPs (Floating Point Operations) required to process each input token scales accordingly. We follow the standard practice for estimating the inference time
2

---

Published as a conference paper at ICLR 2025
FLOPs (Kaplan et al. 2020; Sardana et al. 2024 Snell et al. 2024):
FL O(N x T),
(1)
where N denotes the parameter count of LLM and T denotes the total inference time tokens. We ignore the inference cost stemming from the visual encoder, we use the same vision encoder with the same input image resolution across all experiments. In addition, many current open-source VLMs currently utilize the same CLIP-L vision encoder (Radford et al. 2021).
We highlight that the inference cost of VL.Ms scales proportionally with both the parameters and the number of input tokens processed by the LLM.
In the context of VLMs, the total inference tokens, T,, can be further decomposed as = Q+V+G, where Q represents the text input tokens, i.e., the question/prompt, V represents the number of visual tokens from the vision encoder (after token compression), and G' accounts for the generated tokens. In many real world applications. such as driving assistance systems, the text input remains constant (e.g." "Alert the driver if the scene ahead has hazard"). In these scenarios, the text input can be eached, effectively making Q by bypassing self-attention projections and feed-forward calculations. However, in other interactive applications. Q may vary based on dynamic input. We will study the behavior of the downstream error with F' P'iOPs under both the Q = and varying Q regimes. In our work, ignore the G' term due to our analyzed tasks' short form responses; however, the analysis with varying Q transfers to Q + G as well.
2.2 TOKEN COMPRESSION IN VLMS
As discussed in the previous section, inference FLOPs for VLMs increase proportionally with the number of visual input tokens (e.g., 576 with CLIP-ViT-L visual encoder). Often, the number of visual tokens dominates the total tokens processed by the language model, especially in applications where the text input can be cached or is on the shorter side. Thus, there has been a growing interest in developing approaches to compress the visual information into a fewer number of tokens.
More formally, let the visual encoder be defined as function f(I) X, where X ixd rep- resents a sequence of n vision embedding tokens produced by the encoder from the input image I. Token compression then learns a vision projector go (X) = Y that maps these embeddings X to Y E Rmxd, compressed sequence of m < tokens to be processed by the language model (n m for standard VL.Ms without any token compression). We refer the reader to Section 4.1 for detailed discussion some of the recent token compression algorithms.
Note that token compression doesn't 't refer to using a smaller visual encoder or using smaller image resolutions as inputs to the encoder. These approaches usually either do not decrease the visual token count much (beyond around 224) or lead to a large drops in performance (Li al. 2024a).
TOKENS VS PARAMETERS: INFERENCE TIME SCALING LAWS FOR VLMS
The deployment of vision language models in real-world applications comes with significant chal- lenges, particularly surrounding inference latency and frames per second (FPS). For instance, in real- time systems, such as automotive driver assistance hazard monitoring, maintaining high FPS and quick response times is crucial for safe and effective deployment. Consequently, reducing inference FLOPs while minimizing downstream performance degradation is of critical, practical importance, especially on consumer-grade edge devices, which are often severely compute constrained.
This has led to a growing interest visual token compression for VLMs ($ 2.2p. Alternatively, one could also use smaller LLM to reduce inference cost. However, both of the above factors directly influence the downstream performance ($ 2.1). This raises a key question: Given a fixed inference compute budget for VLMs, what is the optimal trade-off between the language model size and the number of visual tokens processed? In our work, we answer this question by developing scaling laws for VLMs that account for the varying parameter count of the language model component and the number of visual input tokens processed by the language model. As mentioned in Section [2.] assume the inference cost from the visual encoder to be fixed and ignore it from here out.
3

---

Published as a conference paper at ICLR 2025
3.1 TOKENS VS PARAMETERS: SCALING LAW FORMULATION
Recall that the performance of a VLM is primarily governed by the parameter count of the language model and the number of visual tokens processed by the LLM, assuming fixed visual encoder. Accordingly, we model the scaling behavior of VLM performance as:
B N T D
Y(N,
where N denotes the LLM parameters, T denotes the total inference tokens, learnable parameters, and 7) ís measure of model quality. Although traditional scaling laws have been studied in the context of training loss Kaplan et al. (2020), practitioners often use the direct downstream performance assess model quality (Gadre et al. |2024 Goyal al. [0224b]] Liu et al. 2022). We use average performance error on suite of nine commonly used visual reasoning benchmarks ($ 3.2 as measure of model quality Y(N,T).
a, }} are
Below, we summarize the role of each of these learnable parameter in the scaling law (Eq. 2).
LLM Quality Parameter (a): This parameter dictates how the downstream error changes with the complexity of the LLM, i.c., its parameter count. A higher indicates a better language model, such as Llama3-7B outperforming Llama2-7B, often due to superior pretraining.
Visual Token Quality Parameter (3): 3 captures the quality of the visual input tokens fed into the LLM, reflecting the quality of the compression technique. better token compression algorithm would yield higher B, allowing for more reductions of I visual tokens than less effective methods while maintaining the same downstream performance.
Constants A, B. D: A and B are normalizing constants and D) refers to irreducible loss, which cannot be reduced even with the largest N N-sized language model or all T visual tokens (capped at 576 for our choice of vision encoder).
3.2 EXPERIMENTAL SETUP
VLM Training and Evaluation: We use the LLaVA-Next framework (Liu et al. 2024b) to train VLMs with the Qwen-1.5 family of language models as the backbone. Specifically, we utilize the (0.5, 1.8, 4, 7, B-chat models (Bai et al., 2023). The pretraining and finetuning dataset and hyperparameters follow Liu et al. (2024a). except we doubled the effective batch size for finetuning.
To estimate the downstream eтror Y(N,)) we test our trained VLMs on a suite of nine com- monly used benchmarks for evaluating visual reasoning: MME (Fu al. 2024). GQA (Hudson & Manning 2019), A12D (Kembhavi et al. 2016). MMBench (Liu et al. 2024c). MMMU (Yue et al.) 2023), ScienceQA (Lu et al.j 2022). Matha (Lu et al.) 2024)) POPE (Li et al. 2023c), and ChariQA (Masry et 2022). We average the normalized evaluation metric errors to compute P(N, C). For MME, the Cognition and Perception scores were added and normalized, while the Fl score was used for POPE (Liu et al. 2024a). As previously mentioned in Section 2.1 this set of datasets was selected due to their similar prompt and generation length and overall comprehensive- ness.
Visual Token Compression: CLIP ViT-I/14 (Radford et al., 2021) used as the vision encoder for all experiments, and we compress the original 576 tokens {144, 64, 36, 16, 4, 1} using of our variants of TokenPPackr (Li et al. 2024c) which replaces interpolation with a convolution for downsampling (no adding query embedding, refer to App. B for more details).
Fitting Scaling Laws: We fit the proposed scaling law (Eq. 2b on (N,,, N,T) pairs, with N {0.5, 1.8, B and T E {1,4, 16, 36, 64, 144, 576). We use grid-search, for its stabil- ity (Goyal et al. 2024b), to estimate the scaling parameters B, A, B, and D. The final scaling law is evaluated on a N 14B VLM model various T' visual tokens. Further details about the grid-search fit can be found in Appendix A.2
3.3 RESULTS: ESTIMATED SCALING CURVES
Figure D presents the fitted scaling curves, illustrating the variation in average downstream error a function of inference FL.PPs. The scatter sizes represent the number of visual input tokens
4

---

Published as a conference paper at ICLR 2025
processed by the language model, while the color scale indicates the varying number of language model parameters. We make some key observations below.
Log-Linear Relation between Error and Number of Visual Input Tokens: Consider the change in performance for the 7B model as the number of visual input tokens varies (maroon curve in Figure [T) Recent works visual token compression (Li et al. [024c) Shang et al.; 2024) claim little to no performance degradation with token compression. For example, they report similar per- formance to the base model's s 576 tokens even when visual token count is reduced 36 or 144 on certain tasks. However, our scaling curves in Figure reveal a different trend, showing a log-linear decrease in visual reasoning performance as the number of visual inpul tokens is reduced. We be- lieve this discrepancy arises because of the limited downstream benchmarks evaluated in previous works which may not fully capture the VLM's overall capabilities.
Error Varies 5x Faster with LLM Parameters than with Tokens: Recall from the scaling law (Eq. 2 that a represents the LLM quality parameter and 8 represents the visual token quality pa- rameter, both denoting the rate at which they influence the downstream error respectively. From Figure I we observe that for our selection of language model family (Qwen-1.5) and token com- pression algorithm,
0.077 is more than five times larger than / = 0.015, signifying that VLM error increases significantly faster when reducing the LLM parameters compared to reducing the number of visual tokens. Therefore, when minimizing inference FLOPs, it is more effective to prioritize reducing visual tokens (V) first as the impact on performance is less pronounced than reducing the LLM parameters (N).
Scaling Laws Hold for Increases in LLM Scale: We evaluate the accuracy of our scaling laws (fitted on VLMs of 0.5B-7B range) for predicting the perfor- mance for larger models. We estimate the performance of Qwen-1.5 14B using our fitted scaling laws. Our scaling laws esti-
Accuracy the estimated Scaling Laws
Avg. Downstream Error
0.625
Scaing Params
14B
0.600
$
# #LLM Params (N)
Fitted Curves
0.575
Estimated Curve
0.550
0.525
TokemstV
78
mate the performance with error margin
0.500
of less than 2%. as visualized in Figure 2 and Figure 6b The log-linear relationship between the error and number of visual to- kens persists, and the greater influence of the LLM's size compared to visual tokens on performance continues to hold. Thus, for VLMs using 7B language model back- bones, it is still optimal to increase LLM size to 14B while reducing visual token count for fixed inference costs.
16
0.475
36
48
64
0.450
1.8B
0.5B
10a
10a
10?
10a
104
Inference FLOP (O(NV)))
Figure 2: Our scaling laws (fitted on 0,5-7B VLMs), estimate the performance of 14B VLM with error margin of less than 2%.
3.3.1 COMPUTE-OPTIMAL VISUAL REASONING INFERENCE FAVORS MORE LLM PARAMETERS
Observe the pareto optimal curve (black dotted curve) in Figure I Our results reveal a striking insight: under a fixed inference budget, visual reasoning tasks heavily favor increasing LLM pa- rameter count. At any given inference FLOPs (denoted by any potential vertical line in Figure I. the compute-optimal strategy is to allocate FLOPs towards a larger LLM by reducing the number of visual tokens (even to one if the prompt can be pre-computed and cached, Q 0).
similar trend holds in the Q 50 regime, where the optimal number of visual tokens around 16, 97% reduction from the original number of tokens. This be intuitively explained by the fact that since the language model is already dedicating a fixed amount of compute to process text input tokens, the initial increase in the number of visual tokens represents only minor amount of the fraction of compute already being spent. Thus, the optimal tokens increases to 16, compared to just in the cached regime.
In Figure we compare VLMs with varying combinations of LLM size and visual token counts un- der a fixed inference budget. We observe that for many visual reasoning tasks, increasing the size of
5

---

Published as a conference paper at ICLR 2025
Text Tokens (Q) vs Compute Optimal Tokens
Scaling trends for OCR Tasks
100
78
Avg. Downstream Error
0.90
0.56
0.025
80
# # Tokens (Q)
0.85
LLM Params (N)
Avg. Error
0.54
Opeiral
60
0.80
0.52
-18
4Takerdivi
0.50
15
0.48
48
20
64
144
35
78
576
0.58
164
10a
10a
FLOPs O(N(Q + VI)), Increasing V
Inference FLOP (o(NV))
(a) Performance trend changes based LLM sizes when varying number of input text token Q.
(b) Scaling law for VLM token compression and LLM model size on OCR like tasks.
Performance trends when shifting input text token count and benchmark family. Left: For visual reasoning tasks, as the number text tokens increases, the impact of increasing the number of visual tokens V. i.e. reducing compression, becomes more apparent. Intuitively, at a large enough amount of text tokens, initial increases visual tokens are only a minor fraction of the overall compute. Right: When the family of tasks shifts from visual reasoning to OCR/text- understanding, the trends shift: visual token count should be the prioritized instead of LLM size.
Figure 3:
the language model while reducing visual tokens can lead to significant relative gains. This may be in part due to the scaling properties of the LLMs themselves, leading to models with stronger world views that can better extrapolate with less visual information than their smaller counterparts (Rad-]
ford et al. 2021 Wei et al. (022). We note that this trade-off, while effective for visual reasoning, does not extend certain tasks, e.g., document comprehension, text identification, etc., where limited number of tokens may fail to capture the high density of information. We discuss this further
in Section 3.4
Scaling Inference Compute by Simply Repeating Tokens: Many recent works around scaling test-time compute by introducing special tokens (Goyal et al.) 2024a or multiple parallel genera- tions Zelikmann et al. 2024) have shown promising gains in reasoning tasks for language models. We test this notion with VLMs by repeating the visual input tokens (compressed to 4) multiple times. However, we do not observe any performance gains. This is most likely due to the downstream tasks for VLMs being not as reasoning- intensivve, thus highlighting the importance of developing better token compression algorithms and potentially introducing more challenging benchmarks.
3.3.2 VARIATION IN OPTIMAL TOKENS WITH TEXT QUERY LENGTH
In the previous section, we observed that when the text input can be cached (Q = 0), compute optimal inference requires the use of a single visual token paired with the largest possible LLM that fits under the inference budget. This scenario covers many practical applications, such as monitoring systems or scenarios where text input remains static. However, in interactive systems where the text input can be dynamic and long, i.e., high Q, the situation changes.
In Figure 3a we plot the average downstream error against FLOPs across different lengths of text input tokens (Q), with the color of the lines representing the variations in Q. When comparing the performance of the 7B model (solid curves) with the 4B model (dashed curves) at high Q (indicated by the green curves for each model), we observe that there is sharp increase in error inference FLOPs are reduced for the 7B model, particularly when visual tokens are reduced significantly. At certain point (marked by the red dot in Fig. 3a it becomes more advantageous to the 4B model with a higher number of visual tokens rather than the 7B model with fewer tokens.
This phenomenon can be understood intuitively: the LLM processes longer text sequences, the computational cost incurred by text tokens is already considerable. Consequently, increasing the number of visual tokens has comparatively smaller impact the overall inference FLOPs. There-

---

Published as a conference paper at ICLR 2025
% Change from 0.58 LM w/ 576 takens
Performance Change Under Similar Inference Compute
1.88 LM w/ 144 takens
30
tokens
7B LM w/ 36 tokens
-10
GQA
MME
MMMU
POPE
SQA
DecVQA
TextVQA
Visual Reasoning Tasks
Text Recognition Tasks
Figure 4: Performances various LLM size and visual token count combinations with similar inference compute on two families of tasks. For many visual reasoning tasks, increasing the LLM size by decreasing the number of visual tokens improves performance. However, for text recognition tasks, decreasing the number of visual tokens is detrimental to performance.
fore, for higher text token lengths (Q), increasing the number visual tokens leads better per- formance without significantly increasing the computational burden. Thus, the optimal number of visual imput tokens rises with increase in Q. This case demonstrates the need for careful balancing of visual token count and LLM size, especially in scenarios where text inputs are long, to achieve compute-optimal performance without sacrificing accuracy.
3.4 SCALING LAWS FOR OCR TASKS
Until now, we have focused scaling behavior for visual reasoning tasks, highlighting the key finding that using a single visual token with the maximum possible LLM parameters is the inference- optimal configuration. However, is the same valid for all tasks? VLMs have recently been applied to document reading and OCR-style tasks where a single visual token may be insufficient due to the high density of information. Unlike visual reasoning tasks, these tasks lack visual structure in the image and intuitively need more tokens to record the (generally textual) details in the image. We verify the same by fitting scaling laws (Eq. 2 DocVQA (Mathew al. 2021) and TextVQA (Singh et al. 2019) benchmarks, where the tasks require mainly OCR capabilities.
significant gains in average downstream performance from increasing LLM parameters; instead, the number of visual tokens predominantly dictates the performance. This observation is reflected in the scaling law pa- rameters, where the LLM-quality parameter = 0.029 is nearly twice as smaller than the token quality parameter 5 0.048. This trend is stark contrast the scaling parameters observed for visual reasoning tasks where the LLM-quality parameter (a) was more than five times larger than the token parameter ($3.)). This notion of visual tokens playing the significant role in text-in-image recognition and understanding is further echoed in Figure which shows token compression weak- ens VLM performance despite increasing capabilities of the LLM component to compensate.
Figure 3b presents the fitted scaling law for OCR tasks. Notably, there
We find that our scaling laws generalize with other visual token compression algorithms. We share additional scaling law results on L.La VAL.VA-B VA-PruMerge (Shang et al.) 2024) in Appendix 3.5
3.5 GENERALIZING SCALING LAWS TO OTHER TOKEN COMPRESSION ALGORITHMS
We find that the takeways for our proposed scaling laws generalize across visual token com- pression algorithms. We fit scaling laws with VLMs utilizing LLa ParuMrrree (Shagg et al. 2024), one of the first visual token compression projectors,
N {0.5, 1.8, 4/B. and T e 136, 64, 144, 192, 228, 576) following Section Unlike many current projectors (Cai et al. 2024 Hu al. 2024 Li et al. 2024c), LLa LLaVA--rrumerg suffers massive performance drops in extreme token compression regimes, resulting from its training-fre methodology. Therefore, we do not consider these conditions in its scaling laws.

---

Published as a conference paper at ICLR 2025
When using the same A, B, D values fit in Section 3.3 find comparable 0.069, 3 0.008
compared to before (a = 0.077, 3 = 0.015, $ (.3). Similar values for a show that our scaling law is capable of capturing the quality of the LLM across VLM architectures and the decrease in 3 shows that the PruMerge compression algorithm "weaker" than TokenPacker, which was also empirically shown in our Table T. Fitring the scaling laws from scratch results in a 0.077, 8 0.041 where performance error only varies around 2x faster with LLM parameters than with tokens. Thus, even across different VLM architectures, compute-optimal inference for visual reasoning and understanding tasks continues to strongly favor the LLM parameter count, as shown in Figure
Scaling PParams #VVsual Tokens
Scaling Params
Viisual Tokens
48
48
Avg. Downstream Error
0.650
Avg. Downstream Error
d=0.041
# LLMM Params (N)
#LLM Params (N)
Opimal
0.600
Optiall
0.600
Persto
Perato
Optiall
Optimal
0.575
0.575
0.550
0.550
1.88
-1.88
0.525
0.525
0.500
10a
10a
10a Inference FLOP (Ø(N(50 + VJ)))
10a
Inference FLOP (O(V)))
Figure 5: Inference optimal scaling laws for PruMerge: When replacing the token compression algorithm, the main findings still hold: inference-optimal behavior is still to increase the LLM pa- rameter count by reducing visual tokens in fixed compute scenarios.
3.6 ENHAANCING EXTREME TOKEN COMPRESSION ALGORITIIMS WITH USER QUERY
For visual reasoning tasks, our scaling laws empirically show that only very few tokens are required for visual reasoning tasks. This need for extreme compression down to 1, 4, or 16 tokens to operate compute-opimaall represents paradigm shift compared to existing compression work which focus on moderate compression (e.g., reduction by 75%). We find that utilizing the user's query prompt during the token compression process can be critical for retaining relevant information and minimizing performance reductions when tokens are reduced to as few as 1 or 4.
We build over existing token compression algorithms (Li et al. 2024c). to incorporate query- based token compression. A full summary of the updated algorithm can be found in Appendix Follow- ing the training regime of LLaVA-1.5 (Liu et al., 2024a), use the pretrained Vicuna 7B model as the LLM component (Zheng et al. 2023). Based on the importance of high token compression un- derscored by our scaling laws ($ 3.3 focus visual token budgets of {1, 4, 16, 36, 64). result- ing in compression rates of 88.9% to 99.8%. We benchmark our method on a diverse, comprehensive of datasets consisting of visual reasoning and OCR/eext-understandmn tasks: GQA (Hudson & Manning 2019), MMBench (MMB) (Liu et al. 2024c). MME (Fu et al. 2024). POPE (Li et al, 2023ch. ScienceQA (SQA) (Lu al. 2022). Tex (Singh et al. 2019) VizWiz (Gurari et al.
2018), and VQAv2 (Goyal et al. 2017).
We find that incorporating the user's query allows the model to perform better than the baseline and alternative compression algorithms on multiple different datasets. At the one-token level, our variant improves upon the existing algorithm on all benchmarks and can mitigate the original TokenPacker performance gap compared to vanilla I.Ia VA-1.5 by 34% on VizWiz and 122% on both POPE and MMBench. The trend continues at the four-token level. Table displays the full results.
RELATED WORK
4.1 TOKEN REDUCTION IN VISION-LANGUAGE MODELS (VLMS)
VLMs are composed of three key components: (a) a visual encoder that encodes the input images, (b) a language model (LM) that processes the visual tokens from the encoder along with the user
8

---

Published as a conference paper at ICLR 2025
text query, and (c) projector that maps the visual tokens to the input embedding space of the LM. Section A.I contains additional details exploring various projector designs. Often, the number of visual tokens (576 tokens per image for CLIP-ViT-L, for instance) significantly exceeds the number of text tokens, leading to high inference costs. This disproportionate scaling of visual tokens also hinders multi-frame integration due the limited context length of the model. Inference cost is a critical factor in many real world applications of computer vision systems. Thus, reducing the number of visual tokens processed by the language model has become an active area of research.
LLaVA-PruMerge (Shang et al. 2024) and Yu et al. (2024) propose training-free methods that filter out visual tokens (from CLIP) that have a low similarity with the CLS token. TokenPacker (Li al. 2024c), on the other hand, learns a compact token compression module using cross-attenting over visual tokens, allowing for reduced number of tokens while preserving salient information. While the above approaches focus on token reduction without directly changing the visual encoder (CLIP) output, recent works based on Matryoshka Representation (Cai et al. 2024 Hu et al. 2024) modify the CLIP output directly to generate nested CLIP embeddings for a flexible token count. Zhhng et al.
2024) investigate methods that emphasize task-relevant pixels during image processing.
Another approach to reducing inference cost is adaptive token processing, where the compute ded- icated to certain tokens at inference is varied Jain et al. (2024). Many methods prune visual to- kens within the LLM due their lower attention scores compared to the prompt, system, etc., tokens (Chen et al. 2024 Wan et al. (024), a heuristic commonly found in regular text-only LLM KV cache reduction techniques (Zhang et al.] [223]] Oren al. (024). Finally, while focus
paper on image-based VLMs, a host of works (Xu et al.) 22224] Shen et al. (0224 discuss token compression for video processing using VLMs. We defer discussion of these Section A.
4.2 SCALING LAWS
Understanding how the performance of modern deep networks shifts as key design factors, such as the number of parameters or training tokens, are scaled has become a focal point research, particularly as these models continue to grow in size and complexity. Scaling laws offer crucial guidance for optimizing the architecture of such models. Notably, Kaplan et al. (2020); Hernandez) al. (202I): [Hoffmann et al. (2022) do thorough investigation into training compute-optimat language models, highlighting the need to scale pretraining tokens and parameters at the same rate. Cherti al. (2023); Gadre et al.] (2023) perform a similar study on scaling laws for CLIP (Radford) et al.) 2021), corroborating that performance improvements arise from increasing both parameter counts and pretraining image-caption pairs.
Closest to our work, Li et al. (2024a) investigate what factors improve the performance of LLaVA (Liu et al. (003)) They observe performance gains with increasing language model size, visual encoder size, and input resolution. They investigate each of these factors when scaled in- dependently. In contrast, in this work we focus on understanding the optimal trade-off between language model size and the number of visual input tokens, given fixed inference budget to fit in. Note that in our work, visual input token count is varied (decreased) using token compression algorithms ($ D and not by varying the input image resolution using a different CLIP model.
While scaling the pretraining of LLMs has led to emergent capabilities, there has recently been growing interest in improving their reasoning capabilities by scaling inference time compute. Brown et al. (202)) show impressive performance boosts if the language model is allowed mul- tiple attempts on problem. In fact, [Snell al. (2024) show that scaling test time compute by parallel multiple generations at inference gives performance comparable to 14x larger model on math tasks. Goyal al. (2024a) show performance gains by appending special tokens the end of input to scale test time compute. In contrast, we characterize the optimal trade-off between tokens and parameters, for getting the best performance at given fixed test time (inference) compute.
DISCUSSION AND CONCLUSION
In our work, we demonstrate that the optimal trade-off for VLMs inference is to use very few visual input tokens along with the largest possible LLM that fits within the budget. This result has quite important consequences. Existing works aim towards moderate reduction in token count (c.g.. from 576 to 144), while trying to match the performance of the base model (no token reduction). However,

---

Published as a conference paper at ICLR 2025
results show that the community needs to focus towards extreme token reduction (e.g., down to 1, 4 or 16 tokens), as the inference optimal regime requires very few visual input tokens. Note that although extreme token reduction lead to drop in performance compared to the base model, it is still better than using more tokens with smaller L.M.. The performance with very few visual tokens is poised only improve further for extreme reduction. While our findings are focused on visual token compression at the projector level prior to passing into the LLM, we leave the compute-optimal scaling properties of adaptive token processing algorithms that operate within the LLM component for subsequent work. We hope that these critical insights from our paper will guide future research towards developing better token reduction techniques and thus inference optimal VLMs.
develop token reduction algorithms tailored
6
ACKNOWLEDGEMENTS
We thank Leslie Berberian, Devin Willmott, Qiu Chen, and Vijay Sadashivaiah at the Bosch Center for AI for useful discussions and help with running some of the experiments on Bosch's 's compute. We also thank Albert Gu for his feedback on the draft. KI. and SG are supported by funding from the Bosch Center for Artificial Intelligence.
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadch, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022. BRLLhttps:// httpp://///: org. hbs///b////0 00loa/2204.1133 14198
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL attps://arxe org/ abs 2309 16609
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling,
2024. wtthtttss:////// htttt::::aaaiiv.rrg////// abs 2407.4000 21787
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models, 2024.
URL htRls:ttp::/ http:///xxxvvv..aaaaaee abspgg//b//4/44
17430
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models, 2024. URL htthhtts:/// //tpxii////xvvv// org/ abs/ 2403. 06764
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive eagguage-image learning. In 2023 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023. doi: 10.1 00.1009/vp552229.20222 www.:/22............ 022600027..... URL
http://dx. doi. 10.g 1109 (11099//V252222
2023. 00276 76
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and Chunhua Shen. Mobilevim A fast, strong and open vi- sion language assistant for mobile devices, 2023. URL
16886
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards models with instruction tuning, 2023. /R////tpss///aaww...o//////////
langunle
10

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation org/ /bs/
benchmark for multimodal large language models, 2024. URL
2306. 13394
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. Eyal Orgad, Rahim En- tezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexan- der Ratner, Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023. URL aattps://arxiv: abs/ abs/2304.110 14108
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevie, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco. Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muen- nighoff, and Ludwig Schmidt. Language models scale reliably with over-trainig and on down- stream tasks, 2024. URL htttts:///aaiiii wwwpxiiiorgg/bs///oo htt.b////403..885. 08540
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, 2024a. URL
btttss///arxt org/ abs 2310. 02226
Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, and J. Zico Kolter. Scaling laws for data filtering- data curation cannot be compute agnostic, 2024b. URL https://
arxiv. htg/vvorrg/bb... abs 2404. 07177
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. 2017. URL
https://arxiv. http:/iwwwg///a/////x//o 2t..//1111.0087777
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018.
URL
org. abs 1802. 08218
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021.
abs abtp://aaa//2///.///// 220./11122.. 01293
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL attpss://azxi org/ abs/ 2203. 15556
Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large vision- languue models, 2024. URL bttts:///axiv.
brg/abss/405 2405. 19315
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, and Sujoy Paul. Mixture nested experts: Adaptive processing of visual tokens, 2024.
htRl:ttps:/ /tt////axvvvv//////////eel/aaa org/ abs/2407 22002440.2000 19985
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
11

---

Published as a conference paper at ICLR 2025
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi. and Ali Farhadi. diagram is worth a dozen images, 2016.
Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang. Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Leava-next: What else influences visual instruc- tion tuning beyond data?, May 2024a. URL https: pttpps://llaaa-200 ww..huuuu.....ubbaa..aaa watpllicooooooooo
2024-05- 25- lava next abl at ions/
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre pre-training with frozen image encoders and large language models, 2023a. URL https://
arxiv. www. abs /2001 12597
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Vidcochat: Chat-centric video understanding, 2024b. URL https://arxiv: org bogg/abs/en 2305. 06355
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang. Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. TToeepaccer: Efficient visual projector for multimodal Ilm, 2024c. URL https://arxxt
org/abs/ 2407. 02392
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-yid, vid: An image is worth 2 tokens in large language models, 2023b. URL httppss//aaxx...orrr/b///11.111. 17043.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models, 2024d. URL htttps://arxiiv org/ abss/4033 18814
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023c. URL https: org/
0oss/2305.000 10355
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2024a. URL
org/ aww./aaslle. 03744
Haotian Liu, Chunyuan Li, Yuheng Li, Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https://
1lava Ilava-vl.githun ww..ithub..... wtt.gitt.w..oo/oloooo///.oo/oo// wt..ii////////e 2024 01 30- alava-next..
Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better down- stream: Implicit bias matters for language models, 2022. URL httpt:://axxi..orrtttse/
141991119)
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024c. URL
06281
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations (ICLR), 2024.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL https://arxtt wttp:aawwwwww.r.ra.aaaaaaaaae /2003. 1102220...22230 10244
12

---

Published as a conference paper at ICLR 2025
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa document images, 2021. URL httpa///aaaxx...cc////////001110 003998
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi- state mns, 2024.
abs/ 22//2441..0 0610
06104
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
htttps:/aaxx...orrraa/////......
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal:: Accounting for inference in language model scaling laws, 2024, URL /tt.
abs/2401 00448
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models, 2024. URL httts::/arxxvv.rr//bb//
2403. 15388
Leqi Shen, Tianxiang Hao, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval, 2024.
URL https:/: /etttps:/aaxxv.orrtt abs/ 2409. 01156
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. WRRLhttps://axxiv.oom
abs/ 08920]
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https:: http:/:///xxvvvrrr/.e
3bcs/2405 03314
Yixuan Su, Tian Lan, Huayang Li. Jialu Xu, Yan Wang. and Deng Cai. Pandagpt: One model to instruction-follow them all, 2023. URL (nttps::: //trx/aaavvvv.//.///aa org /at/a/bb///////////
Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference, 2024. URL htttps:/:/ www.ziir.rrr//ooooooo abs 2206 18139
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2024a. URL aattps://arxiv.
0ttp/2311.0000
Xidong Wang. Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Iims 1000 images efficiently via hybrid architecture, 2024b. URL https://
arxiv.org/ abs 0209/44990000 02889
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022. UtLhttpps:// /wwpv..iivvv./////////../.a.aa org/ abs wttpg//bb/220..000 07636
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models, 2024. URL https://arxiv..rr
abs/2404. 03384
Jaqqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Slot-vim: Slowfast slots for video- language modeling, 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong
Sun. Minicpm-v: A gpt-4v level mllm on your phone, 2024. URL htt : /arxivvv wwx.v.orgg/..
abs /208.448.. 01800)
13

---

Published as a conference paper at ICLR 2025
Gaotong Yu, Yi Chen, and Jian Xu. Balancing performance and efficiency: A multimodal large language model pruning method based image text interaction, 2024. URL htttps://axxiv.. org/abs/2400 01162
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https:::
//arriiv.rrg/abe/243..009 09629
Jiaxin Zhang, Wentao Yang. Songxuan Lai, Zecheng Xie, and Lianwen Jin, Dockylin: A large multimodal model for visual document understanding with efficient visual slimming, 2024. URL
19101
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. Hzo: Heavy- hitter oracle for efficient generative inference of large language models, 2023. URL https:::
//axxiv.rgg/abs/
14048
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging aa-juge with mt-bench and chatbot arena, 2023. URL/https:/aaxxi...
abs/23066 05685
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of scall-scale large multimodal models, 2024. URL https::/aaxii..ooott
abs/2402 14289
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023. URL https://
arxiv. wrkw.oorg/abs/2304.. 10592
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL RELATED WORKS
A.1.1 VISION PROJECTOR DESIGN
To bridge the gap between the separate image and text modalities presented by the vision encoder and language model respectively, vision projectors map the image tokens from the vision encoder into the language space. Many design choices for the projector exist. Numerous VLMs utilize query-based projectors, which combine the embeddings of visual tokens with that of query tokens via cross-attention similar mechanisms, like the Q-Former projector introduced BLIIP-2 (Li al.) 2023a) and used in following work (Dai et al. 2023 Zhu et al., 2023). Other VLMs use simple linear projectors MLPs connect the encoder and LLM (Liu et al., 2023; 2024a Su al. 2023 While most architectures use the projectors to create new tokens to feed into the LLM alongside text, some architectures like Flamingo (Alayrac et al.] [022) or CogVLM (Wang al.) (2024a) directly interweave the visual information into the language model. In our work, we will be focusing on projectors that fall in the former category.
A.1.2 ADDITIONAL APPROACHES FOR EFFICIENT VLMS
Apart from reducing the number of visual input tokens to the language model, people have explored various other techniques, including a mix quantization (Liu al. 2024a) and smaller encoders or language models Yao et al. 2024 Chu et al. 2023 Zhou et al. 2024 for improving inference.
VLMs utilized in video processing often combine decreases in vision encoder output size with token compression techniques to prevent excessive latency and memory constraints. Visual tokens are often merged temporally across frames (Xu et al.) 2024, Shen et al., 2024) as well as spatially for individual frames (Xu et al.] 2024). Vision encoders, such as Q-Former (Li et al.] (223a), are preferred over more traditional CLIP models due to their ability to extract a smaller fixed number of tokens per image (Weng et al.
2024 Li al. 2024b). Although compression techniques used for video processing often can reduce token counts by large margins, they are rarely evaluated on image datasets, and when they are, compress visual tokens very little not at all (Li al. 2023b).
Adaptive token processing, where the compute dedicated to certain tokens during inference is var- ied Jain al.] (2024), another approach to reducing the cost of inference. Many methods prune visual tokens within the LLM due to their lower attention scores compared to the prompt, system, etc., tokens (Chen et al. 2024 Wan al. 2024). heuristic commonly found in regular text-only LLM KV cache reduction techniques (Zhang et al. 2023 Oren et al. 2024). Finally, while we focus our paper on image-basee VLMs, host of works (Xu al. 2024] Shen et al. (2024) discuss token compression for video processing using VLMs. We defer a discussion of these to Section A.
A.2 GRID SEARCH DETAILS
While there are many choices of optimizer for fitting the scaling laws like curve-fitting in SciPy, gradient descent based solvers, etc. We observed that these are not stable and give varying so- lutions. We converged to using grid-search to fit the scaling laws, similar the recent works like Goyal et al. (024b). The grid-search range for each of the parameters were as follows: a, 3 E [0,0.1), B. (0,1), 1}.
A.3 ADDITIONAL RESULTS FOR SCALING LAWS
We find that our original scaling laws are able to generalize and predict the performance of VLMs at the 14B scale despite only being fitted up the 7B scale. Our predictions result in less than 2% error between the predicted and actual VLM performance on visual reasoning and understanding tasks at the 14B model parameter scale. Performance is measured as described in Section 3.2
15

---

Published as a conference paper at ICLR 2025
Accuracy of the estimated Scaling Laws
Accuracy of the estimated Scaling Laws
Avg. Downstream Error
0.625
14B
Avg. Downstream Error
077
677
0.600
#L Params (N)
0.600
#LMM Params (N)
0.575
0.575
0.550
0.550
-79
-78
0.500-
0.475
0.475
0.450
144
1.88
0.450
144
1.88
576
576
0.58
0.58
100
10.
10a
10a
10
102
10'
10
Inference FLOP (O(NV)))
Inference FLOP (o(N(50 + V))))
(a) Scaling law prediction for 14B LLM VLM (b) Scaling law prediction for 14B LLM VLM Q
Figure 6: Scaling law predictions at various Q. The scaling laws fitted based LLMs up to the 7B scale generalize well to the 14B scale, resulting in less than 2% error between predicted and actual VLM performance.
Overall VLM Architecture
Query-Based Compression
LLM
Fed trabedded
Visual Tokens
Rechaze
Query-Based Compresaion
Query
V
Vision Encoder
Convolationl Downnampling
Cross Attention
-
MLP
Key Value
Rechape
Text Tokens
Cempressed
Visul
Figure 7: Our query-based convolutional cross-attention compression technique. User input text tokens are first processed through the LLM backbone to generate text embeddings that then combined with the visual tokens. Within the projector, the query-embedded visual tokens are downsampled via convolution. Next, local cross-attention is applied between the downsampled tokens and their respective visual tokens regions. The compressed tokens finally pass through an MLP into the LLM.
B PROMPT-BASED TOKEN COMPRESSION
Our scaling laws show that, for visual reasoning tasks, it is optimal utilize the largest LLM back- bone by reducing the number of visual tokens to remain within budget. Many current token compres- sion algorithms focus on compression 144, 64, 36 tokens; however, the token count can be as low as for many situations. Thus, improving the ability of extreme compression algorithms would enable the use of larger LLM-bbccbones for the same inference cost, achieving optimal inference compute usage.
The following section details our updates to the existing token compression algorithm (Li et al., 2024cb to incorporate query-based token compression. Figure 7 summarizes our query-based con- volutional cross-attention compression technique.
User Query Information Injection: To make our projector prompe/query-depeddent, we add the text embedding the user's most recent prompt to the image embeddings from vision encoder. We do this by taking the last hidden states prior to the LM head of the user input from the language model as the representation embedding via linear projection and added to the image visual token embeddings. These fused tokens are later used as the query component for cross-ate gross-attention. The text-embedding can easily be cached for applications where the prompt is static part of predetermined set. Even if
the user's overall query. The hidden state converted into the text
16

---

Published as a conference paper at ICLR 2025
Method
# Token
GQA
MMB
MME
POPE
SQA
TextVQa
VizWiz
VQAv2
LLaVA-1.5
576
62.0
64.3
1510.7
85.9
66.8
58.2
50.0
78.5
PruMerge
r~32
57.2*
60.9
1350.3
76.3
68.5
56.0
45.2*
72.0
TokenPacker (TP)
36
59.6
62.8
1440.9
83.3
71.0*
53.2*
50.2
75.0
Matryoshka Multi.
36
60.3
64.8
85.5
52.8
Matryoshka Query
36
58.8
63.4
1416.3
81.9
66.8
51.0
73.7
Updated TP (Ours)
36
60.5
62.5
1442.0
84.5
70.6
53.3
50.1
75.8
TokenPacker
58.9*
62.7%
1378.8*
83.7*
68. 1
52.5
50.5*
74.4*
Matryoshka Query
57.6
61.9
1408.5
67.5
49.8
71.1
Updated
59.0
62.2
1408.0
83.4
70.7
51.3
74.5
TokenPacker
56.2*
61.5*
1347.6*
81.7*
68.5
49.2
45.7*
70.5
Matryoshka Query
53.0
56.5
1176.1
77.6
65.1
49.4
64.1
Updated TP
56.5
62.1
1390.3
81.8
68.6
48.7
45.0
70.6
TokenPacker
53.4*
58,7* 75
1262.4* *
80.7
69.4
46.2*
41.1*
66.9
Matryoshka Multi
52.6
59.5
78.4
49.4
Matryoshka Query
50.8
54.4
1144.0
74.5
65.0
48.5
61.0
Updated TP
53.5
59.4
1269.1
81.3
69.9
46.8
44.1
67.3
No Visual Tokens
37.7
21.0
697.8
43.4
63.6
41.7
44.4
41.0
Table Comparison of various token compression methods for VLMs at different compression rates. All models use the Vicuna-1.5 7B model as the language backbone. A denotes benchmark results for other techniques we evaluated, while best scores are bolded, and second best underlined. Our updated method outperforms alternatives, including its predecessor, on almost all benchmarks extremely high compression regions (visual tokens reduced or 4), highlighting the benefit of utilizing the user query in the compression algorithm for these situations.
the prompt varies, the rext-embedding can be pre-calculated prior to processing the image and KV values cached and re-used when processing the visual and text tokens together for generation.
Token Downsampling with number of visual tokens passed into the L.M,, we utilize region-based, cross-attention mechanism that downsamples the vision encoder tokens, X, into a more information-dense form. The mecha- nism hinges on the property that the X can be viewed patchification of the image. Li et al. (2024c d) passes the "2D" version of X through a downsam- pling function that compresses the input by s factor where each resulting token corresponds with a $ X s region in the original input. After this, cross-attention each downsampled token and the corresponding tokens in its ear interpolation-base based downsampling techniques (Li et al., 2024c Wang et al. (022b) by using a learnable depth-wiss wise 2D convolution filter of kernel size and stride s, providing better expressivity.
and Learnable Convolutions: To compress the
/ v grid due to the vision encoder's
applied independently between region. We improve upon bilin-
B. QUERY-BASE BASED CONVOLUTIONAL CROSS- ATTENTION RESULTS
Table presents the results of our QueCC algorithm in comparison to previous methods, includ- ing TokenPacker (Li et al. 2024c), LLa VaruMrrrerr (Shang et al.) 2024). Matryoshka Multimodal Models (Cai al. 2024 and Matryoshka Query Transformer (Hu et al. 2024), in low token regimes. We find that at our method performs better than alternatives at the highest levels com- pression multiple different datasets. At the one-token level, method outperforms other meth- ods on six of the eight datasets and is able to mitigate some the shortcomings of the original TokenPacker by reducing its gap between vanilla LLaVA-1.5 on VizWiz by 34% and ~12 on both POPE and MMBench. The trend continues at the four-token level. Our model also exhibits strong performance on GQA, MME, SQA, and VQAv2 across compression rates, signaling the prospects of using the user's query to identify key tokens.
B.2 ABLATIONS OF PROMPT-BASED TOKEN COMPRESSION
We ablate the importance of the query injection and convolutional downsampling components and report the results in Table 2 We find at extreme levels of compression that combining query and convolution can magnify the benefits either adding only query or only convolution; e.g., TextVQA performance token count increased by 0.7 percentage points (pp) with both convolution and
17

---

Published as a conference paper at ICLR 2025
Token
Model
GQA
MMB
MME
POPE
SQA
TextVQA
VizWiz
VQAv2
Conv and Query
53.5
59.4
1269.1
81.3
69.9
46.9
14.1
67.3
Query Ouly
53.3
59.2
1267.7
81.3
68.8
46.3
41.7
66.6
Conv Ouly
53.6
57.5
1215.5
80.6
69.1
46.4
45.6
66.7
No Conv, No Query
53.4
58.7
1262.4
80.7
69.4
46.2
41.1
66.9
Conv and Query
56.5
62.1
1390.3
81.8
68.6
48.7
45.0
70.6
Query Only
56.4
62.0
1345.9
82.3
70.7
70.6
Conv Only
56.7
60.6
1310.4
82.1
69.0
49.4
No Conv, No Query
56.2
61.5
1347.6
81.7
68.5
49.2
70.5
16
Conv and Query
59.0
62.2
1408.0
83.4
70.7
51.3
47.7
74.5
Query Only
56.6
61.4
1354.3
82.1
69.6
50.7
41.2
71.5
Conv Only
58.9
62.5
1402.3
82.5
69.6
52.6
45.7
74.1
No Cony, No Query
58.9
62.7
1378.8
83.7
68.1
52.5
50.5
74.4
Table 2: Comparison of model ablations across different token counts and configurations, Best scores are bolded, and second- becond-best scores are underlined for clarity, Adding both query and convo- lutional components can help boost the baseline performance and can mitigate performance drops that are associated with each individual component.
query while using only one of the components led to at most 0.2 pp increase. addition, combining the two can mitigate performance drops that are associated with utilizing only query convolu- tion, as seen MMB at one token where using only convolution drops performance by more than pp but performance outperforming the baseline by 0.7 pp: similar situation can be seen for MME.
only be restored but also improved when adding query, eventually
We also compare with LLaMA- VID eLi al. (023b) which also has strong performance in extreme com- pression regimes. We compare the performance of method its ported performance at similar token compression levels and show that are able to outperform it in certain tasks despite LLaMA- VVID utilizing a stronger vision encoder (Li et al. 2023b). Our analysis shows both our approach and theirs are compet- itive, which also validates the key point wanted make that query-based compression is necessary under extreme compression. In addition, LLaMA- utilizes a separate text decoder model to process the user query, while our method utilizes the existing LLM within the VLM model.
Token
Model
GQA
POPE
SQA
TextVQA
16
LLaMA-VID
58.2
83.1
67.4
50.8
QueCC
59.0
83.4
70.7
51.3
LLaMA-VID
56.2
83.5
68.7
49.1
QueCC
56.5
81.8
68.6
48.7
Table 3: Comparison of LLaMA-VIL and QueCC mod- els across different visual/content token counts. LLaMA- VID results, obtained from (Li et al.) 2023b), utilizes the context tokens, resulting in one addition overall token.
18


Reviews:
[{"review_id": "ubFTJ1Eu2b", "reviewer": "Reviewer_78vE", "summary": "This paper investigates the balance between the number of visual tokens and the size of LLM  in Vision Language Models to optimize inference costs. The authors discover that for visual reasoning tasks, using the largest feasible LLM with just one visual token is the most compute-efficient. They introduce a prompt-based token compression method for high compression ratios, which focuses on selecting relevant tokens based on user queries. Experiments show that their approach outperforms other compression techniques at extreme token reductions, highlighting the importance of developing algorithms for extreme token compression. The findings suggest that for visual reasoning, prioritizing a larger LLM over more visual tokens is crucial for maintaining performance within limited inference budgets.", "questions": "1. Will one visual token be enough for other tasks (e.g, VLMs for detection)? The focus on minimizing the number of visual tokens to a single token might risk overfitting to the specific datasets used in the experiments. It's uncertain how well this extreme compression would generalize to unseen data or datasets with different characteristics. \n2. Will the proposed scaling law generalize to other VLM architectures? The authors only conducted experiments on one type of VLM.\n3. What is the inference time and performance trade-off? Does the scaling law still hold? The computed FLOPs can be inaccurate.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of establishing the scaling law among visual token numbers and model sizes is interesting. They explore optimizing inference costs in VLMs by using a single visual token with the largest possible LLM within a given budget.\n2. The paper offers a thorough analysis of the trade-offs between LLM size and the number of visual tokens, covering various scenarios and use cases. This comprehensive approach provides a deeper understanding of VLM optimization.\n3. The paper is well-organized and written in a clear and concise manner.", "weaknesses": "1. The main concern is the generalization of the scaling law. The paper focuses on visual reasoning tasks and may not fully explore other types of tasks where a single visual token might not be sufficient. For instance, tasks that require detailed image analysis might not benefit as much from such extreme token compression. \n2. While the scaling laws developed in the paper are insightful, they are based on a specific set of experiments and models. The findings are heavily dependent on the specific LLMs and VLMs used in the experiments. Different architectures might yield different optimal points in the trade-off between the number of visual tokens and LLM size, which could limit the applicability of the results. It's unclear how these laws would apply to other VLM architectures or if they would hold as new more complex models are developed in the future.\n3. The proposed prompt-based token compression method adds complexity to the VLM pipeline. This could make the approach more difficult to implement and integrate into existing systems compared to simpler token reduction techniques.\n4. The paper does not discuss the training costs associated with the proposed compression method. It's possible that training such a method to be effective, especially at high compression ratios, could require significant computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qEBOL9sxL", "reviewer": "Reviewer_iUzh", "summary": "This paper explores optimizing VLMs to reduce inference latency by balancing model size and visual token count. The authors demonstrate that, for visual reasoning tasks, using the largest feasible LLM within a fixed inference budget, while drastically minimizing visual tokens (often to a single token) yields optimal performance.", "questions": "Refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper provides valuable insights showing that larger LLMs enhance visual reasoning performance more than reducing visual tokens. Also, the introduced compression algorithm QueCC demonstrates better performance on benchmarks with high compression, proving its effectiveness.", "weaknesses": "While the paper demonstrates that larger model sizes can be more effective than increasing token counts for visual reasoning tasks, this approach appears less effective for OCR-related tasks, as acknowledged by the authors. Given that many real-world applications require fine-grained visual understanding, the proposed compression method may not fully address these demands, as evidenced by its performance on the TextVQA benchmark in Table 1. Although the authors provide valuable insights, their claim that \"only one visual token is needed\" may be overly naive, given the trend in VLM research toward advancing fine-grained visual comprehension capabilities.\n\n---\n\nThe authors somewhat addressed my concerns, although some efficiency perspective seems insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qeKRgbUUNS", "reviewer": "Reviewer_WUho", "summary": "The authors unveil the inference time scaling law, which characterizes the optimal tradeoff between the number of visual tokens and LLM parameters. The law reveals that the larger LLM matched fewer vision tokens under the fixed inference budget is the most optimal. Besides, the paper proposes a prompt-based VLM compression manner dubbed QueCC and conducts comprehensive experiments on it.", "questions": "1. The author uses the scaling law of [1] as an analogy. They replace the length of the text token in [1] with the length of the vision token and get the reversed conclusion. However, since section 3.3.2 points out the influence of text token length, it is more reasonable to include it in the discussion of Formula 2.\n2. Although this paper exceeds the page limit, it will not affect my judging.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors do not limit the compression of VLM to vision, but dynamically explore the relationship between LLM and visual tokens. The motivation of the paper is sufficient.\n2. The practical guideline provides a novel insight into VLM efficiency. The author carefully points out the limitations of the scaling law, which is not applicable to OCR tasks.\n3. The authors conduct comprehensive experiments on various metrics and show an improvement. Furthermore, the discussion is in detail.", "weaknesses": "1. The employment of cross-attention in QueCC to compress information is common [1].\n2. LLaMA-VID [2] and VoCo-LLaMA [3] have already done token compression in extreme regimes, which is impressive. The author should compare their performance with QueCC. It seems that QueCC is inferior to VoCo-LLaMA on benchmarks such as GQA and MME.\n3. There is a lack of ablation experiments, especially the analysis of depth-wise 2D convolution and the injection of text-embedding.\n4. The authors’ scaling law seems to have no direct relationship with the method they proposed.\n\n[1] Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A. and Carreira, J., 2021, July. Perceiver: General perception with iterative attention. In *International conference on machine learning* (pp. 4651-4664). PMLR.\n\n[2] Li, Yanwei, Chengyao Wang, and Jiaya Jia. \"Llama-vid: An image is worth 2 tokens in large language models.\" In *European Conference on Computer Vision*, pp. 323-340. \n\n[3] Ye X, Gan Y, Huang X, Ge Y, Shan Y, Tang Y. VoCo-LLaMA: Towards Vision Compression with Large Language Models. arXiv preprint arXiv:2406.12275. 2024 Jun 18.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bL931wYPJX", "reviewer": "Reviewer_jj75", "summary": "This paper discusses the challenge of high inference latency in VLMs. The authors explore the optimal balance between the number of visual tokens and LLM parameters for a fixed inference budget. They find that the optimal method in visual reasoning tasks is achieved using the largest possible LLM and minimizing visual tokens, even just one token. So they also propose a new approach using prompt-based compression for high-compression settings.", "questions": "1. Could you explain more about the discrepancy of the result between previous work and your work in **Log-Linear Relation between Error and Number of Visual Input Tokens**? Why limited downstream benchmarks lead to the discrepancy?\n2. The method is quite similar to LLaMA-Vid, but the paper didn't compare it in its experiment and didn't show the differences between them. Have you tried to compare QueCC with LLaMA-Vid and VoCo-LLaMA?\n3. Is the Convolutional Downsampling necessary? By adjusting the number of MLP output tokens, I can also adjust how many tokens to compress. There is a lack of ablation study proof for both User Query Information Injection and Convolutional Downsampling.\n4. In User Query Information Injection, if this is the first time for visual token and text token entering this LLM, there is no way to get the last hidden state of the text, how can I perform such an operation?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper discusses balancing the number of visual tokens and LLM parameters for a fixed inference budget, which is a novel perspective for studying effective methods to accelerate speed.\n2. They also propose a method that makes the paper more comprehensive. \n3. The writing is clear and concise.", "weaknesses": "1. The method is quite similar to LLaMA-Vid, but the paper didn't compare it in its experiment and didn't show the differences between them.\n2. Experiments are insufficient. The paper could also compared with VoCo-LLaMA[1] and LLaMA-Vid[2], which are also efficient in high-compression settings. In addition, they lack an ablation study.\n\n[1] Ye, Xubing, et al. \"VoCo-LLaMA: Towards Vision Compression with Large Language Models.\" arXiv preprint arXiv:2406.12275 (2024).\n\n[2] Li, Yanwei, Chengyao Wang, and Jiaya Jia. \"Llama-vid: An image is worth 2 tokens in large language models.\" European Conference on Computer Vision. Springer, Cham, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iacvuGfnyH", "reviewer": "Reviewer_Laf9", "summary": "This paper addresses a crucial trade-off between visual token count and language model size in Vision-Language Models (VLMs). The authors argue that for visual reasoning tasks, optimal inference performance is achieved by maximizing the LLM size within a given inference budget, even if it means drastically reducing the number of visual tokens. They propose a novel method, \"QueCC\" for compressing visual tokens at inference time, demonstrating significant improvements in accuracy and efficiency.\n\nQuoting from the abstract: \"for visual reasoning tasks, the inference-optimal behavior in VLMs is achieved by using the largest LLM that fits within the inference budget while minimizing visual token count — often to a single token.\"\n\nAnd is shown with a logical experimental setup along with strong baselines. The authors also outline the potential shortcoming and also show that visual recognition and textual recognition from images have different goals and the more tokens are often needed in the second case to hold on to accuracy.", "questions": "The paper is clear and achieves a strong threshold for the problem defined. While one can further improve the paper, I think the paper as is is strong enough to be accepted to ICLR. \n\nI am happy to champion the paper unless other reviewers find something glaring I am missing.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper tackles a highly relevant problem in the rapidly evolving field of VLMs. Balancing computational resources between visual processing and language modeling is critical for achieving optimal performance.\n\n1) Well-designed Experiments: The authors conduct thorough experiments on various visual reasoning benchmarks, including GQA, CLEVR, and SNLI-VE. They systematically vary the number of visual tokens and LLM sizes, providing valuable insights into the relationship between these factors.\n\n2) Strong Empirical Results: The proposed token compression method consistently outperforms baseline models, achieving state-of-the-art results on several benchmarks. The gains are particularly impressive at extremely low visual token counts, demonstrating the effectiveness of the approach in resource-constrained scenarios.\n\n3) Clarity and Presentation: The paper is well-written and easy to follow. The authors clearly explain their motivation, methodology, and results, making the contributions accessible to a broad audience.", "weaknesses": "Have no major weaknesses from a technical standpoint. However, the related work can have a bit more coverage. \n\nWhile the paper provides a comprehensive overview of token compression techniques, it could benefit from a more in-depth discussion of related work on adaptive compute -- be it in Dynamic Sparsity, Elastic models (MatFormer, Flextron etc.,) and early exits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)