VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning
Authors: Run Luo
Renke Shan
Longze Chen
Ziqiang Liu
Lu Wang
Min Yang
Xiaobo Xia
Venue: NeurIPS 2025 poster ()
Keywords: Large vision-language models
vision concept modeling
token compression
efficiency
Primary area: applications

Abstract: Large vision-language models (LVLMs) have emerged as foundational tools for real-world AI applications. Despite their remarkable capabilities, current LVLMs process entire images at the token level, leading to significant inefficiencies compared to human cognition, which selectively focuses on high-level vision concepts. This token-level redundancy becomes increasingly problematic for high-resolution images and long video sequences, resulting in large computational costs and limited scalability in practical applications. To address this limitation, we introduce the concept of a vision concept model, a novel paradigm that enables LVLMs to dynamically extract the most relevant vision concepts from complex inputs, based on task-specific instructions. To optimize this vision concept modeling process, we propose VCM, a self-supervised framework that leverages vision-language correlations across diverse instances. VCM is designed to learn meaningful vision concepts without the need for expensive concept-level annotations. At its core, it employs a forward-backward optimization algorithm that supports LVLMs to adjust concept granularity and spatial alignment dynamically. Experiments demonstrate that VCM remarkably reduces computational costs (e.g., achieving up to 85\% fewer FLOPs for LLaVA-1.5-7B), while maintaining strong performance across a series of vision-language tasks. The codebase is available at https://github.com/RainBowLuoCS/VCM.

VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tunine
Run Luo 1,2,3
Renke Shan' Longze Chen
Zigiang Liu 1:2
Lu Wang'
Min Yang 1.3* Xiaobo Xia 1,5x
Shenzhen Key Laboratory for High Performance Data Mining,
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences
Shenzhen University of Advanced Technology
"Naional University of Singapore
Moo Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University Science and Technology of China
P.LLUOSSIAT.CC. MIN.YANG @ SIAT.AC.CN
XIAOBOXIA.UNI GMAIL.COM)
Abstract
Large vision-language models (LVLMs) have emerged as foundational tools for real-world AI applications. Despite their remarkable capabilities, current LVLMs process entire images at the token level, leading to significant inefficiencies com- pared to human cognition, which selectively focuses on high-level vision concepts. This token-level redundancy becomes increasingly problematic for high-resolution images and long video sequences, resulting in large computational costs and lim- ited scalability in practical applications. To address this limitation, introduce the concept of a vision concept model, a novel paradigm that enables LVLMs to dynamically extract the most relevant vision concepts from complex inputs, based on task-specific instructions. To optimize this vision concept modeling process, we propose VCM, self-supervised framework that leverages vision-language correlations across diverse instances. VCM is designed to learn meaningful vi- sion concepts without the need for expensive concept-level annotations. At its core, it employs a forward-backward optimization algorithm that supports LVLMs adjust concept granularity and spatial alignment dynamically. Experiments demonstrate that VCM remarkably reduces computational costs (e.g.. achieving up 85% fewer FLOPs for LLaVA-115-7 1---... while maintaining strong perfor- mance across a series of vasion-language tasks. The codebase is available at https: / com./RainBoLLuCC//OOO
Introduction
Large vision-language models (LVLMs) [1, 2, 3, 4, 5, 6, 7, 8] play critical role in addressing wide range of vision-language tasks and have become cornerstone for enabling general artificial intelligence to interact with the real world, such in embodied intelligence [9, 10] and autonomous driving [11, 12, 13]. However, current LVLMs process entire images at the token level, which is inefficient compared to humans, who analyze information and generate content at the conceptual level, extracting relevant vision concepts with minimal effort. This inefficiency becomes particularly pronounced when dealing with higher-resolution images or longer video inputs, which humans can easily handle, but LVLMS struggle with it due to the rapidly increasing computational cost. This
"Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A: Black (16 vision tokens)
A: Black (376
vision tokeus)
KM Visuaatzatoon
Language Model
O: What color is the dog's collar?
w/ VCM
Gap Reduced and Time Save!
VCM
Figure I: Illustrations of VCM in enhancing efficiency and dense perception capability. VCM can select relevant vision concept tokens based on instructions, significantly reducing redundant attention computations LVLMs. This reduction lowers both training and inference costs while maintaining strong performance, as shown in (a). Additionally, VCM enhances the dense concept prediction capability of the vision encoder, as illustrated in (b) through K-Means visualizations of dense feature maps from the last layer of CLIP VIT. These improvements enable broader applicability dense perception vision-language tasks.
limitation, arising from the lack of vision concept model, restricts the usability of LVLMs in practical applications.
In this work, formally define the concept of vision concept model. The vision concept model can dynamically determine the required vision concepts, including their quantity and spatial locations, based given instructions. Note that previous methods [14, 15, 16] have attempted improve the efficiency of LVLMs by compressing vision tokens using variable-length queries or pruning the tokens through attention mechanisms. However, these methods fail provide semantically meaningful concepts grounded in the corresponding spatial locations of vision inputs, making them unsuitable for vision-language tasks beyond visual question answering As such, they cannot be categorized as vision concept models.
To bridge this gap, propose VCM, supervupe vision concept modeling framework. Specifi- cally, VCM leverages vision-language correlations across multiple sampled instances and employs vision-language fine-tuning to build vision concept model, without costly concept-level annotations. To enable the optimization with theoretical support for the vision tokens of varying lengths, we design forward-backwant algorithm based on the dynamic programming process [17], which supports adaptive and dynamic length optimization. This allows the vision concept model to dynamically output required vision concepts and keep their corresponding spatial locations according to given instructions, as shown in Figure 1. Benefiting from the vision concept model, VCM not only signifi- cantly reduces the computational cost of training and inference in LVLMs (e.g.. 85% fewer FLOPs for 115-7B while maintaining strong visual question answering performance but also enhances the vision encoder's capabilities in other dense perception vision languaee tasks, such as zero-shot image classification, open- vocabulayy object detection, and open-vocabulary semantic segmentation.
Before delving into details, our main contributions can be summarized as follows. (1) We formally define the vision concept model, which dynamically determines the required vision concepts, includ- ing their quantity and spatial locations, based on given instructions. This model is applicable variety of vision-language tasks. (2) We propose the VCM framework, which employs the correla- tions between vision and language across multiple sampled instances and a theoretically supported forward-backwart algorithm to enable dynamic vision concept learning without requiring expensive fine-grained annotations. (3) We conduct extensive qualitative analyses, quantitative experiments, and ablation studies to validate the effectiveness and efficiency of VCM.
We review related works and detail their difference from method in Appendix A.

---

(b)
(c)
Figure 2: Correlations between vision tokens and text prior. (a) Positive correlation between the text response and minimum vision tokens: the more image-related keywords in the text response, the longer the minimum token length required. (b) Negative correlation between the text instruction and minimum vision tokens: the more image-related keywords the text instruction, the shorter the minimum token length required. (c) Negative correlation between the difference in text response and instruction and minimum vision tokens: the greater the gap in image-related keywords between the text response and instruction, the shorter the minimum vision token length required.
2 Preliminaries
Problem setting. The LVLM architecture generally consists a vision encoder fv. a modality pro- jector fM. and a large language model (LLM) fLLM with its language model head (LMH) fIMH- The vision encoder, typically a pre-trained image encoder like CLIP's vision model [18], converts input images X into vision tokens fv ) The projector aligns these vision tokens with text instruction X encoded by the LLM's 's word embedding X", which enables the LLM to process vision data effec- tively. For simplicity, where M denotes the length of vision tokens, N denotes length of text tokens, and d is the hidden dimension size. That is usually M >> N. The LLM then integrates the aligned vision information and instructions to produce response X which is learned as po R f (LLM([LLHH (HV; H'))), pa- rameterized by 0. Note that due to large number of vision tokens, the training and inference of LVLMs are inefficient [19]. Therefore, a vision concept model exists that can decrease the value of M, the efficiency of LVLMs during both training and inference will be improved'.
define that H fM([v(XX
E Rx and H' X'
R-xxd
Goal and challenges. The primary goal of vision concept modeling (VCM) is to learn relevant vision conceptual information H. from H based language priors. The vision conceptual information can be applied to various vision-language tasks, leading to significant improvements in performance or efficiency. There are two key challenges in VCM. (1) Fine- Fine-grained labeled data scarce, as manual annotation is labor-intensive, abundant unlabeled visual question answering (VQA) datasets effectively, (2) The length of vision concepts varies dynamically across different examples, making it challenging to model and optimize. Below we present how to handle the two challenges and implement VCM.
and inefficient. This makes it difficult to leverage the
Vision Concept Modeling
3.1 Correlations between Vision Tokens and Text Prior
The proposed VCM utilizes the correlations between minimum required vision tokens and text priors. For instance, general instruction like "Can you describe this image in detail?" typically demands information from the entire image, resulting in longer and keyword-rich responses. In contrast, specific prompt like "Where is the person in yellow?" focuses on a distinct region, requiring less vision information and yielding shorter responses. To address the challenge of limited fine-grained annotations and to explore this relationship, we sample 5K instances from the LLaVA instruction fine-tuning dataset [20] and use GPT-4o to identify image-related keywords in both responses and instructions. We then applied VisionZip [16] with 24 different vision token lengths to generate
We provide more analysis the efficiency improvement from the Stating--oinn operation (FLOP) perspec- tive. Interested readers can check Appendix D details.
Herr HV is sparse version of H.. with ||H |H llo.
3

---

Semantic Aligmment Pretraining
Instruction Token
W here the person in yellow
Keyword Sclectur Module
Keyword Selector Module
Avg Posling
I. arge Language Model
Global
Giktal
Softmarlfas (""" B(K)
Toben
Tsaa Semantic A lignment
Taken
VCM
(b) Keyword Selection
here the person in yellow
VienFuoddr Priielae
Projection
Vison Incoder
W and Embbelding
CrossA PresAttion Attention
LoCK
persen
passen yallows
Sebeeter 8
Visual Inputs
Instruction & Response
omasks yelloww
Taken
Global Viddos
Concept
Viion
Bespesse
Tokea
Ioput
parsan
Takes
Video
Yokon
Output
(a) Workflow Architecture
(c) Vision Concept Modeling
Estimated pseudo label
Figure 3: Overview of our VCM framework. (a) The workflow architecture: vision concepts are extracted from image inputs based on instruction priors and fed into the L.M to generate correspond- ing answers. (b) Adaptive keyword selection module: image-relevant keywords (highlighted red) selected by calculating text-image similarity, retaining keywords with scores above the average. (c) Implicit contrastive sampling module: keywords the instruction are randomly masked, and the VCM loss is computed with input image features for end-to-end optimization.
responses via the LLaVA model, using GPT-4o as judge to determine the minimum necessary token length. Experimental results in Figure 2 indicate a negative correlation between keyword count in instructions and the required vision token length, while the keyword count in responses difference in keyword counts between responses
showed a positive correlation. Furthermore, and instructions reduces noise and exhibits more stable negatíve correlation. We also estimated the vision concept length by scaling the vision token length by v2 after local merging. This adjustment shows more consistent correlation with text priors, which provides an efficient way to estimate vision concept length for self- supe-sueee modeling.
3.2 Semantic Alignment for Keyword Selection
To efficiently and accurately extract keywords, we designed keyword selector module fks posed of a multi-head self-attention (MHSA) layer for semantic alignment optimization. Dur- ing the pre-training phase, obtain the global text features G the global vision features GV. and the global language model text features G through keyword selector module fKs. vi- sion encoder fv. and large language model fLIM by global average pooling, respectively, For- mally, we have G AvgPool (MHSAA[[ (xx") (
and GLiM AvgPool (foell/fMMMMMMM x where [:] denotes the concatenate operation, and fv2t repre- sents the vision to text linear project in modality project fM. Then, map GT, G and GT LLL into semantic space by the language model head fiMI and calculate the semantic alignment (SA) loss as
)). G fver VirT(Avgr
LSa = -p((mm (P((MMMGG[MM) (ogp(iii(("')))
pl fi (MMH(GILM) log pi..((MMM''
To better incorporate text prior, we adopt a multi-head cross-attention (MHCA) layer followed by vision-to-text linear layer fvzr as a modality projector fM to extract vision information and align it to the text space as follows:
S = AvgPool (x' (x(3( 1o00=//roooo/ss(XX/// HH, (MHCAA" ('eeftvv)) (vx
))), (1)

---

where S denotes the difference of instruction and response, frzv represents the text-to-vision linear transformation used for learnable vision query QV, and @ means the broadcast add operation. Then the total loss in the pretraining stage can be defined as:
fil log p( X ( (J (JIMM/G/" H H)))) + a LSA,
where H' = X phase, the frozen keyword selector efficiently extract accurate keywords, which are then used for vision concept modeling. For example, the keywords in the instruction and response are adaptively determined using the following formula:
T and a is the constant coefficient setting to 0.05. In the instruction fine-tuning
K Sof Soffmaxx(MHSAA (H H ((G)) = Sof Softmax(MHSA X/)
(2)
where an indicator function (A > (K)) is used to extract the keyword set from X and X
Figure 3(b) illustrates the process involved in the keyword selection component.
3.3 Vision Token Selection via Dynamic Programming
To tackle the challenge of dynamically varying lengths of vision tokens, we propose a forward- backward algorithm for dynamic vision token selection. Specifically, for aligned vision tokens HV of length M, given text priors T with keyword number difference Ney between instruction and response, we use binary classification head feLS obtain classification results as follows:
= (HL) = JcLs((/ver(MMCAA(( V + (T2.
= h
(3)
The length of the vision concept of the target is linearly estimated based on min-max normalization
Nup Nat
).. where Nm = 10, Aajj = -35, and S is the information domain scalar used for maximum vision token length control. Additionally, when randomly masking keywords in the instructions with a mask ratio sampled from Uniform(O. 1). the estimated vision concept length increases. This indicates that during the optimization process, the unsafe scenario of reducing necessary vision information and introducing additional hallucinations noise does not occur. Therefore, such a random masking strategy not only enhances the learning of vision concept modeling but also provides handle to control the vision token length in inference.
as L = M S (1
N
Nymia
However, since we can only estimate the target length L without knowing the exact positions of these features, there are multiple alignment possibilities. Exhaustively searching for all alignments has time complexity of O(2"). ), which is computationally prohibitive. To address this, we reformulate the problem into dynamic programming process [17] with time complexity of O(M²) as below. Note that we have taken into this procedure may be somewhat complex mathematically. Therefore, also provide an illustration for better understanding (cf., Table 6 of Appendix E).
Extension of target sequence. The target sequence Z' is initialized as a sequence of L symbols, where each symbol represents retained vision concept Zv {z. {2 with 2 E {+), where indicates a retained vision concept. The physical meaning of vision concepts here refers to continuous segment of retained vision tokens. To account for possible alignments, the target sequence z is extended by inserting blank symbols 0 between and around the retained vision concept Z [0, zY, 0, ziz 0 ... zY. 0). The length of the extended sequence Z. is hence 2L + 1.
Forward probability initialization. The forward variable att, represents the probability of aligning the first tokens of the i sequence Y to the first tokens of the extended target sequence zV, where a't, 1) = p(22,
.,x Y ..]). The initialization is as follows. At the first time step t = 1, we set (1, = p00ly)) and a1, = p(*lyY), where p(0lyY) is the probability of aligning the first input token to the first blank symbol in the extended target sequence. For 7 > 1, the forward variable is initialized to (1, 1) = 0 for > 2.
Forward probability transition. The forward probabilities are computed iteratively from t 2 to M a = to 2 using the recurrence (tlation:ax(t, 7) = p(z) (a - 1,1) + a(t - 1, 1 1)), where p(z) \y>' is the probability of aligning the 1-th input token to the I-th token in the extended target sequence, p(z) ly>) a(l - 1, 1) the probability of staying at the current state, and p(>) lx!)-
a(t - 1,7 - 1) is the probability of transitioning from the previous state.
5

---

Backward probability initialization. The backward variable [t. 1) represents the probability of aligning the remaining tokens of the imput sequence (from M) to the remaining tokens of the extended target sequence (from to 2T + 1), where ((t, = p( [ ; ".. 2 221+1 " MM)) The initialization is follows. At the last time step (t M), we set (MM 2L 1) = p(0ly and
2(M = p(*ly)>), where 2/, + 1 is the final blank symbol in the extended target sequence. For <L + 1, the backward variable is initialized to ( = 0 for I 2L.
Backward probability transition. The backward probabilities are computed iteratively from 1= M - to and 2L to using the following recurrence relation: ((1,1) = (8(L + 1,1) + 8(1+ 1(++1,1+1)) p(z) ly where B(L + 1.1) p(z) (yY) represents the probability of staying at the current state, and ((t + +1,++1111 p(z) Y means the probability of transitioning to the next state.
3.4 Objective of VCM and Gradient Computation
The probability of the extended target sequence Zd given the input sequence YV is computed as:
21+1
( = (M,) + (M,, 2L + +) = +) 3(1,2) = Σ alt,l) ((t, E ML.
(4)
The loss is then defined as the negative log- lig-likelihood
LvCm = llo log Y
(5)
The gradient of the loss with respect to logits y (z))) is computed using the posterior probability (t,,),, p(z) y represents the probability of aligning the t-th input token to the I-th token in the extended target sequence 1), = (lt, 3(1,))/(ZZ((Y) The gradient is then given by: aCvey = ((>lly)) (t, (see detailed mathematical derivations in Appendix C), where an (y)
indicator function Hy (p(+lx)) > p(0ly))) is used extract the vision concept H. from HV via segment merging (SM) operation H. SM(HV, YV, Iyv (p(+)y)) p(Oly))).
For better understanding, we provide detailed pseudocodes for the training pipeline of LvCm in Algorithm 1 and for the SM operation in Algorithm 2 respectively (check Appendix E for more details). The weighted average method is used to extract the concept level features H from the features HV of the token level, giving the physical meaning of the importance of the score Y which is more suitable for other scenarios. The next token prediction loss is defined as:
LNTP - log (JIMM((IIM((( H H'))..
The total loss in the instruction fine-tuning stage then defined as
Lkal LNTP + E(r) LvCM:
where c(r) is a coefficient function used to balance two loss functions. Note that when the mask ratio r is small, the VCM loss is not accurate, the coefficient is appropriately small. Besides, when r is large, the VCM loss is more accurate and appropriately increases, using the hyperbolic tangent smoothing function. More mathematical details about e(r) can be found in Appendix B. After vision concept modeling, we can effectively control the model to precisely output vision concepts of different lengths based on the mask ratio
Experiments
4.1 Experimental Setups
Datasets. To justify our claims and demonstrate the superiority of VCM, several representa- tive tasks are involved, which include visual question answering. zero-shot image classification, open-vocabulary object detection, open-vocabulary semantic segmentation, and video understand- ing. Specifically, for visual question answering (VQA) evaluation, we conduct experiments on 11 widely adopted mmsee benchmarks, including VQAV2 (VQA V2) [21], GQA [22], VisWiz [23].

---

Mask
person, yellow
person
Q: where the person yellow? A: the center the image.
Image
w/o VCM
w/ VCM
Figure 4: Visualization results about our VCM. (Top) Visualization of VCM with different instruc- tions. From left to right, the visual representation becomes increasingly sparse, leaving corresponding vision tokens to unmasked keywords (highlighted in red). (Bottom) K-Means visualization of dense feature maps of CLIP ViT. We show the raw images, the K-Means results without VCM, and those of our fine-tuned model by VCM.
SciQA [24], POPE [25], MME [26], MMBench (MMB) [27], SEED-Image (SEED) [28], MMvet [29], TextVQA (VQA") [30], and MMStar [31]. Also, RefCOCO [32] is used for the region-level VQA task. Besides, we employ COCO [33] and OV-COCO [34] for zero-shot image classification and open- vocabulary object detection, and use ADE- 150 [35] and ADE-847 [35] for open- vece-vocaullary semantic segmentation. To test the video understanding capability of VCM, 4 common video question answer- ing benchmarks, TGIF-QA [36], MSVD-QA [37], MSRVTT-QA [37), and ActivityNet-QA [38], included. Here LMMs-Eval toolkit [39] is employed for fair comparison. More details of the tasks and datasets are included in Appendix E.
Models. We follow the architecture from the LLaVA series [20, 1]. where an LVLM consists of three key components: LLM for next token prediction, vision encoder for extracting vision features, and an image-text projector for vision and language aligmment. Besides, we utilize a multi-head self-attention layer for the keyword selector module and multi-head cross-attention layer for vision concept modeling, both employing 16 heads. For LLVVA-15--7113BB LLa [1], and Video-LLaVA [40], we adopt the original paper's publicly available inference setting.
Training strategies. We follow the widely used two-stage setting for training. Specifically, includes lingon-language pre-training and instruction fine-tuning. The language models and ViT separately pre-trained, while the projector is randomly initialized. To initially align the feature space between vision and language modalities, we utilize aligned dataset [20]. Afterward, perform instruction fine-tuning of the pre-trained model on vision-language instruction datasets. The Adam W optimizer [41] is exploited, with learning rates 5 x 5 and 2x 10 for aforementioned two stages respectively. The instruction fine-tuning stage is trained with two epochs with 3% warmup strategy. The experiments are conducted with 8 NVIIIA A100-80G GPUs.
4.2 Qualitative Analysis
As shown in Figure 4(top), we visualize the performance of VCM on visual question answering (VQA) examples. From left right, we demonstrate the results of masking different numbers of keywords in the instruction. As fewer keywords (highlighted in red) are masked, more irrelevant tokens are removed, and the vision concepts are progressively refined. The visualizations indicate that, although VCM discards some overall image details, it effectively retains key vision concepts required for correctly answering the question. More visualization cases can be found in Appendix G.
We further demonstrate the vision concept representation of the vision encoder trained with VCM by performing K-Means clustering [42] on dense feature maps. Specifically, pixels with high cosine similarity grouped into the same cluster. For clarity, clusters containing only small
7

---

Table I: Performance 11 image-basee baage-bbasd VQA benchmarks. '#Vision Tokens' is the number of vision tokens fed to the LLM backbone. **' indicates that unfreezing the image encoder in the instruction fine-tuning stage.
Methons
Vissio
VQA
GQA
SeIQ
VOA
BLIP-2(45)
32
65.0
41.0
19.6
61.
85.3
1293.8
31.2
66.4
22.4
Instretti [3]
32
49.2
36.5
85.1
1391,4
36.0
53.4
35.6
32.7
IDRFICS-P 9m (44)
64
38.4
35.5
53.5
81.9
1177.3
46.2
45.0
30.0
21.6
IEFFIC (44]
64
58.5
52.1
26.1
156
56.3
32.5
Owen
145)
56.2
mMl UI-Owll (46]
84.6
1786.4
57.8
1.
50.0
53.6
30.5
LLaWs-15 (20)
575
77.6
61.7
46.5
06.3
86.0
1793.3
64.1
57.3
30.2
37.8
33.0
58.6
vision token reduction methods
MQT MCTELLaVA [14)
1796.2
[19]
46.7
57.1
31.7
SpameV [15)
57.6
51.6
81.
1721.0
55.8
32.0
FDrop [48]
192
76.1
57.1
51.3
70.2
82.3
1766.0
63.2
54.7
30.5
56.1
32.1
57.8
VisonZip L16]
192
77.4
60.1
31.2
08.2
84.9
1834,0
63.4
59.1
32.6
37.8
32.3
59.1
Ours
15.7
57.2
+2.4
Table 2: Results of region-level VQA and zero-shot image classification tasks. The best result in each case is shown in bold.
Region-level VQA (Re(COCO)
Image Classification (COCO)
Methods
CLIP
VCM
BBox testB
Mask testB
Thing Masks
Stuff Masks
testA
val
testA
val
Top 1
Top 5
Top
Top
LLaVA-v1.5
VIT-B/16
16.6
43.7
16.8
33.6
56.0
25.9
LLLVA-vI5
VIT-B/16
18.9
45.4
35.7
19.1
45.6
35.4
42.7
70.6
39.8
64.6
LLaVA-v1.
VIT-L/14
14.9
40.1
29.5
14.4
29.6
28.3
52.0
11.8
27.9
LLaVA-v1.5
VIT-L/14
16.5
41.3
16.8
41.6
31.9
69.7
25.4
47.0
number of pixels are removed. As shown in Figure 4(bottom), the CLIP VIT trained with VCM exhibits significantly improved performance in grouping pixels belonging to the same object into single cluster, compared to the baseline without VCM. For example, in the visualization, clusters notably more accurate. These K-Means clustering
corresponding to "face", "hat", and "glasses" results provide intuitive evidence for the enhanced vision concept representation capability of CLIP ViT. Additional visualization cases can also be checked in Appendix G.
4.3 Benchmark Comparison
To evaluate the effectiveness of VCM, we conduct experiments on 11 widely used image-based VQA benchmarks and compare our method against existing token reduction state-of-the- -f-t---rrt (SOTA) meth- ods [19, 15, 16, 14, 48]. Then, we explore VCM in enhancing the fine-grained perception capabilities of LVLMs on and region-level VQA and zero-shot image classification tasks. Finally, we validate the broad applicability of VCM in improving other vision-language tasks such open- vocabuocyy object detection and open-vocabulary semantic segmentation. provide additional experiments about acceleration effects of VCM in high-resolution scenarios and video understanding tasks, well as its generalization and scalability across different architectures in Appendix F.
Results of bmsee VQA. As shown in Table 1, we apply VCM to train LLaVA-1.5 and then evaluate achieved performance. When retaining only 144 tokens, our method outperforms PDrop [48], Sparse [15], and VisionZip [16] with fewer vision length by 5.2%, 5.0%, and 2.9%, respectively. Furthermore, VCM outperforms the vanilla model with 144 vision tokens retained. This means that VCM method brings strong vision representation capabilities.
Results region-level VQA and zero-shot classification. We study the performance of the LLaVV model trained with VCM on the region-level VQA task. Specifically, given bounding box mask

---

Table 3: Results of open- open-vocabulary (OV) object detection and open open-vocabulary (OV) semantic
segmentation tasks. The best result in each case is shown in bold.
OV Object Detection
OV Semantic Segmentation
OV-COCO
ADE-150
ADE-847
Method
CLIP
Method
CLIP
AP
AP
AP50
mloU
mAcc
mloU
mAcc
SAN [51]
VIT-B/I6
27.5
45.6
10.1
21.1
SAN [51]
VIT-L/14
32.1
50.7
12.4
25.2
F-VLM [52]
VIT-B/16
16.0
36,9
31.4
Cat-Seg [53]
VIT-B/16
27.2
41.2
8.4
16.6
F-VLM [52]
VIT-1/14
44.3
35.2
Cat-Seg [53]
ViT-I/14
31.5
46.2
20.5
F-VL--VCM V-V
ViT-B/16
20.6
47.1
38.6
Cat-Scg-VCM
VIT-B/16
29.4
45.5
9.1
21.3
VIT-1114
12.6
48.7
39.2
Cat-Seg+VCM
ViT-I/14
34.8
53.1
11.9
23.1
Table 4: Ablation study of the coefficient function e(r), weighted average method in SM operation, semantic alignment, mask strategy, and information domain scalar S on 4 image-based VQA bench- marks. The best result in each is shown in bold. The default setting used in the main experiments marked as underline.
weighted average
semantic alignment
mask strategy
#max tokens
c(r)
S
SciQA
VizWiz
POPE
MME
Avg.
1/4
144
60.7
46.5
73.4
1231.5
56.1
1/4
144
61.7
46.7
75.3
1235.2
57.0
1/4
144
62.4
47.6
75.8
1334.7
58.4
1/4
144
63.1
47.8
76.2
1344.2
58.7
V
1/2
288
62.9
46.9
1369.7
58.9
K
1/4
64.1
48.2
76.6
1350.6
59.3
1/6
72
61.1
46.4
74.0
1234.9
56.4
1/8
36
61.7
46.6
76.1
1244.1
57.2
label, we highlight the corresponding region in the image and send the image to model for VQA related to that region. As shown in Table 2, using VCM significantly enhances the ability of LVLMs to understand vision concepts in specific regions of the image. These results are consistent with the phenomena observed in qualitative analysis. Afterward, for zero-shot image classification the COCO validation set [33], we extract dense feature maps from the CLIP ViT using panoptic masks (thing and stuff) from the COCO Panoptic dataset [49] and mask pooling operations [50]. The corresponding accuracy is reported in Table 2. The results show that VCM significantly improves the classification capabilities of CLIP VIT panoptic masks.
Results of open-vocabulary object detection and semantic segmentation. We follow the setup of F-VLM [52], which freezes the CLIP-ViT backbone and extracts multi-scale feature maps for object detection. As shown in Table 3, replacing the CLIP VIT with the VCM-trained version significantly improves performance on OV-COCO [34] (e.g. 9.2 vs. 12.6 for Aprovel, 44.3 vs. 48.7 for APhase and 35.2 39.2 for APso). These results highlight the effectiveness of VCM in apen-vocabular object detection. Besides, we apply the VCM fine-tuned model to CatSeg [54], where the dense features of CLIP ViT utilized in cost-aggregation module. The segmentation model is trained on the COCO Stuff dataset [53] and evaluated on ADE-150 and ADE-847. As shown in Table 3, VCM consistently improves performance across all test datasets. These results indicate that using VCM not only effectively accelerates the inference of LVLMs but also enhances the dense perception capabilities of the vision encoder.
4.4 Ablation Study
To investigate impact of different components our model design, we conduct series of studies on 4 image-based VQA benchmarks. Note that all experiments are performed with batch size 128. The number of training steps is set to 500 for fast evaluation.
ablation
As shown in Table 4, using the coefficient function can enhance overall model performance, which validates the effectiveness of this design. Then, incorporating weighted averaging method in the concept extraction process, which integrates importance scores into next-token prediction optimiza-

---

tion, further improves overall model performance. Additionally, using semantic alignment training during the pre-training phase can improve overall performance, validating the effectiveness of the keyword selector module design. Furthermore, the introduction of random masking strategy can serve as a data augmentation method, enhancing the effectiveness of VCM.
We also study the influence of the information domain. As demonstrated in Table 4, a larger information domain, which retains more visual information, generally leads to better performance. However, as the information domain size decreases, the shorter sampling length stabilizes training and can offer additional performance gains. Selecting an information domain size of 1/4 achieves favorable trade-off between computational cost and performance, since it maintains competitive accuracy while reducing FLOPs. Based on these findings, we adopt the 1/4 information domain configuration across all experiments.
5 Conclusion
In this paper, we propose vision concept modeling (VCM) that efficiently extracts vision concepts for various iision-language tasks. VCM utilizes large-scale rision-language instruction data with coarse supervision, employs implicit contrastive sampling, and integrates a dynamic concept extraction algorithm optimized through forward-backward framework. Extensive experiments demonstrate that our method reduces computational costs while maintaining strong performance in a series of scenarios. Ablation studies further validate its effectiveness and scalability.
Acknowledgements
Min Yang is supported by Guangdong Basic and Applied Basic Research Foundation (2025B 151000032 and 2024A1515030166) Xiaobo Xia supported by MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China (Grant No. 2421002).
References
[1] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[3] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning. In NeurlPS, 2024.
[4] Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.. 15232, 2024.
[5] Xiaobo Xia and Run Luo. Gui-rl: A generalist r rl-style vision- lssoun-aagguae action model for gui agents. arXiv preprint arXiv:2504. /0458, 2025.
[6] Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on languaee models. In NeurIPS, pages 3122-3156, 2024.
[7] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprins ar0iv:2304. 10592, 2023.
[8] Yuchen Zhou, Jiayu Tang, Shuo Yang. Xiaoyan Xiao, Yugin Dai, Wenhao Yang, Chao Gou, Xiaobo Xia, and Tat-Seng Chua. Logic unseen: Revealing the logical blindspots of vision- language models. arXiv preprint 2xXv:2508.11317 2025.
10

---

[9] Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision- language-action models for embodied ai. arXiv preprint arXiv:2405. 14093, 2024.
[10] Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. cimbspttial-benche Benchmarking spatial understanding for embodied tasks with large vision-language models. arXiv preprini arXiv:2406.05750, 2024.
[11] Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vision-language models and end-to-end autonomous driving. arXiv preprint arXiv:2410.22313, 2024.
[12] Hanlin Tian, Kethan Reddy, Yuxiang Feng, Mohammed Quddus, Yiannis Demiris, and Panagio- tis Angeloudis. Large (vision) language models for autonomous vehicles: Current trends and future directions. Authorea Preprints, 2024.
[13] Yuchen Zhou, Jiayu Tang, Xiaoyan Xiao, Yueyao Lin, Linkai Liu, Zipeng Guo, Hao Fei, Xiaobo Xia, and Chao Gou. Where, what, why: Towards explainable driver attention prediction. In ICCV, 2025.
[14] Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision- languaee models. arXiv preprint arXiv:2405.19311 19315, 2024.
[15] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevim: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024.
[16] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but necessary in vision language models. arXiv preprint arXiv:2412.0448 2024.
[17] Richard Bellman. Dynamic programming. Science, 153( 1966.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[19] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large liniun-language models. ECCV. pages 19-35, 2025.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024.
[21] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904-6913, 2017.
[22] Drew A Hudson and Christopher Manning. Gqa:
new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700-6709 2019.
[23] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608-3617, 2018.
[24] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrIPS, 2022.
[25] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object ballucination in large vision- languaee models. arXiv preprint arXiv:2305.1033. 2023.
11

---

[26] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng. Ke Li. Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[27] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023.
[28] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint MKKi:23077.1122: 2023.
[29] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 308.020900, 2023.
[30] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317-8326, 2019.
[31] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, al. Are the right way for evaluating large vision-language models? arXiv preprint arXiv: 20330, 2024.
[32] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. In EMNLP, pages 787-798, 2014.
[33] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXivv1504.00322. 2015.
[34] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, pages 106-122, 2022.
[35] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[36] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In CVPR, pages 2758-2766. 2017.
[37] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan Hc, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, pages 1645-1653, 2017.
[38] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127-9134, 2019.
[39] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang. Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.
[40] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
[41] I Loshchilov. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
[42] Stuart Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory. 28(2):129-137, 1982.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICMI. pages 19730-19742, 2023.
12

---

[44] IDEFICS. Introducing idefics: An open reproduction of state-of-the-ant visual language model. https:// http:://huggingface. ct/blog/ idefics.. 2023.
[45] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprini arXiv:2311.07522 2023.
[46] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug- Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.022 029999....999.... 2023.
[47] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprini arXiv:2403.15338 2024.
[48] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-languag, models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.1722 2024.
[49] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In CVPR, pages 9404-9413, 2019.
[50] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969, 2017.
[51] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, pages 2945-29222222 2954,2954,00 2023.
[52] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vim: Open- vocabulary object detection upon frozen vision and language models. arXiv preprint 1XXi::2009116339 2022.
[53] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pages 1209-1218, 2018
[54] Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seun- gryong Kim. Cat-seg: Cost aggregation for open-vocabulay semantic segmentation. In CVPR, pages 11133412,, 2024.
[55] Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, al. Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint 0rXiv:2409.05840 2024.
[56] Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504. 10.0004004458 10458, 2025.
[57] Run Luo, Xiaobo Xia, Lu Wang. Longze Chen, Renke Shan, Jing Luo, Min Yang, and Tat- Seng Chua. Next-omni: Towards any-to-any omnimodal foundation models with discrete flow matching. arXiv preprint 2rXh::251..3721,, 2025.
[58] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang. Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision- language model. In ECCV, pages 408 424, 2024.
[59] Xuanlin Li, Yunhao Fang, Minghua Liu, Zhan Ling, Zhuowen Tu, and Hao Su. Distilling large lanion-language model with -iitt--diitiitttion generalizability. In ICCV, pages 2492-2503. 2023.
[60] Zilun Zhang, Tiancheng Zhao, Yulong Guo, and Jianwei Yin. Rs5m and georselip: A large scale vision-language dataset and large vision-language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[61] Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in-context learning for large vision-language models. arXiv preprint 15774 2024.
13

---

[62] Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anoma- lygpt: Detecting industrial anomalies using large vision-lang Laiiugg models. In AAAI, pages 1932-1940, 2024.
[63] Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large vision-language models. In Neur/PS, pages 50168-50188, 2024.
[64] Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang. Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchical feature alignment for vision- vision-language model pretraining. In NeurIPS, pages 35959-35970, 2022.
[65] Mu Cai, Jianwei Yang. Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In ICLR, 2025.
[66] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[67] Runpei Dong. Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang. Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.1 /199,, 2023.
[68] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-langura models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[69] Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. Benchmark evalua- tions, applications, and challenges of large vision language models: A survey, arXiv preprint arXiv:2501.00 1819,, 2025.
[70] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. arXiv preprint vrXiv22202.10938 2022.
[71] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models, arXiv preprint wxXi:2442..0025. 2024.
[72] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:220..07800, 2022.
[73] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
[74] Shuai Li, Jian Xu, Xiao-Hui Li, Chao Deng, and Lin-Lin Huang. Qg-vtc: Question-guided visual token compression in milms for efficient vqa. arXiv preprint arXiv:2504.006544 2025.
[75] Zhihang Liu, Chen- Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, and Hongtao Xie. Hybrid-level instruction injection for video token compression in multi-modal large language models. arXiv preprint arXiv:2503. 16036 2025.
[76] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.00 1.3895, 2025.
[77] Jieneng Chen, Luoxin Ye, Ju He. Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi-modal models via visual context compression. In NeurlPS, 2024.
[78] Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vasion-language models. arXiv preprins arXiv:2405. 19315, 2024.
[79] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433 2015.
14

---

[80] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608-3617, 2018.
[81] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pages
2015.
[82] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.
15

---

A Related Work
A.1 Large Vision-Language Models (LVLMS)
Benefiting from the success of large language models (LLMs) and the availability of diverse image- text instruction data [20, 55, 56, 57J, LVLMs have made significant strides in recent years [58, 59, 60, 61, 62, 63, 64, 65]. For example, LLaVA [20] and MiniGPT-4 [66] have demonstrated strong cross-task generalization by integrating vision encoders with LLMs through simple connectors and training on instruction data. LLaVA-NeXT [1] has significantly enhanced visual perception by employing dynamic resolution techniques. Dream [67] attempts to generate images and text in the interleaved context concurrently. DEEM [4] simplifies model architecture and enhances robustness by using diffusion models to extract vision features instead of traditional vision encoders. Readers can refer to [68, 69, 70, 71] for more details and recent advances in LVLMS.
A.2 Vision Token Reduction for LVLMs
Vision token reduction a key technique for improving the efficiency of LVLMs. Previous
works, egg., EVTT [72] and ToMe [73], lower computational costs by pruning less important tokens| [74, 75, 76]. Differently, methods such
(A)
Merge
Merge
Pruse
as LLa Volta [77]. QWen-VL [2]. and MQT-
Attrndio Matro
Atizztinn Mersmbe
Alteelie Nedssnkn
LLaVA [78], use clustering or Q-former to com- press tokens into fixed-length representations, but these token merging methods fail to retain the relative order and positional relationships of the original tokens. Recent strategies like FastV [19], Sparse VLM [15], PyramidDrop [48], and Vi- sionZip [16] leverage language model signals and fixed pruning ratios to enhance efficiency. However, it is challenging for them to control the number of pruned vision tokens. Moreover, these methods often disrupt vision concept modeling due over-compression, resulting in noise and misalignment between vision and semantic con- cepts, which restricts the model's applicability.
Figure 5: Comparison of three different vi- sion token reduction paradigms. (a) Regu- training-free token pruning method based on threshold filtering of the attention matrix. (b) Regular token merging method based on the atten- tion mechanism and fixed-length trainable query vectors. (c) Our method, through adaptive local merging and filtering modeled by VCM, allows precise control of the vision concept length while keeping the positional information.
To overcome these concerns, we propose VCM, which uses rigorous theory-driven optimization to enable dynamic vision token selection and vision concept modeling. Our VCM not only allows precise control of the vision concept length, but also preserves the relative order and positional relationships of original vision tokens. As illustrated in Figure 5, compared vision token compression paradigms (a) and (b), our method (c) achieves better results with shorter and more controllable vision concept length, without losing position information or vision concept. VCM therefore can improve the efficiency of LVLMs and broaden their applicability to various lision-language tasks.
B Mathematical Expression of E(r)
The coefficient function E(r) can be expressed mathematically follows.
Step 1. Compute a smooth S-shaped function using the hyperbolic tangent (tanh) function: + tanh (k- (2 r 1))
(6)
2 where k is a parameter that controls the growth rate. Larger values of A result in steeper transitions, while smaller values result in smoother transitions.
Step 2. Scale and shift s to the desired range (a, b):
<(r)
+ (b a)
(7)
16

---

where a = 0.2, b = 1.2, and k in our default setting.
C Proof of Gradient Computation about LVCM
Proof. We provide the proof of the gradient of LvcM with respect to yo (z)) step by step.
1 (Chain rule). By definition,
Lvem = log Y))
SO
0LvcM θy (z)
0p(Z)
P(Z By)
2 (Marginal probability and alignment decomposition). By the forward-backward algorithm,
2L+1
p(z' = Σ
((t,))))(t
Only the terms with /' = / depend on yÝ (z,") ) thus
0p(Zy YV) 1a((,1) oy: Ty (x)) ((1.1) a(l,1)
03(1, 1)
Qy: (z
3 (Forward/backward variable dependence). From the recursive definitions, both a(l, and
(1,, 1) are linear p(z) ly)), which is the softmax over logits yÝ. Thus, for fixed (1, I).
0p(Z V YV) = ((t, ly.)
0p(z)
(Softmax chain rule). By the softmax derivative,
Öp(z) y σy (z))
- = -6
where
11, ifk=1
81
0, il
5
(Final assembly, detailed).
By the chain rule, the gradient of the loss with respect to the logits is
@LveM dy
24+1
QLveM
Ap(z)ly7) 0y (2,)
E
Jp(z)lxP)
where
0LvcM 0p(z)ly.)
alt, 3),, k) p(Z 1x
YV
0p(z)
Substituting the softmax derivative, we have
8CvCM = (x) 1=1
26+1
Σ
YV) p(ZN)T)) p(ZMlx\) - p(zNlx)1))
att, ((t,k)
25++1
a(t,k)
(BRI p(z)lx²\))
PV)
17

---

Now separate the k = 1 term and the is + terms:
a(t,t))(tt
a(t,k)s(t,k)
p(Z Y (1 -i>pi(li)))
p(ZNlY)) p(R
alt,1)
21+ ((,
p(Z YY)
p(Z
Since EREt
2L+1 ((t,)) P(2
1, this simplifies to
0LvCM = p(2)ly8)) ((1,
(z)
Posterior definition: Define the posterior probability
(1,,))
r(t,
Therefore, the final result is
QLvcM
p(z)lx))) - (t,,1) (z)
This concludes the proof.
D VCM Efficiency Analysis Based on Floating-Point Operations (FLOPs)
Evaluating the computational complexity of LVL.Ms requires examining key components such as the self-attention mechanism and the feed-forward network (FFN). Specifically, FLOOPs can be expressed as
FLOPs = T x (4nd² + d 2ndm),
(8)
where T is the number of transformer layers, n is the sequence length, d is the hidden dimension size, and ma represents the intermediate size of the FFN. During training, we can estimate E/n] R d/4 and m R 4d. Therefore, we can approximate that FLOPS ~ T x (12nd + 2n²d). We can see that the computational complexity is strongly influenced by the sequence length n. In typical vision-language tasks, the sequence length is defined as
n = nlys + Rmmg + Tins + Tress
where Pays. Fiing- Rins, and frees denote the lengths of the system prompt, vision tokens, instruction, and response, respectively. Note that Ring is often much larger than the other three, sometimes by a factor of 20 [16, 19] as Ramg 2 20 x (Nays + Nim + neee). Therefore, reducing Bing is essential for improving the efficiency of LVLMs. Here, we analyze the expectations of FLOPs under two scenarios. (1) The original case where n is unchanged. (2) The case where n is scaled by 1/8 by
VCM with information domain S 1/4. Let the variance of be of
Case (1). Based on the above analysis, we first have E(n] = d/4 and E(n?] = 02 +dd/16. Afterward, we can derive
En = T (12df E[n] + 2d E[n²]) = T x (25d² /8 2do
(10)
Case (2), When is scaled by 1/8, the new expectation is
E₂ = T x (12d2 E[n') + 2d E(n
(11)
where E[n'] = d/32 and E[n = 0, /64 + d' 11044. Substituting these values gives
Eg T (3d /8 + do + d8 /512).
(12)
Ratio about expectations of FLOPs. The ratio of the scaled expectation (case (2)) to the original expectation (case (1)) can be approximated by discarding negligible terms as follows:
Eg
T x (3d' /8 + do2 + d3 /512) 3d /8
3
R
(13)
E(
T x (25d'/ / + 2do
25d3/8
25'
This shows that VCM can significantly reduce computational costs, egg. achieving an 85% reduction with respect to FLOPs for LLaVA-1.5-7B.
18

---

Table 5: Overall descriptions of the evaluation benchmarks for visual question answering (VQA).
zero-shot image classification (ZERO.), open-vocabulary object detection (OV-OD) -peablocabulary
semantic segmentation (OV-SS), and video understanding (VIDEO.).
Tasks
Datasets
Descriptions
Eval Splits
Metries
VQA [21]
Scene understanding QA
test-dev
VQA Acc (*) [79]
GQA [22]
Scene understanding QA
test-dev
VQA Acc (*) [79]
VizWiz [80]
Scene understanding
test-dev
VQA Acc [79]
SciQA [24]
External knowledge QA
val val
VQA Acc [7)
POPE [25]
Visual ballucination
Acc (?)
VQA
MME [26]
External knowledge
val
VQA Acc (*) [79]
MMB [27]
Visual comprehension
dev-en
Acc (1)
SEED [28]
Visual comprehension
val
Ace (t)
MM [29]
Visual comprehension
val
Acc (t)
VQA [30]
Text understanding QA
val
VQA Ace (*) [79]
MMstar [3]]
Knowledge leakage QA
val
VQA Acc (*) [79]
RefCOCO [32]
Region-level VQA
val, teath, teat
CIDEr (1) [8]]
Zero Zero-Shot.
COCO Panoptic [49]
image classification
val
Ace (t)
OV-OD
ov-COCO [34]
Open-vocabulary object detection
val
AP ,P35 (PPs. [34]
OV-SS
ADE20K [35]
Open vocahulary semantic segmentation
val
mloli, mAce (1) [82]
TGIF-QA [36]
Video understanding
test
Acc (1)
VIDEO.
MSVD- [37]
Video understanding
test
(f)
MSRVTT-QA [37]
Video understanding
test
Acc (1)
Aetivity QA [38]
Video understanding
test
Acc (?)
E
Additional Implementation Details
More details of evaluation. In this paper, we demonstrate the superiority of VCM on several tasks, including visual question answering, zero-shot image classification, open-vocabulary object detection, open-vocabulary semantic segmentation, and video understanding. All these evaluation tasks and metries are listed in Table
Table 6: byde-b display of the original probabilities, table, and y table (rounded three decimal places) in dynamic programming.
Y
P(U)
P(*)
[
1
==
2
[
I
1= 5
1
[=
2
1=
3
[
I == 5
total
0,800
0.200
0.800
0.200
0
0
0
1,818
0.19
0
0
0
2.014
0.600
0.400
0.480
0.400
0.120
1.163
0.819
0.032
2.014
3
0,200
0.800
0.096
0.704
0.104
0.096
0
0.248
1.677
0.074
0.015
0
2.014
0,300
0.700
0.029
0.560
0.242
0.140
0.029
0.075
1.447
0.464
0.027
0.002
2.014
0.700
0.300
0.020
0.177
0.562
0.115
0.118
0.052
0.458
1.449
0.042
0.014
2.014
0,900
0.100
0.018
0.020
0.664
0.068
0.210
0.047
0.051
1.723
0.167
0.027
2.014
0.100
0.900
0.002
0.034
0.068
0.659
0.028
0.005
0.088
0.177
1.708
0.036
2.014
0,300
0.700
0.001
0.025
0.031
0.509
0.206
0.002
0.046
0.133
0.943
0.890
2.014
More details of VCM. To further elaborate on the details of our method, we provide a comprehensive overview of the process described in Section along with the complete pseudocode for the training pipeline in Algorithm 1. The VCM loss is derived through vision concept length estimation, objective expansion, forward and backward initialization, and state transitions. Gradients are then computed to enable optimization.
Furthermore, further clarify the dynamic programming optimization process of VCM, we present the complete procedure of computation, optimization, and inference for VCM when the input vision sequence length and the output vision concept length is As shown in Table 6, the forward probabilities are strictly optimized according to the recursive formulas described in the main text. Based on these, we compute the & table, where the sum of each row remains constant, representing the total probability of all reasonable alignment paths discussed in the main text. Moreover, we provide corresponding visualized case in Figure 6 to help readers better understand the training and inference processes. During training, maximize all feasible alignment paths (blue) and minimize
19

---

impossible path
possible path
most possible path
V3
V4
y
h2
hg
1
h44
h2
h4
hg
hg
hy
hg
3/
3'
y/
y
y/
y
y's
weighted average
hy
h2
0
h
h2
where is the red bird
Large Language Model
the right the image.
Figure 6: An example of the workflow in VCM. Specificcl, the VCM loss computes all possible alignment paths with the extended target sequence, maximizing the probabilities of feasible paths (blue paths) while minimizing the probabilities of infeasible paths (yellow paths), which ensures that the output concept length meets expectations. During training and inference, the most probable path (red path) is selected, and a segment merging (SM) operation is performed, where adjacent retained vision features are weighted and averaged based on their probability scores. The extracted vision concept features are then combined with the instruction and fed into the LLM generate the corresponding answer.
all infeasible alignment paths (yellow). During inference, directly select the most probable path red among all possible alignment paths as the optimal path.
More details of SM. To extract vision concepts from the most probable path, employ the segment merging (SM) operation. However, since the length and number of vision concepts vary samples, using double for-loop for feature extraction introduces significant computational bottleneck, severely reducing training efficiency. This limitation is one of the primary reasons previous methods [19, 15, 16, 14] avoid using variable-length features during training.
To address this issue, we implement a parallelized SM operation capable of simultancously handling multi-end features of varying lengths and positions. The PyTorch implementation of this operation provided in Algorithm Compared to the double for-loop approach, our method achieves 100x speedup, making the SM operation computationally efficient and effectively imperceptible.
F Supplementary Experiments Results
Results on VQA for high-resolution scenarios. To further demonstrate the generalization capabili- ties of VCM, we apply LLaVA-NeXT [1], which supports high-resolution scenarios. Compared to LL., LLaVA-NeXX divides the image into four parts, resizes the original image, and converts it into five independent images. The vision encoder processes each image to extract vision tokens, which are then merged. While this method improves model performance, it significantly increases the number of vision tokens. As shown in Table 7, our proposed VCM consistently main- tains strong performance under high-resolution settings. Specifically, when reducing the number of vision tokens to only almost 5% of the original one (160 vs. 2880), our method achieves more competitive performance than SparseVLM PDrop, and VisionZip. Furthermore, VCM demonstrates
20

---

Table 7: Performance on 6 image-basee based VQA benchmarks under high resolution conditions. The best result in each case is shown in bold.
#Vision Tokens
Methods
VQA
GQA
SciQA
POPE
MME
MMB
Avg. (%)
LLaVA-NEXT [1]
2880
80.1
64.2
70.2
86.5
1842.1
67.9
72.4
Sparse VLM [15]
160
66.3
51.2
67.5
74.8
1542.1
63.1
63.0
PDrop [48]
160
69.7
53.9
67.8
83.1
1587.4
64.8
65.9
VisionZip [16]
160
71.4
55.5
68.3
83.4
1630.2
60.1
66.2
VCM
160
76.8
60.1
69.6
87.1
1723.4
65.5
70.1
Table 8: Performance on 4 video understanding benchmarks. The best result in each case is shown in bold.
#Vision Tokens
Methods
TGIE-QA MSVD-QA MSRVTT-QA ActivityNet-QA
Avg. (%)
Video-LLaVA [40]
2048
47.1
69.8
56.7
54.2
FastV [19]
198
23.1
38.0
19.3
30.6
27.8
Sparse [15]
198
44.7
68.2
31.0
42,6
46.6
VisionZip [16]
136
42.4
63.5
52.1
43.0
50.3
VCM
136
45.4
66.8
54.9
42.7
52.5
outstanding performance on visual hallucination tasks (e.g., POPE), validating its effectiveness in concept extraction for high-resolution images.
Results video understanding. To further demonstrate the effectiveness of our method, validate the acceleration performance of multi-frame inputs in video understanding scenarios on Video-LLaVa [40]. As shown in Table 8. compared with other token reduction methods [48, 15, 16], our method achieves better performance at the same or smaller vision token number, which verifies that our VCM exhibits stronger vision representation capabilities.
Generality and scalability of VCM. To explore whether VCM can achieve further performance gains through increased data and model scale, we conduct experiments using a larger dataset. We increase training steps and use larger language model (Vicuna-1 13B) under the default configuration. As shown Table increasing the amount of instruction- tuninn data and scaling up the language model both improve performance, which demonstrates the strong scalability of our VCM.
In addition, we conduct experiments the QwenVL architecture verify the architectural general- ization ability of VCM. Specifically, to ensure fairness, we use fully open-source LLaVA VA data [20] and load the official weights of Qwen2VL. [2] for pre-training and instruction fine-tuning. As shown in Table 9, under the same number of training steps, our method significantly reduces the vision token number and accelerates inference while slightly sacrificing performance. This demonstrates that our VCM has strong generalization ability across different architectures.
FLOP comparison. To better demonstrate the speed improvement brought by our method, we compare the FLOPs under different configurations with other methods [19, 48, 15], as shown in Table 10. It can observed that under various configurations, have reached the state-of-the-arr levels while achieving superior performance. This is attributed to the parallelized optimization of the SM operation and the powerful representation capabilities vision concept modeling, which highlight the superiority of VCM.
FLOPs and latency of our method
21

---

Table Results of VCM generality and scalability on image-based VQA benchmarks. The best
result in each case is shown in bold.
#Vision Tokens
Methods
LLMs
Training Steps
SciQA
POPE
VQA
GQA
Avg. (%)
Owen2-VL Owen2-V1.+VCM
Qwen2-7b
500
1326
73.5
80.2
69.8
57.8
70.3
Qwen2-7h
500
576
74.1
79.2
68.3
56.6
69.6
LLaVA-vI.5+VCM
Vicuna-T7 7b
500
144
64.1
76.6
56.8
43.6
60.3
LLVVA-V1.+VVCM
Vicuna- 7b
1000
144
65,4
78.7
59.3
46.1
62.4
LLaVA-vI.++VCM
Vicuna- 13b
1000
144
67.8
80.3
62.3
49.5
64.9
Table 10: Performance of VCM under different vision token configurations. In addition to the theoretical analysis of FLOPs, we also provide a comparison of performance, FLOPs, and latency under different vision token lengths in practical scenarios.
Methods
VVision Tokens
FLOPs
Latency (ms)
GQA MMB MME POPE SciQA SEED VQA"
MMVet
Avg. (%)
LLaVA-v1.5 [20]
576
4.62
$7.82
62.0
1862.1
85.9
66.8
58.6
58.2
30.5
FastV [19]
49.6
53.4
68.6
48.1
26.3
50.7
PDrop [48]
128
1.62
37.77
56.0
61.1
1664.2
82.3
69.9
53.3
55.1
30.8
58.5
VLM VLM [15]
128
1.72
33.28
58.4
64.5
1746.1
85.0
68.6
58.2
56.7
29.0
60.3
VCM
128
L.71
31.42
61.2
63.7
1789.2
86.1
68.5
62.7
56.2
33.4
62.0
PastV [19]
64
1.29
27.30
46.1
47.2
1255.4
38,2
68.7
43.7
47.8
19.6
44.5
PDrop [48]
1.18
43.41
41.9
33.3
1092.3
55.9
69.2
40.0
45.9
30.7
44.5
VT [15]
1,30
29.89
53.8
60.1
1589.2
77.5
69.8
52.2
53.4
24.9
56.1
VCM
G More Visualization Results
To further demonstrate the effectiveness of our method, we provide additional visualization results. Specifically, they include vision concept extraction in Figure 7, vision concept enhancement of the vision encoder in Figure 8, open-vocabulary object detection in Figure 9, and open-vocabulary semantic segmentation in Figure 10. Overall, these visualization results provide strong evidence for the superiority of our VCM.
H Broader Impacts
The proposed VCM framework advances the efficiency and applicability of LVLMs. It obviously reduces computational costs while enhancing semantic alignment between vision and text, enabling scalable deployment in swsuuccecoonstranne environments such edge devices and mobile platforms. Furthermore, VCM's fine-grae nne-grained concept extraction has the potential to improve performance in critical applications like autonomous driving and medical imaging. where efficient and precise visual understanding is vital. Nevertheless, its deployment must account for possible biases inherited from pre-trained models. Ensuring responsible AI usage remains key consideration for broader societal impact. Overall, VCM represents a step forward in sustainable and accessible vision-language intelligence, paving the way for efficient multimodal learning across diverse real-world scenarios.
I Limitations
In our experiments, we primarily rely on an adaptive keyword selection strategy. However, this way may introduce certain biases, as the selected keywords may not always represent true keywords. Additionally, to simplify modeling, we use min-max normalization for estimated vision concept length. This coarse-grained design may negatively impact the model's performance. In the future, we aim to employ open-source language models perform more refined keyword selection for instruction data. By providing finer-grained additional information, we hope to further improve the performance of the proposed method.
22

---

Mask
year; clock
year
Q: what is the year above the clock?
A: 1856.
jacket, person in the center
Mask
jacket
Q: what color is the jacket of the person in the center? black.
Mask
bus, orange number
orange number
What orange number the bus? 96.
Figure 7: More visualization results of VCM on different instructions. From left to right, the visual representation becomes increasingly sparse, leaving corresponding vision tokens unmasked keywords (highlighted in red). VCM can capture the corresponding vision concepts relatively accu- rately by masking different numbers of keywords across different instructions, which demonstrates its robustness and effectiveness.
Image
w/o VCM
w/ VCM
Figure 8: More K-Means visualization of the dense feature maps of CLIP VIT trained with VCM. Specifically, we show the raw images, the K-Means results without VCM, and those of our fine-tuned model by VCM. As we can see, VCM has significant conceptual enhancements in multiple scenarios, such as animals, food, humans, and near/far perspectives, proving the generalizability and potential of our method.
23

---

Algorithm 1 Pseudocode for VCM Loss and Gradient Computation
Input:
Logit sequence Y = [yY y2 ....... .Y where M is the input sequence length. Information domain scalar S E (0, 1/2] for length adjustment. Mask ratio ~ Uniform(O, 1). Keyword count prior Niex.
Output:
VCM loss Lvem and gradients
@Cvcy dy
Initialization:
Set L = [M S- Norm( (orm()Njea)) as the length of the target sequence.
Sample target sequence x with L symbols:
zv [zy 2g
EYE E (*}
Here, x denotes retaining the token, and denotes blank. Interleave between and around the symbols:
z' 10. z)
zY. 0)
The resulting length of ZV is 2L
Initialize all forward variables ax(1, and backward variables 8(t, 1) to 0.
Set (1, 1) p00ly a(,2) = -
B(M, 2L + 1) = p(0 y and ((M,2L) p()
Forward Pass: for t = 2 to M do
for 1 to 2L do
(t,
lxY) (a(t - 1,1) + a(t - 1))
end for end for Backward Pass:
for - M 1 to 1 do
for 1 to 2L + do
8(t, (B(t + 1.0) + 1,1+ 1)) p(zily))
end for end for Compute Loss:
p = EZL+1 a(M, 3(M,1)
LvCM = - log p(z'
Compute Gradients:
for ( t 1 to M do for I to 2L do
Compute posterior probability:
(t, a(t,1) B(t,t)
p(Z
Compute the gradient for yY (z)): ):
dLvCM p(zz) - xtt,
QyY (z))
end for end for Optimization:
Update y. using gradient descent:
θLveM
yY(a)) yY(2)) 11 (x))
where 7 is the learning rate.
24

---

Algorithm 2 Pseudocode of Merging Segments with Scores (100x faster than a double loop)
def tergee_ageentts_ttt_ooooooe
x_mask, scores):
Merge adjacent features with mask=1 using score-weighted averaging, discarding features with mask=0.
Args:
x (torch. Tensor): Input features, shape [B, S, D] x_mask (torch. : Binary mask, shape [B, s].
scores (torch.eensor) Scores for each feature, shape [B, S].
Returns:
new_x (list [torch. Tensor]) Merged features, each element shape [N,
D]
B, S, = sizize
# Mask features and scores
masked_x x x_mask. unsqueeze masked_scores scores x_mask
# Extend mask and scores for cumulative sums expanded_x_mant torch cat (ax_[aakk, torch. 1), dtype=x_mask, dtype, device=x_mask. (aaaskdevvce)]_.. dim=1)
expanded_scorest torch. cat ([scores, torch. (BB 1), dtyperscores. dtype, device=scores.devics device)), dim=1) # Compute cumulative sums
cumsum_mask torch. cumsum(expander (wxpanded_x_ppeeeeeee x_mask, dim=1)
cumsum_x torch. cumsum (masked_x scores. unsqueeze(-1) dim=1)
Weighted sum of features
cumsun_scores torch. cumsum (nassed_sssdddssres, dim=1) # Cumulative sum scores
# Identify segment boundaries segment_mask mask = x_mask * ((cumsun_mask[: [:, 1:] - cumsum_mask :-1]) == 0)
new 0]
for i in range(B):
# Extract cumulative sums and segment masks for the current sample cumsum_x_curree cumsum_x[i]
cumsum_scores_cee
cumsum_scores [i]
Sumsum_mask_curre cumsum_mask [i] [:-1]
segment__nasx_curreen segment_maskkii..oo bool)))
# Get segment-wise cumulative sums
segment_sums mnanuu_ [segment satintt_msskkeoe current, :] # [N, D]
segment_score__ sums = cwwsum_scores_ceemmmmo ceroorresccurnnttee (wwgmeeeteetttttnnnaanamn htt___msssu_rrmaar current] # [N]
# Extend cumulative sums for segment-wise computation gopanddsegment_ste
torch cat [torch. ((1, D), dtype=
segment_sume dtype, devicewsegment sameeetttsuss_oee device), (igment_sums]
sxpard_segment_scomm wwwrs_sooruuueeeeee sumssssss torch. cat (aoorc zeros (1,), dtype= segment_score_sums.co dtype, swrricessommen_sccrreeeoo
device),
segment__ccre_sunne
diff _umf_ frrranddsegmet... sumb [1:, :]
Begandd_egment_smme
[N. D]
liff_score_sums = expand_segmen wegment_seosrrreeeeeeeeeeeeeesse sums [1:]
# [N]
# Prevent division by zero
diff ueff_score_ett sums torch. whereddiff_scoreee
0, torch. ones mes_like
diff_score_sums:
# Compute score-weighted mean
weighted__een__ogmmeee diff_sums / diff_score_stt stt.eee_ssuuuuuuuueeeeeeee [N, D]
unsqueeze (-1)
append segnnnssgmntts)
return new_x
25

---

IAMOEE
Figure 9: Visualization of open-vocabulary object detection results. The red boxes are for novel categories. The blue boxes are for base categories.
Figure 10: Visualization of open-vocabulary semantic segmentation. The images are sampled from the COCO val set [33].
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the contributions and scope in both the abstract and introduction, supported by series of justifications.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
27

---

Justification:
The proof of gradient computation about the loss LvCm is provided in Ap-
pendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide extensive experimental details the main content (see Section 4) and appendix (see Appendix E and Appendix F).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the full instructions to access publicly available data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//iiss. wttpccpppppc/eeeeeeee..ecc.ccctt gguides/CodeSeiee iobbmmmssiinis sssssnoliccyyii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details in Section and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Following previous works, trial was performed in each experimental case due to time consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide such implementation details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attpss//neuut http///eeerpppeeeeee/eeeeee httppubbii//hhhhhhcuccceeeeeee
Answer: [Yes]
Justification:
We adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide the broader impact statement in Appendix H.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper involves only publicly available datasets and models. No specific safeguards were necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the relevant papers and provide links to existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets. Therefore, no documentation is applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects. Therefore, the inclusion of participant instructions, screenshots, and compensation details is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
32

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does not involve LLMs for any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standary components.
Please refer to LLM policy (https: /nnurripe..p///// cccConnferences//22222222 /LLM) for what should or should not be described.
33


Reviews:
[{"review_id": "cSPN2kXCF6", "reviewer": "Reviewer_DmHA", "summary": "The paper introduces Visual Concept Model (VCM), a new type of large vision language models (LVLMs) where visual embeddings focus on semantic concepts, forming a shorter sequence. The training procedure of VCM follows the standard pre-training and instruction tuning pipeline. In the pre-training stage, the authors propose to train an additional keyword selection module via a semantic alignment loss, and a multi-head cross-attention layer to enhance visual representation. During instruction tuning, the keyword selection module identifies keywords in the text, which are then used to compute the proposed VCM loss. Through experiments, the authors reveal a negative correlation between the required visual token length and the keyword number difference between text instructions and responses. This relationship helps estimate the sequence length of the visual concept embeddings. Finally, vision concept embeddings are selected based on a learned assignment probability, optimized via the VCM loss with the help of dynamic programming. Extensive experiments across diverse benchmarks and tasks are conducted to validate VCM’s effectiveness.", "questions": "The reviewer is uncertain about the following details:\n1. What is the objective of the binary classification in equation (3)? How to train the linear $f_{CLS}$?\n2. How to obtain each probability $p(z_l|y_t)$ when conducting forward and backward probability transition in section 3.3?\n3. Please explain the implicit contrastive sampling module and the corresponding training process. These content seem missing in the main text and only appear in the caption for Figure 3.\n4. How does the proposed method address the first key challenge mentioned in line 73?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is well-motivated and novel. Representing the visual inputs by a shorter yet more semantically meaningful visual embedding sequence can address the sequence length limitation of current LVLMs and narrow their gap to human cognition. The paper introduces several novel modules and optimization objectives to obtain the visual concept embeddings that potentially shed lights to developing new LVLMs.\n2. The analytical experiment that reveals the correlation between keyword number and minimum visual sequence length is interesting.\n3. The evaluation is comprehensive with promising results.\n\nWeaknesses:\nWhile the proposed method is novel, its presentation could be enhanced by simplifying notations and incorporating additional details for clarity. Current presentation increases the difficulty for fully understand the paper. The reviewer may consider lower the score if the following concerns are not well addressed in the rebuttal.\n1. The notations are complicated:\n - The keyword selection module $f_{KS}$ and  'MHSA' in line 102 and equation (2) appears to be the same thing. Consider unifying these notations.\n - Similarly, consider simplifying $f_{V2T}$ in section 3 to $f_M$.\n - The text prior $T$ in line 121 is undefined, which the reviewer assumes to be the keywords.\n - Typo in line 121.\n - Consider using a notation like $f_{T}$ to represent the LLM's word embeddings (by regarding them as an embedding function) instead of $X^T$ to align with other notations.\n2. Several components are left unexplained. Please refer to the questions.", "was_revised": true, "final_justification": "The authors provide detailed explanation to the ambiguities in the original text, which had addressed my concerns. I am willing to maintain my recommendation for acceptance because of the novelty of the method, and raise my score for the better clarity. The authors should refine the manuscript as promised after the rebuttal.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1**: The notations are complicated.\n\n**Response:**\n\nYour careful attention to the consistency and clarity of the notation has been invaluable in helping us improve the quality and readability of our work. We fully agree with all your suggestions and are grateful that you took the time to identify these important improvements. \n\nIn the revised version, we will: (1) unify $f_{KS}$ and 'MHSA' in line 102 and Equation (2) to use $f_{KS}$ consistently; (2) simplify $f_{V2T}$ to $f_M$ in Section 3; (3) add a clear definition for the text prior $T$ in line 121 (confirming it refers to keywords); (4) fix the typo in line 121; and (5) use $f_T$ instead of $X^{\\top}$ to represent the LLM's word embeddings for better notational consistency.\n\nWe are committed to implementing all these changes carefully. Thank you for your constructive feedback.\n\n> **Q1**: What is the objective of the binary classification in equation (3)? How to train the linear $f_{CLS}$?\n\n**Response:**\n\nThank you for your comments. In Section 3.3, we have provided a detailed explanation of the training process. Here, we would like to further elaborate on the specific steps involved.\n\nThe binary classification in Equation (3) generates binary logits $Y^V$, which indicate whether each visual token should be retained. These logits are combined with the target sequence $Z^V$—obtained by expanding the token length using the prior distribution patterns illustrated in Figure 2—and subjected to optimization through the forward-backward algorithm. During this end-to-end training process, the linear classifier $f_{CLS}$ is trained to effectively learn the mapping between visual tokens and their likelihood of being selected.\n\n> **Q2**: How to obtain each probability $p(z_l|y_t)$ when conducting forward and backward probability transition in section 3.3?\n\n**Response:**\n\nThank you for the feedback. Here, we provide further clarification.\n\nFirstly, the logits $Y^V = \\[ y_t^V \\]^M_{t=1}$ are produced as outputs from the linear classifier $f_{CLS}$. Then, the probabilities $p(z_l^V | y_t^V)$ are computed using $p(z_l^V | y_t^V) = \\text{Softmax}(Y^V)$. Essentially, this represents a binary classification probability since $z_l^V \\in \\[\\star, \\emptyset\\]$. \n\nWe will incorporate this explanation into the paper to make the presentation clearer and more accessible. Thank you once again for your insightful comments, which have been instrumental in enhancing the clarity of our work.\n\n> **Q3**: Please explain the implicit contrastive sampling module and the corresponding training process. These content seem missing in the main text and only appear in the caption for Figure 3.\n\n**Response:**\n\nAs discussed in lines 125–130 of the main text, this module fundamentally serves as a data augmentation mechanism and does not contain trainable parameters. Specifically, we randomly apply masking to the extracted keywords, and VCM leverages the masked data to generate extended visual concepts. This ensures that the subsequent LLM processing does not encounter hallucinations or conflicts due to missing information.\n\nIn Table 4, we conducted ablation experiments to validate this masking strategy. The results demonstrate that this approach introduces greater data diversity without adding additional risks, significantly enhancing the effectiveness of VCM’s training and improving its final performance. \n\nAdditionally, we will include references to  Figure 3 in the revised version to ensure smoother readability and make the explanation more intuitive for readers. Thanks for your valuable insights, which have helped us refine and improve the clarity of our work.\n\n> **Q4**:How does the proposed method address the first key challenge mentioned in line 73?\n\n**Response:**\n\nWe sincerely appreciate the reviewers' valuable feedback. Here, we provide further clarification on this point.\n\nBy utilizing VCM's forward-backward optimization algorithm, we can efficiently perform self-supervised learning on large-scale VQA datasets that lack fine-grained annotations. This process allows us to construct a robust visual concept model. Furthermore, we employ the implicit contrastive sampling module as a data augmentation technique, which further increases the diversity of the training data and enhances the effectiveness of visual concept modeling.\n\nWith these two contributions, we address the first key challenge mentioned in line 73. Unlike previous methods, which require fine-grained annotations—not only identifying the keywords in the text but also aligning each keyword with specific regions in the image-our approach eliminates the dependency on such detailed labels, making it both more scalable and efficient. \n\n\n\nThank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community."}, {"review_id": "mjytCGwyK0", "reviewer": "Reviewer_UGcq", "summary": "This paper proposes a vision concept modeling (VCM) method to reduce visual tokens in VLMs. In details, VCM dynamically extracts task-relavant visual concepts (tokens) with a self-supevised instruction-finetuned technique,  and the key is a forward-backward optimization algorithm for adaptive concept granularity and spatial alignment. The proposed VCM can reduce computational costs and maintain strong performance across various vision-language tasks, including VAQ, zero-shot classification, and OV dense perception tasks.", "questions": "1. This paper focuses on static images/videos, it is unclear how VCM would handle temporal dynamics in complex videos?\n\n2. Have you tested VCM on non-English datasets? How does reliance on English keyword priors affect performance in other languages?\n\n3. Does VCM provide mechanisms to detect hallucinations or uncertain predictions when concepts are compressed (e.g., via confidence scores)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- **Simple yet effective idea.** This paper introduces the first adaptive visual concept modeling method to address inefficient token-level processing in VLMs. The proposed VCM reduces computational costs (e.g., achieving up to 85% fewer FLOPs for LLaVA-1.5-7B), enabling VLMs to scale to high-resolution images.\n- **Empirical validation.** Comprehensice experiments across 16 VQA datasets including image-, region-, and video-level benchmarks, zero-shot classification, OV dense prediction tasks.\n- **Scalability.** The authors have validated on a series of LLaVA architectures, such as LLaVA-1.5-7/13B, LLaVA-NeXT and video-LLaVA.\n\n## Weaknesses\n- **Strong dependence on the text.** The proposed VCM relies on keywords statistics from text instructions / responses, which may introduce bias in underspecified prompts.\n- Dynamic programming introduces $O(M^2)$ complexity for sequence alignment, which potentially limits scalability to long videos.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1**: Strong dependence on the text\n\n**Response:**\nThe keyword selection module in VCM provides reliable keyword selection results, while the random masking strategy integrated into the VCM algorithm introduces error tolerance and robustness toward incorrect keyword predictions. Together, these two features ensure efficient and accurate learning, reducing the dependency on keyword selection. To verify these two aspects, we conducted the following experiments:\n\nTo better illustrate the reliability of the keyword selection module, we randomly selected 1K samples from the 5K samples mentioned in line 85 of the paper, labeled by GPT-4o, and calculated the success rate of the module in selecting the correct keywords during training. The results are summarized below:\n\n| Training Steps | 0.5k | 1k   | 1.5k | 2k   | 2.5k | 3k   | 3.5k | 4k   | 4.5k | 5k   |\n| :------------- | ---- | :--- | :--- | :--- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Accuracy       | 19.4 | 48.9 | 67.4 | 76.5 | 83.6 | 88.9 | 91.3 | 92.6 | 94.1 | 94.3 |\n\nAs shown in the table, as training progresses, the keyword selection module effectively learns the relationships between text and images, ultimately achieving a 94.3% accuracy in selecting correct keywords.\n\nTo further validate VCM's robustness, during inference, we randomly replaced selected keywords with non-keywords at varying replacement probabilities *r* to simulate unreliable predictions by the keyword selection module. We then tested the model and calculated the average visual concept token length across SciQA, POPE, and MME datasets. The results are summarized below:\n\n| r    | Mean Token | SciQA | POPE | MME    |\n| :--- | :--------- | :---- | ---- | ------ |\n| 0%   | 64         | 69.1  | 85.2 | 1719.6 |\n| 15%  | 76         | 68.9  | 85.1 | 1720.3 |\n| 30%  | 89         | 69.0  | 84.9 | 1725.4 |\n| 45%  | 96         | 68.7  | 85.0 | 1726.8 |\n| 60%  | 103        | 68.9  | 85.1 | 1728.7 |\n\nFrom the table, it can be observed that VCM’s masking strategy ensures strong error tolerance. When the keyword selection module produces incorrect predictions, VCM tends to generate longer visual concepts to ensure the downstream LLM receives sufficient prior information for task reasoning.\n\n> **W2&Q1**: Dynamic programming introduces O(M2) complexity for sequence alignment, which potentially limits scalability to long videos. This paper focuses on static images/videos, it is unclear how VCM would handle temporal dynamics in complex videos?\n\n**Response:**\n\nCompared to previous methods based on attention mechanisms[1,2,3,4], VCM has the same theoretical computational complexity of *O*(*M*^2), with no significant difference in this regard. However, VCM can be effectively combined with downsampling techniques, where long videos are first downsampled and then further compressed using VCM. This combination allows VCM to be efficiently applied to long video understanding tasks. To further validate the scalability of our method in complex long video scenarios, we selected the challenging long-video benchmark dataset, Video-MME, for evaluation and comparison. Specifically, we processed 64 frames per input, and the experimental results are illustrated in the figure below.\n\n| Method      | Token$\\cdot$Frames | Video-MME |\n| :---------- | :----------------- | :-------- |\n| Video-LLaVA | 256$\\cdot$64       | 42.1      |\n| VCM         | 17$\\cdot$64        | 40.9      |\n\nCompared to the baseline method Video-LLaVA, VCM reduces the input sequence length for a single frame from 256 to 17, while maintaining strong performance on the Video-MME benchmark. This demonstrates that our method, like previous approaches, is plug-and-play and exhibits excellent scalability.\n\n> **Q2**: Have you tested VCM on non-English datasets? How does reliance on English keyword priors affect performance in other languages?\n\n**Response:**\n\nTo investigate the impact of non-English data on VCM's generalization capability, we collected the LLaVA-ZH Chinese dataset. We mixed it with the original LLaVA dataset and trained the model for 500 steps on the combined dataset. The model's performance was then tested on MMB-ZH and MMB-EN, and the experimental results are presented below.\n\n| Method         | Token | MMB-zh | MMB-en |\n| :------------- | ----- | :----- | :----- |\n| LLaVA+llava-zh | 576   | 44.6   | 46.5   |\n| VCM+llava-zh   | 144   | 43.3   | 45.8   |\n\nAs shown in the table, VCM can successfully train and test on Chinese data, achieving performance comparable to that on English data. This demonstrates VCM's strong cross-linguistic generalization capability.\n\n> **Q3**:Does VCM provide mechanisms to detect hallucinations or uncertain predictions when concepts are compressed (e.g., via confidence scores)?\n\n**Response:**\n\nWe conducted an ablation study to evaluate the impact of different confidence score thresholds on model performance during conditional reasoning after training for 500 steps, where the thresholds were also used to detect and filter uncertain information, influencing the model's performance.\n\n| Thres | SciQA | VizWiz | POPE | MME    | RefCOCO-testB (mask) |\n| :---- | :---- | :----- | :--- | ------ | -------------------- |\n| 0.5   | 64.1  | 48.2   | 76.6 | 1350.6 | 26.8                 |\n| 0.55  | 64.6  | 49.8   | 76.8 | 1355.2 | 27.3                 |\n| 0.6   | 64.7  | 49.7   | 76.8 | 1354.7 | 27.4                 |\n| 0.65  | 64.4  | 49.5   | 76.4 | 1353.3 | 27.0                 |\n\nAs shown in the table, increasing the confidence score threshold during inference helps filter out noisy and uncertain information, thereby improving the model's performance on downstream tasks. This indicates that the confidence scores in our end-to-end optimization design carry practical physical significance, aligning with the design principles stated in line 176 of the paper. Specifically, confidence scores represent the uncertainty in concept compression and can be effectively utilized to design mechanisms for detecting hallucinations or uncertain predictions.\n\n[1] MQT-LLaVA: Matryoshka Query Transformer for Large Vision-Language Models\n\n[2] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models\n\n[3] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction\n\n[4] VisionZip: Longer is Better but Not Necessary in Vision Language Models\n\n\n\nThank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community."}, {"review_id": "FuQjgxdBtO", "reviewer": "Reviewer_u1dV", "summary": "This paper proposes the framework of vision concept modeling through a three stage pipeline: instruction prior, key word extraction and dynamic programming selection of visual concept tokens, compresses visual tokens for MLLMs. The aim is to reduce computation while maintaining multi-vision language understanding tasks. Compared with state-of-the-art approaches, VCM cuts FLOPs by up to 85%, and with just 144 tokens it matches or surpasses a 576-token LLaVA-1.5 on eleven VQA benchmarks.", "questions": "Q1: If we perform self-supervised contrastive learning directly on vision tokens (without using text keywords), how large is the performance gap compared to VCM? Can this comparison demonstrate the necessity of the keyword module?\n\nQ2: What is the average VQA score of MQT-LLaVA under the 144-token condition? Does VCM's advantage stem from differences in search space or from textual priors?\n\nQ3: Under 1024×1024 high-resolution images and 30 fps video streaming conditions, what are the comparative metrics of VCM versus VisionZip and SparseVLM in terms of GPU latency and memory consumption?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n- The empirical study about relationship of keywords in instruction，keywords in response and needed visual token length provides valuable insights for future work.\n- Compared to previous methods，VCM can dynamically select and merge visual tokens while maintaining the original spatial order, which is critical for dense visual prediction tasks.\n\nWeaknesses:\n- Ablation study is not sufficient\n  - The length estimation employs min-max normalization with a fixed domain-specific threshold τ, lacking sensitivity analysis; whether task-specific τ values are required for different tasks remains unaddressed.\n  - Lack the effectiveness of direct token selection learning without a keyword module.\n  - No direct comparison was made with variable-length token encoding methods (e.g., Matryoshka Query Transformer) within the same token count range. \n- The figure of the overall framework is not clear enough. I suggest you can bind G^T,G^V..., with the corresponding tokens in the figure.", "was_revised": true, "final_justification": "The authors’ rebuttal successfully resolved all of my major concerns:\n\n- **Sensitivity of τ (W1.1)**: They ran an ablation showing that task-specific τ values do not improve performance and explained why; they also proposed a more flexible inference-time alternative.  \n- **Framework clarity (W2)**: They agreed to redraw the diagram to bind symbols to tokens.  \n- **Effectiveness of keyword module (Q1/W1.2)**: An extra experiment confirmed that removing KM hurts performance, proving its necessity.  \n- **Comparison with MQT-LLaVA at 144 tokens (Q2/W1.3)**: The authors provided the missing number and articulated why textual priors give VCM an edge.  \n- **High-res/video metrics (Q3)**: New tables show VCM is competitive in latency and memory, with clear explanations of efficiency gains.\n\nBecause these issues were convincingly addressed with data and sound reasoning, I raised my score.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1.1**: The length estimation employs min-max normalization with a fixed domain-specific threshold τ, lacking sensitivity analysis; whether task-specific τ values are required for different tasks remains unaddressed.\n\n**Response:**\n\nWe hypothesize that the fixed domain-specific threshold $\\tau$ refers to the information domain scalar $S$. To investigate the impact of the dynamic information domain $S$ on VCM, we categorized the training data. From the results in the ablation study shown in Table 4, we observed that for localization and fine-grained perception tasks, a smaller information domain yields better results. Conversely, for tasks requiring global information, a larger information domain is more effective.\n\nBased on this observation, we categorized the instruction-tuning data by task type. For open-ended tasks, we set the information domain to $1/2$, while for fine-grained perception tasks like the RefCOCO localization task, we set the information domain to $1/4$. We then conducted mixed training for 500 steps, and the experimental results are presented below.\n\n| Method          | SciQA | VizWiz | POPE | MME    |\n| :-------------- | :---- | :----- | :--- | ------ |\n| mix-S (1/4&1/2) | 60.5  | 44.6   | 70.7 | 1288.8 |\n| fix-S （1/4）   | 64.1  | 48.2   | 76.6 | 1350.6 |\n\nAs shown in the table, using mixed training with different information domains for different tasks did not lead to further performance improvements. We believe this is because dynamic information domains introduce challenges for the forward-backward optimization algorithm, making it difficult for VCM to discriminate whether the varying learning lengths are caused by differences in information domains or differences in keyword count. This conflict ultimately led to degraded performance.\n\nCompared to dynamically altering the information domain during the training phase, we believe that masking keywords to adaptively select task-specific lengths during inference could be a more flexible and efficient approach.\n\n> **W2**: The figure of the overall framework is not clear enough. I suggest you can bind G^T,G^V..., with the corresponding tokens in the figure.\n\n**Response:**\n\nWe sincerely thank the reviewers for their detailed and thoughtful feedback, which has greatly improved the readability and completeness of our work. In the revised version, we will incorporate binding to make the definitions more precise and clearer.\n\n> **Q1&W1.2**: Lack the effectiveness of direct token selection learning without a keyword module (KM).\n\n**Response:**\n\nWe conducted a complete retraining of VCM using only vision token selection configurations and then performed comparative experiments under the condition of 144 tokens. The experimental results are presented below.\n\n| Method | MME    | SciQA | RefCOCO-testA (mask) | RefCOCO-testB (mask) |\n| :----- | :----- | :---- | :------------------- | -------------------- |\n| w/ KM  | 1842.7 | 69.1  | 16.8                 | 41.6                 |\n| w/o KM | 1756.6 | 67.8  | 11.6                 | 36.8                 |\n\nAs shown in the table, relying solely on vision tokens for keyword selection prevents the model from leveraging the latent space of large language models, thereby limiting its ability to search for beneficial prior information. Furthermore, this approach fails to establish a robust visual concept model, making it unsuitable for dense perception tasks beyond VQA, such as RefCOCO. Consequently, its applicability is significantly restricted.\n\n> **Q2&W1.3**: What is the average VQA score of MQT-LLaVA under the 144-token condition? Does VCM's advantage stem from differences in search space or from textual priors?\n\n**Response:**\n\nWe sincerely thank the reviewers for their thorough review. As presented in Table 1 of the manuscript, we have already provided a performance comparison between MQT-LLaVA and VCM under the 144-token condition (58.6 vs. 60.8). It can be observed that VCM demonstrates a clear advantage. We believe this stems primarily from two key factors: \n\n1) **Enhanced utilization of textual priors**: VCM benefits more effectively from the latent space of textual priors, enabling better alignment between visual features and textual features.  \n\n2) **Higher information density**: At equivalent token lengths, visual concepts carry richer information content compared to visual tokens, thereby representing more information and further boosting performance.\n\n> **Q3**:  Under 1024×1024 high-resolution images and 30 fps video streaming conditions, what are the comparative metrics of VCM versus VisionZip and SparseVLM in terms of GPU latency and memory consumption?\n\n**Response:**\n\n| Method      | Token | Flops (T) | Memory (Mb) | Avg  |\n| :---------- | :---- | :-------- | :---------- | ---- |\n| LLaVA-NeXT  | 2880  | 20.8      | 18952       | 72.4 |\n| +VisionZip  | 160   | 2.21      | 13176       | 66.2 |\n| +SparseVLM  | 160   | 2.14      | 14054       | 63.0 |\n| +VCM        | 160   | 2.07      | 13232       | 70.1 |\n| Video-LLaVA | 2048  | 14.4      | 16854       | 54.2 |\n| +VisionZip  | 136   | 1.78      | 11234       | 50.3 |\n| +SparseVLM  | 198   | 2.31      | 12876       | 46.6 |\n| +VCM        | 136   | 1.69      | 11380       | 52.5 |\n\nWe sincerely thank the reviewers for their suggestions. In response, we provide a comparison of VisionZip and SparseVLM with VCM regarding FLOPs and GPU memory across high-resolution scenarios (LLaVA-Next) and video streaming (Video-LLaVA), as shown in Appendix Tables 7 and 8. From the tables, it is evident that VCM achieves the best overall results while maintaining competitive performance in both speed and memory consumption. \n\nThis advantage can be attributed to the following three reasons: \n\n1. **Model Parameter Efficiency**: VCM employs a simple two-layer attention module, which introduces minimal additional parameters compared to previous methods such as SparseVLM and VisionZip.\n\n2. **Computational Complexity**:  VCM utilizes a forward-backward optimization algorithm, ensuring its computational complexity remains equivalent to methods[1,2,3,4] based on attention mechanisms at $O(M^2)$.  \n\n3. **Parallel Optimization**: VCM incorporates parallel optimization techniques, as detailed in Algorithm 2 of the appendix, significantly enhancing its runtime efficiency.\n\nWe believe these attributes make VCM highly effective and efficient for both high-resolution and video-streaming scenarios. Thank you again for your constructive feedback, which has helped us emphasize these strengths more clearly in the revised version.\n\n\n\n[1] MQT-LLaVA: Matryoshka Query Transformer for Large Vision-Language Models\n\n[2] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models\n\n[3] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction\n\n[4] VisionZip: Longer is Better but Not Necessary in Vision Language Models\n\n\n\nThank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community."}, {"review_id": "7LhLNo2vDl", "reviewer": "Reviewer_RMxa", "summary": "This paper addresses the computational inefficiency of Large Vision-Language Models (LVLMs) which typically process all visual tokens from an image regardless of the specific query. The authors introduce Vision Concept Modeling (VCM), a novel self-supervised framework that allows a model to dynamically select a sparse, relevant set of \"vision concepts\" based on the user's textual instructions. At its core, VCM uses a forward-backward optimization algorithm to learn this adaptive token selection without requiring expensive, fine-grained annotations. This approach is shown to drastically reduce computational costs, achieving up to an 85% reduction in FLOPs for LLaVA-1.5-7B, while maintaining strong performance across a wide range of vision-language tasks.", "questions": "My questions aim to clarify two counter-intuitive experimental results:\n\n1.  **Ablation Anomaly (Table 4):** The ablation study shows that using 36 vision tokens surprisingly outperforms using 72 tokens. Could the authors provide a more detailed explanation for this counter-intuitive result? A convincing analysis would strengthen my confidence in the method's robustness.\n\n2.  **Fine-tuning Impact (Table 1):** Fine-tuning the vision encoder (`VCM*`) led to slightly lower average performance than keeping it frozen (`VCM`), which is unusual. Could the authors clarify the potential reasons for this, such as optimization instability? Addressing this surprising result would demonstrate a more thorough experimental analysis.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n* **Broad Applicability and Enhanced Capabilities:** The method's effectiveness extends beyond standard VQA, significantly enhancing the performance of dense perception models. For instance, it improves the $AP_{50}$ score in open-vocabulary object detection and boosts performance in open-vocabulary semantic segmentation across multiple datasets.\n\n* **Novel and Principled Methodology:** The use of a forward-backward algorithm based on dynamic programming is a technically sound and original approach to selecting a variable number of vision tokens. This provides an efficient solution ($\\mathcal{O}(M^2)$ complexity) to an otherwise intractable search problem, all without requiring expensive, fine-grained annotations.\n\n### **Weaknesses**\n\n* **Heuristic-Based Concept Length Estimation:** The model relies on a fixed, heuristic formula to estimate the required number of vision tokens (`L`). This length is calculated using min-max normalization of keyword counts, not learned adaptively, which the authors acknowledge may negatively impact performance.\n\n* **High Dependency on Keyword Selection:** The framework's performance is critically dependent on the accuracy of the keyword selection module. The paper concedes that this process can introduce biases, meaning an inaccurate keyword selection could compromise the entire vision concept extraction process.\n\n* **Lack of Analysis on VCM's Own Overhead:** The efficiency analysis primarily highlights the FLOPs reduction within the LLM itself. The paper does not provide a detailed breakdown of the computational overhead introduced by the VCM components (e.g., keyword selection and dynamic programming), which must run before the LLM processes the tokens.\n\n* **High Architectural Complexity:** The VCM framework is architecturally complex, integrating multiple distinct components such as a keyword selector, a cross-attention projector, and the dynamic programming module. This complexity could pose a barrier to easy implementation and adoption when compared to simpler token reduction methods.\n\n* **Computational Analysis Deferred to Appendix:** While the paper claims significant computational savings, the detailed theoretical analysis of FLOPs is located in the appendix rather than the main paper. Including a more direct summary of this analysis in the main body would better substantiate the efficiency claims upfront.\n\nI am willing to raise my rating if the authors can satisfactorily address these questions in their rebuttal.", "was_revised": true, "final_justification": "Thank you for the thorough and well-prepared rebuttal. Your responses, supported by new experimental results, have successfully addressed all of my initial concerns.\n\nSpecifically:\n\n* The new ablations on heuristic length (W1) and keyword selection robustness (W2) are very convincing.\n\n* The latency breakdown (W3) provides the missing analysis on VCM's overhead.\n\n* Your explanations for the counter-intuitive results in the ablation (Q1) and vision encoder fine-tuning (Q2) were particularly insightful, clarifying that these are known phenomena consistent with prior work and community best practices.\n\nGiven the strength of this rebuttal, my concerns have been resolved. I am now happy to support the paper and will raise my rating accordingly.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> **W1**: Heuristic-Based Concept Length Estimation.\n\n**Response:**\n\n\nTo investigate the impact of estimated length versus precise length on performance, we conducted the following comparative ablation experiments. Specifically, we trained the model for 500 steps using 5K samples (as mentioned in line 85 of the paper), which were labeled by GPT-4o.\n\n| length      | Token | SciQA    | VizWiz   | POPE     | MME        |\n| :---------- | :---- | :------- | :------- | -------- | ---------- |\n| estimated   | 144   | **63.5** | **47.2** | 76.2     | 1334.4     |\n| api-labeled | 144   | 62.9     | 47.1     | **76.4** | **1337.2** |\n\n------\n\nAs shown in the table, the experimental results under precise length and estimated length settings are quite similar. This is attributed to VCM's inherent error-tolerance mechanism:  \n\n1) **Handling underestimated lengths**: When the estimated length is shorter than ideal, VCM compensates by merging more vision tokens into a single vision concept token, thereby increasing the average information capacity of each vision concept token.  \n\n2) **Handling overestimated lengths**: When the estimated length is longer than necessary, VCM generates longer vision concept tokens, ensuring that the subsequent LLM can still extract sufficient information for downstream tasks.\n\n> **W2**: High Dependency on Keyword Selection.\n\n**Response:**\nThe keyword selection module in VCM demonstrates reliable performance in selecting relevant keywords. Additionally, VCM's random masking strategy introduces error tolerance and robustness against incorrect keyword predictions. Together, these two components ensure efficient and accurate learning, thereby reducing the dependency on keyword selection. To separately validate these two aspects, we conducted the following experiments:\n\nTo better illustrate the reliability of the keyword selection module, we randomly selected 1K samples from the 5K samples (as mentioned in line 85 of the paper) labeled by GPT-4o and computed the success rate of keyword selection by the module during training. The experimental results are reported in the table below.\n\n| Training Steps | 0.5k | 1k   | 1.5k | 2k   | 2.5k | 3k   | 3.5k | 4k   | 4.5k | 5k   |\n| :------------- | ---- | :--- | :--- | :--- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Accuracy       | 19.4 | 48.9 | 67.4 | 76.5 | 83.6 | 88.9 | 91.3 | 92.6 | 94.1 | 94.3 |\n\nAs shown in the table, during pretraining, the keyword selection module progressively learns the relationships between images and text, ultimately achieving a 94.3% success rate in selecting the correct keywords.\n\nTo further validate the robustness of VCM, we simulated unreliable predictions by the keyword selection module during inference. Specifically, we randomly replaced selected keywords with non-keywords at different replacement probabilities $r$, and then tested the model while recording the average visual concept token length.\n\n| r    | Mean Token | SciQA | POPE | MME    |\n| :--- | :--------- | :---- | ---- | ------ |\n| 0%   | 64         | 69.1  | 85.2 | 1719.6 |\n| 15%  | 76         | 68.9  | 85.1 | 1720.3 |\n| 30%  | 89         | 69.0  | 84.9 | 1725.4 |\n| 45%  | 96         | 68.7  | 85.0 | 1726.8 |\n| 60%  | 103        | 68.9  | 85.1 | 1728.7 |\n\nAs shown in the table, VCM exhibits strong error tolerance. When the keyword selection module makes incorrect predictions, VCM tends to generate longer visual concepts to ensure that the subsequent LLM can access sufficient prior information for downstream tasks.\n\n> **W3**: Lack of Analysis on VCM's Own Overhead.\n\n**Response:**\n\nWe sincerely thank the reviewers for their valuable suggestions, which have significantly improved the quality of our work. In response, we have added the following experiments to provide more detailed insights into the latency overhead of VCM components and compared them with other methods. Specifically, we separated the latency contributions of the original LLM and the additional modules introduced by each method for more precise comparisons.\n\n| Method    | Token | Not LLM(ms) | LLM(ms) | Total |\n| :-------- | :---- | :---------- | :------ | ----- |\n| LLaVA     | 576   | 0           | 57.82   | 57.82 |\n| FastV     | 128   | 2.39        | 28.31   | 30.70 |\n| PDrop     | 128   | 10.01       | 27.76   | 37.77 |\n| SparseVLM | 128   | 6.93        | 26.45   | 33.28 |\n| VCM       | 128   | 2.88        | 28.36   | 31.24 |\n\nAs shown in the table, two key observations can be made. First, the non-LLM overhead introduced by VCM is only slightly higher than FastV. This is because our method utilizes a lightweight design consisting of only a two-layer attention module, resulting in no significant increase in parameter count compared to previous methods. Second, our approach incorporates weighted averaging and parallel optimization techniques, as detailed in Appendix Algorithm 2, which further reduces the latency caused by the additional components in our method.\n\n> **W4**: High Architectural Complexity.\n\n**Response:**\n\nWe sincerely thank the reviewer for their careful review. In fact, while VCM may appear theoretically complex, in practice it consists of only two attention layers, resulting in no significant increase in parameter count compared to previous methods. Additionally, its computational complexity remains the same as earlier approaches, at $O(M^2)$. Moreover, we have implemented parallel optimization techniques, enabling VCM to operate efficiently.\n\nAs a plug-and-play method, VCM has consistently demonstrated its simplicity, efficiency, and versatility across diverse model architectures (e.g., LLaVA [1], QwenVL [2]) and various application scenarios (e.g., standard images, high-resolution images, and video). This further highlights its generalizable nature and adaptability for different use cases.\n\n> **W5**: Computational Analysis Deferred to Appendix.\n\n**Response:**\n\nWe sincerely thank the reviewer for their thoughtful suggestions, which have helped improve the readability of our work. Due to page limitations, we chose to place the performance comparisons in the appendix. In the revised version, we will include a dedicated discussion and highlight the location of the relevant tables to ensure readers can easily find and focus on them.\n\n> **Q1**: Could the authors provide a more detailed explanation for this counter-intuitive result?\n\n**Response:**\n\nUsing fewer tokens yielding better performance on certain benchmarks is a normal phenomenon and has been observed multiple times in previous works, such as MQT-LLaVA [1] (Table 1, MMB), FastV [2] (Table 1, MMMU), PDrop [3] (Table 1, POPE), and VisionZip [4] (Table 11, VizWiz). This arises because different tasks demand varying levels of perceptual granularity from the model. \n\nFor instance, tasks like VizWiz emphasize local perceptual abilities, where excessive tokens may introduce noise and redundant information, leading to reduced performance compared to fewer tokens. Conversely, tasks like MME rely heavily on global details, where more tokens provide greater benefits, making such phenomena less likely to occur. This difference in task requirements explains the seemingly counterintuitive results observed for certain benchmarks.\n\n> **Q2**: Fine-tuning the vision encoder led to slightly lower average performance than keeping it frozen, which is unusual. \n\n**Response:**\n\nFor LLaVA and subsequent works [3,4,5,6] based on LLaVA, freezing the vision encoder has become the default configuration. This is not due to increased computational overhead—since the vision encoder's computation cost is negligible compared to that of large language models—but rather because experiments have shown that jointly training the vision encoder with low-resolution images and limited data often yields suboptimal results.\n\nA commonly accepted explanation within the community is that freezing the vision encoder helps preserve the pre-trained knowledge in the encoder, preventing its disruption when trained on small datasets. To ensure fair comparisons, we also report results for models with the vision encoder unfrozen in Table 1. As can be observed, unfreezing the vision encoder parameters indeed leads to suboptimal performance, particularly for the baseline LLaVA-v1.5 model.\n\n[1] LLaVA: Large Language and Vision Assistant\n\n[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond\n\n[3] MQT-LLaVA: Matryoshka Query Transformer for Large Vision-Language Models\n\n[4] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models\n\n[5] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction\n\n[6] VisionZip: Longer is Better but Not Necessary in Vision Language Models\n\n\n\nThank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the multi-modal research and the open-source community."}]

Author rebuttal:

Meta-review: This paper proposes a novel method, namely "vision concept modeling," to address token-level redundancy for high-resolution images in Vision-Language Models. By learning visual concepts in a self-supervised manner, the proposed method significantly reduces computational costs while maintaining strong performance on downstream tasks. Reviewers acknowledged this as a valuable and timely contribution to the field.

While the initial reviews raised constructive questions regarding ablation studies, experimental comparisons, and methodological heuristics, the authors provided a thorough and effective rebuttal that addressed these points comprehensively. The reviewers were satisfied with the authors' responses and engaged productively during the discussion period. There is a clear consensus that this will be a solid paper once the promised revisions are incorporated into the final manuscript. Therefore, the recommendation is Accept.
Decision: Accept (poster)