Title: VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large vision-language models

vision concept modeling

token compression

efficiency
Abstract: Large vision-language models (LVLMs) have emerged as foundational tools for real-world AI applications. Despite their remarkable capabilities, current LVLMs process entire images at the token level, leading to significant inefficiencies compared to human cognition, which selectively focuses on high-level vision concepts. This token-level redundancy becomes increasingly problematic for high-resolution images and long video sequences, resulting in large computational costs and limited scalability in practical applications. To address this limitation, we introduce the concept of a vision concept model, a novel paradigm that enables LVLMs to dynamically extract the most relevant vision concepts from complex inputs, based on task-specific instructions. To optimize this vision concept modeling process, we propose VCM, a self-supervised framework that leverages vision-language correlations across diverse instances. VCM is designed to learn meaningful vision concepts without the need for expensive concept-level annotations. At its core, it employs a forward-backward optimization algorithm that supports LVLMs to adjust concept granularity and spatial alignment dynamically. Experiments demonstrate that VCM remarkably reduces computational costs (e.g., achieving up to 85\% fewer FLOPs for LLaVA-1.5-7B), while maintaining strong performance across a series of vision-language tasks. The codebase is available at https://github.com/RainBowLuoCS/VCM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the computational inefficiency of Large Vision-Language Models (LVLMs) which typically process all visual tokens from an image regardless of the specific query. The authors introduce Vision Concept Modeling (VCM), a novel self-supervised framework that allows a model to dynamically select a sparse, relevant set of \"vision concepts\" based on the user's textual instructions. At its core, VCM uses a forward-backward optimization algorithm to learn this adaptive token selection without requiring expensive, fine-grained annotations. This approach is shown to drastically reduce computational costs, achieving up to an 85% reduction in FLOPs for LLaVA-1.5-7B, while maintaining strong performance across a wide range of vision-language tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n* **Broad Applicability and Enhanced Capabilities:** The method's effectiveness extends beyond standard VQA, significantly enhancing the performance of dense perception models. For instance, it improves the $AP_{50}$ score in open-vocabulary object detection and boosts performance in open-vocabulary semantic segmentation across multiple datasets.\n\n* **Novel and Principled Methodology:** The use of a forward-backward algorithm based on dynamic programming is a technically sound and original approach to selecting a variable number of vision tokens. This provides an efficient solution ($\\mathcal{O}(M^2)$ complexity) to an otherwise intractable search problem, all without requiring expensive, fine-grained annotations.\n\n### **Weaknesses**\n\n* **Heuristic-Based Concept Length Estimation:** The model relies on a fixed, heuristic formula to estimate the required number of vision tokens (`L`). This length is calculated using min-max normalization of keyword counts, not learned adaptively, which the authors acknowledge may negatively impact performance.\n\n* **High Dependency on Keyword Selection:** The framework's performance is critically dependent on the accuracy of the keyword selection module. The paper concedes that this process can introduce biases, meaning an inaccurate keyword selection could compromise the entire vision concept extraction process.\n\n* **Lack of Analysis on VCM's Own Overhead:** The efficiency analysis primarily highlights the FLOPs reduction within the LLM itself. The paper does not provide a detailed breakdown of the computational overhead introduced by the VCM components (e.g., keyword selection and dynamic programming), which must run before the LLM processes the tokens.\n\n* **High Architectural Complexity:** The VCM framework is architecturally complex, integrating multiple distinct components such as a keyword selector, a cross-attention projector, and the dynamic programming module. This complexity could pose a barrier to easy implementation and adoption when compared to simpler token reduction methods.\n\n* **Computational Analysis Deferred to Appendix:** While the paper claims significant computational savings, the detailed theoretical analysis of FLOPs is located in the appendix rather than the main paper. Including a more direct summary of this analysis in the main body would better substantiate the efficiency claims upfront.\n\nI am willing to raise my rating if the authors can satisfactorily address these questions in their rebuttal.\n\nQUESTIONS:\nMy questions aim to clarify two counter-intuitive experimental results:\n\n1.  **Ablation Anomaly (Table 4):** The ablation study shows that using 36 vision tokens surprisingly outperforms using 72 tokens. Could the authors provide a more detailed explanation for this counter-intuitive result? A convincing analysis would strengthen my confidence in the method's robustness.\n\n2.  **Fine-tuning Impact (Table 1):** Fine-tuning the vision encoder (`VCM*`) led to slightly lower average performance than keeping it frozen (`VCM`), which is unusual. Could the authors clarify the potential reasons for this, such as optimization instability? Addressing this surprising result would demonstrate a more thorough experimental analysis.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the framework of vision concept modeling through a three stage pipeline: instruction prior, key word extraction and dynamic programming selection of visual concept tokens, compresses visual tokens for MLLMs. The aim is to reduce computation while maintaining multi-vision language understanding tasks. Compared with state-of-the-art approaches, VCM cuts FLOPs by up to 85%, and with just 144 tokens it matches or surpasses a 576-token LLaVA-1.5 on eleven VQA benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- The empirical study about relationship of keywords in instruction，keywords in response and needed visual token length provides valuable insights for future work.\n- Compared to previous methods，VCM can dynamically select and merge visual tokens while maintaining the original spatial order, which is critical for dense visual prediction tasks.\n\nWeaknesses:\n- Ablation study is not sufficient\n  - The length estimation employs min-max normalization with a fixed domain-specific threshold τ, lacking sensitivity analysis; whether task-specific τ values are required for different tasks remains unaddressed.\n  - Lack the effectiveness of direct token selection learning without a keyword module.\n  - No direct comparison was made with variable-length token encoding methods (e.g., Matryoshka Query Transformer) within the same token count range. \n- The figure of the overall framework is not clear enough. I suggest you can bind G^T,G^V..., with the corresponding tokens in the figure.\n\nQUESTIONS:\nQ1: If we perform self-supervised contrastive learning directly on vision tokens (without using text keywords), how large is the performance gap compared to VCM? Can this comparison demonstrate the necessity of the keyword module?\n\nQ2: What is the average VQA score of MQT-LLaVA under the 144-token condition? Does VCM's advantage stem from differences in search space or from textual priors?\n\nQ3: Under 1024×1024 high-resolution images and 30 fps video streaming conditions, what are the comparative metrics of VCM versus VisionZip and SparseVLM in terms of GPU latency and memory consumption?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a vision concept modeling (VCM) method to reduce visual tokens in VLMs. In details, VCM dynamically extracts task-relavant visual concepts (tokens) with a self-supevised instruction-finetuned technique,  and the key is a forward-backward optimization algorithm for adaptive concept granularity and spatial alignment. The proposed VCM can reduce computational costs and maintain strong performance across various vision-language tasks, including VAQ, zero-shot classification, and OV dense perception tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- **Simple yet effective idea.** This paper introduces the first adaptive visual concept modeling method to address inefficient token-level processing in VLMs. The proposed VCM reduces computational costs (e.g., achieving up to 85% fewer FLOPs for LLaVA-1.5-7B), enabling VLMs to scale to high-resolution images.\n- **Empirical validation.** Comprehensice experiments across 16 VQA datasets including image-, region-, and video-level benchmarks, zero-shot classification, OV dense prediction tasks.\n- **Scalability.** The authors have validated on a series of LLaVA architectures, such as LLaVA-1.5-7/13B, LLaVA-NeXT and video-LLaVA.\n\n## Weaknesses\n- **Strong dependence on the text.** The proposed VCM relies on keywords statistics from text instructions / responses, which may introduce bias in underspecified prompts.\n- Dynamic programming introduces $O(M^2)$ complexity for sequence alignment, which potentially limits scalability to long videos.\n\nQUESTIONS:\n1. This paper focuses on static images/videos, it is unclear how VCM would handle temporal dynamics in complex videos?\n\n2. Have you tested VCM on non-English datasets? How does reliance on English keyword priors affect performance in other languages?\n\n3. Does VCM provide mechanisms to detect hallucinations or uncertain predictions when concepts are compressed (e.g., via confidence scores)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Visual Concept Model (VCM), a new type of large vision language models (LVLMs) where visual embeddings focus on semantic concepts, forming a shorter sequence. The training procedure of VCM follows the standard pre-training and instruction tuning pipeline. In the pre-training stage, the authors propose to train an additional keyword selection module via a semantic alignment loss, and a multi-head cross-attention layer to enhance visual representation. During instruction tuning, the keyword selection module identifies keywords in the text, which are then used to compute the proposed VCM loss. Through experiments, the authors reveal a negative correlation between the required visual token length and the keyword number difference between text instructions and responses. This relationship helps estimate the sequence length of the visual concept embeddings. Finally, vision concept embeddings are selected based on a learned assignment probability, optimized via the VCM loss with the help of dynamic programming. Extensive experiments across diverse benchmarks and tasks are conducted to validate VCM’s effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method is well-motivated and novel. Representing the visual inputs by a shorter yet more semantically meaningful visual embedding sequence can address the sequence length limitation of current LVLMs and narrow their gap to human cognition. The paper introduces several novel modules and optimization objectives to obtain the visual concept embeddings that potentially shed lights to developing new LVLMs.\n2. The analytical experiment that reveals the correlation between keyword number and minimum visual sequence length is interesting.\n3. The evaluation is comprehensive with promising results.\n\nWeaknesses:\nWhile the proposed method is novel, its presentation could be enhanced by simplifying notations and incorporating additional details for clarity. Current presentation increases the difficulty for fully understand the paper. The reviewer may consider lower the score if the following concerns are not well addressed in the rebuttal.\n1. The notations are complicated:\n - The keyword selection module $f_{KS}$ and  'MHSA' in line 102 and equation (2) appears to be the same thing. Consider unifying these notations.\n - Similarly, consider simplifying $f_{V2T}$ in section 3 to $f_M$.\n - The text prior $T$ in line 121 is undefined, which the reviewer assumes to be the keywords.\n - Typo in line 121.\n - Consider using a notation like $f_{T}$ to represent the LLM's word embeddings (by regarding them as an embedding function) instead of $X^T$ to align with other notations.\n2. Several components are left unexplained. Please refer to the questions.\n\nQUESTIONS:\nThe reviewer is uncertain about the following details:\n1. What is the objective of the binary classification in equation (3)? How to train the linear $f_{CLS}$?\n2. How to obtain each probability $p(z_l|y_t)$ when conducting forward and backward probability transition in section 3.3?\n3. Please explain the implicit contrastive sampling module and the corresponding training process. These content seem missing in the main text and only appear in the caption for Figure 3.\n4. How does the proposed method address the first key challenge mentioned in line 73?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}