Title: Compress & Cache: Vision token compression for efficient generation and retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: token compression

llava
Abstract: This work aims to compress the vision tokens of an LVLM into a representation that is simultaneously suitable for (a) generative and (b) discriminative tasks, (c) is nearly lossless, and (d) storage-efficient. 
To this end, we propose C&C, a novel compression method that leverages the LVLM itself for task-agnostic visual token compression.
Unlike prior methods that perform token reduction on-the-fly, our approach offloads computation to a dedicated, upfront indexing stage, effectively decoupling compression from generation. This enables learning more powerful representations for generation during inference. 
At the core of C&C is a ``double-forward pass'' training strategy. During the first forward pass, the LLM (of the LVLM) creates a bottleneck by compressing the dense visual tokens into a few summary tokens. Subsequently, the second forward pass processes the language instruction(s) alongside the summary tokens, used as a direct replacement for the image ones. 
The training of C&C is guided by two key losses: an autoregressive loss applied after the second pass that provides a direct optimization objective for reconstructing the original information flow, and a contrastive loss applied after the first pass to bolster the representational strength of the summary tokens, particularly for discriminative tasks. Moreover, we propose stage-specific adapters for further enhancing performance. C&C  produces highly informative compressed representations. An in-depth ablation study confirms the efficacy of our approach. For generative tasks, we achieve a 2x higher compression rate without compromising capabilities, setting a new state-of-the-art. For discriminative tasks, we establish new state-of-the-art results on image retrieval and compositionality benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to optimize the inference speed of VLMs in a RAG scenario.\n\nSpecifically, given a pretrained VLM, the authors perform a first inference pass using an image, a pre-defined summarization prompt, and a small set of summary tokens. The summary tokens are then cached and used for further inferences alongside a query prompt.\n\nThe authors claim that this should significantly improve speed, as during \"online\" inference, the model would then process only around 6% of the image tokens, which the authors claim constitutes the overwhelming part of the input sequence compared to the textual query.\n\nTo make this possible, the authors fine-tune a pretrained VLM, in their case LLaVA 1.5, via LoRA adapters, using two losses:\n\n1. A contrastive loss, applied on the summary tokens after the first forward pass.\n2. An auto-regressive loss, applied on the generations produced by the model, given the text instructions and the summary tokens.\n\nThe two losses are then summed and jointly optimized using two LoRA adapters, one for summarization and one for generation. The adapters are trained for 10,000 steps using data from LLaVA-665K and CC3M, representing generative and contrastive tasks, respectively. The authors aim for a 1:1 sampling ratio between the two.\n\nThe fine-tuned models are then comprehensively evaluated across several text generation and image/text retrieval benchmarks, and the results show that C&C largely outperforms existing token pruning and matryoshka methods, even surpassing the original LLaVA baseline in several cases.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is well-written and easy to follow.\n2. The paper addresses a relevant problem, namely improving the inference speed of VLMs, with a focus on RAG/indexing scenarios, which are relevant for real-world use cases.\n3. Their method is comprehensively evaluated across various datasets and tasks and compared to several recent token pruning and matryoshka methods, largely improving over them.\n\n### Weaknesses\n\nMy primary concern regarding this paper is the confounding effect of LoRA fine-tuning. In particular, I am surprised to see that C&C improves over the baseline in several cases, e.g., +3.1% on VisWiz with 32 tokens, +0.1% on MMBench with only 16 tokens, representing a 97% reduction in the original number of tokens.\n\nThus, I wonder how much of this performance improvement (both over LLaVA and other methods) can be attributed to the LoRA fine-tuning and how much to C&C. I believe that a comparison between a LLaVA model fine-tuned via LoRA using a similar procedure (as much as possible) and C&C would provide clearer insights.\n\nBeyond that, there are some minor details:\n\n1. I believe that the dataset size comparison between EVA-CLIP and C&C on line 271 is unfair, as it compares the size of the dataset used for training this specific CLIP model versus the size of a fine-tuning dataset.\n2. Table 1, for MMB, the best-performing method with 36 tokens is Matryoshka Multi, but its entry is not bolded.\n3. Small typo in line 47 of the supplementary, \"performs\" should be \"performance\".\n\nQUESTIONS:\n1. How are the summary tokens initialized?\n2. While the authors provide a FLOPs-based comparison between indexing and inference in Figure 5, I wonder how the inference speed would vary (and compare to other token pruning methods) under different scenarios, e.g., when the image is given by the user, and no retrieval is performed. Would it then be more efficient to use a matryoshka method compared to two VLM inference steps?\n3. The paper states that they aim for a 1:1 sampling ratio between LLaVA 665k and CC3M. Do the authors aim for an overall 1:1 ratio, or do they strive to maintain this ratio in each batch? Would a different sampling ratio improve performance?\n4. It would be interesting to know the total runtime of the LoRA fine-tuning process.\n5. To my understanding, the first-stage LoRA adapters are optimized for both the contrastive and auto-regressive objectives, while the second-stage adapters are optimized only for the auto-regressive objective. Is this correct?\n\nLIMITATIONS:\nThe authors address some limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper presents a novel method, named C&C, to compress vision tokens into more compact features for a given VLM with minimal performance loss. The proposed method employs a “double-forward” calculation flow with two training objectives, enabling the VLM itself to extract compressed tokens that are simultaneously suitable for generation, discrimination, and storage. Extensive experiments across various scenarios demonstrate the effectiveness and efficiency of the proposed method.

STRENGTHS AND WEAKNESSES:
### Strengths
1. The overall writing and organization of this paper is good.
2. The proposed method is simple, effective, and straightforward.
3. The experiments are sufficient to demonstrate the effectiveness of the proposed method.

### Weaknesses
The illustrations can be further refined for clarification and better readability. For example, Figure 2 does not show that the proposed method does not require finetuning the entire LLM, and the stage-specific LoRA modules are also missing from this figure.

QUESTIONS:
Regarding the storage perspective, the proposed method needs to store a summarized version of images with shape $R^{k' \times d}$, where $d$ is the hidden dimension of LLM. In contrast, the original LLaVA-1.5 only need to store features with shape $R^{k \times d_v}$, where $d_v < d$ is the dimension of the vision encoder output. It would be better to clarify this difference to illustrate the actual compression difference, rather than just comparing the number of tokens.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Compress & Cache (C&C) to compressing vision tokens in Large Vision Language Models (LVLMs). Unlike prior works that focus on on-the-fly token compression, C&C introduces an offline token compression and caching strategy, effectively decoupling the compression process from inference. The author of the paper proposed double-forward pass training strategy, which uses the LVLM itself to generate compact visual summary tokens suitable for both generative and discriminative tasks. C&C employs both an autoregressive loss and a contrastive loss, and further improves performance using stage-specific LoRA adapters. The method achieves state-of-the-art results on a wide range of VQA, image captioning, and image-text retrieval benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The novelty of the paper is good. The paper introduces a double-forward pass strategy that reuses the same LVLM for both compression and generation.\n\n2. The incorporation of both autoregressive loss (L_AR) and contrastive loss (L_disc) allows C&C to perform well in both domains.\n\n3. The experiments performed in the paper is comprehensive. The author of the paper conducted sufficient tasks with C&C and compare with many baseline methods. The results shows robustness of the method.\n\n4. The ablation and FLOPs results are very helpful.\n\n**Weaknesses**\n\n1. The most significant weakness of the proposed method is the lack of runtime latency results on the real hardware. The FLOPs analysis it good but it remains in theoretical computation. A comprehensive testing on GPU or other computing devices will be more convincing.\n\n2. For any compression work, the main purpose is to maintain performance and reduce computation. Even the performance is good, it is still unknow if the robustness is affected. Some works report the result variance, or test the method in perturbed or adversarial input to show how robust the compressed model is. My concern on this paper is how the C&C performs when in such scenario?\n\nQUESTIONS:\nplease refer to weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces C&C (Compress & Cache), a novel paradigm for compressing vision tokens in Large Vision-Language Models (LVLMs). C&C decouples compression from inference by performing a one-time, offline indexing step where the LVLM itself compresses numerous vision tokens into a few compact \"summary tokens.\" These cached tokens are then used for efficient online inference. The method is trained with a \"double-forward pass\" strategy, jointly optimized by an autoregressive loss for generation and a contrastive loss for discrimination. C&C aims to create a unified, nearly lossless representation for both generative and discriminative tasks, and experimental results show it sets a new state-of-the-art on various benchmarks, particularly in high-compression settings and for Visual RAG.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The core contribution is the \"Compress & Cache\" paradigm, shifting compression from an on-the-fly task to an offline process. This is a highly novel and practical approach, perfectly suited for real-world applications like Retrieval-Augmented Generation (RAG) and on-device deployment, where pre-indexing is feasible. It elegantly bypasses the inherent limitations of online compressors.\n\n2. The \"double-forward pass\" strategy, which enables the LVLM to act as its own compressor, is a key technical highlight. This \"self-compression\" mechanism leverages the LLM's intrinsic capabilities for information integration, allowing it to create high-fidelity summary tokens without relying on external modules, which is crucial for its near-lossless performance.\nRobust and Comprehensive Experiments: The paper's claims are substantiated by rigorous and comprehensive experiments. The extensive ablation studies (Tables 7, 8, 9) systematically validate each design choice, including the dual-loss function, the double-forward pass, and stage-specific LoRA adapters, providing strong evidence for the method's effectiveness.\n\n**Weaknesses**\n1. The method's core strength is also a limitation. It is not suitable for real-time applications (e.g., live video analysis) where pre-processing is impossible. While this is an intrinsic property of the paradigm, it constrains the method's scope of application.\n\n2. The paper claims C&C is a general method, yet all experiments are confined to the LLaVA family. The LVLM landscape includes diverse architectures (e.g., Qwen-VL, CogVLM) with different vision-language interfaces. The effectiveness of the \"self-compression\" mechanism is not guaranteed to transfer across these architectures. By limiting validation to a single model family, the paper's claim of generalizability remains insufficiently supported by empirical evidence.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}