Title: Vision-centric Token Compression in Large Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Token Compression

Long Context LLMs

Large Language Model

Visual-Text

Vision-centric
Abstract: Real-world applications are stretching context windows to hundreds of thousand of tokens while Large Language Models (LLMs) swell from billions to trillions of parameters.
This dual expansion send compute and memory costs skyrocketing, making $\textit{token compression}$ indispensable.
We introduce Vision Centric Token Compression ($\textbf{Vist}$), a $\textit{slow–fast}$ compression framework that mirrors human reading: 
the $\textit{fast}$ path renders distant tokens into images, letting a $\textbf{frozen, lightweight vision encoder}$ skim the low-salience context; 
the $\textit{slow}$ path feeds the proximal window into the LLM for fine-grained reasoning.
A Probability-Informed Visual Enhancement (PVE) objective masks high-frequency tokens during training, steering the Resampler to concentrate on semantically rich regions—just as skilled reader gloss over function words.
On eleven in-context learning benchmarks, $\textbf{Vist}$ achieves the same accuracy with 2.3$\times$ fewer tokens, cutting FLOPs by 16\% and memory by 50\%.
This method delivers remarkable results, outperforming the strongest text encoder-based compression method CEPE by $\textbf{7.6}$\% on average over benchmarks like TriviaQA, NQ, PopQA, NLUI, and CLIN, setting a new standard for token efficiency in LLMs. The project is at https://github.com/CSU-JPG/VIST.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a textual token compression approach using a lightweight vision encoder to process the text contents and a visual resampler to obtain a concentrated text input. The experiments show the proposed method can achieve higher compression ratio than previous method and well maintain the baseline performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The proposed method is interesting, and it is impressive to see that it achieves strong empirical results.\n\n2. The masking technique is both simple and effective, leading to a notable improvement in performance.\n\n## Weaknesses\n\n1. The paper aims to compress text tokens for LLMs using a visual encoder. Given this, it is unclear why the authors chose to use an LLM (TinyLlama) instead of a multi-modal LLM (MLLM). Clarification on this design choice would strengthen the paper.\n\n2. Experiments are conducted solely on TinyLlama. The results would be more convincing if validated on a wider range of baseline models.\n\n3. The method for splitting tokens between the visual encoder and the LLM is not clearly described. Additionally, it is not clear how performance is affected if the assignment of text tokens between the visual encoder and LLM is swapped. Further analysis or ablation studies on this aspect would be helpful.\n\nQUESTIONS:\n1. I recommend that the authors evaluate their method using additional LLMs to strengthen the experimental results and demonstrate the generalizability of their approach.\n\n2. In Table 1, while the reduction in TFLOPs is relatively modest, the throughput shows a significant improvement. I would appreciate clarification on how these metrics were measured, and an explanation for this discrepancy. Understanding the underlying reasons for this phenomenon would help me better interpret the results.\n\nLIMITATIONS:\nNo, the authors did not discuss the negative societal impact of their work in Appendix J.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VIST (Vision-centric Token Compression), a framework for efficiently extending the context length of large language models (LLMs). \n\nVIST draws inspiration from human selective reading, where readers focus on informative, low-frequency words and skim over high-frequency ones. Following this slow-fast strategy, VIST converts distant, low-salience text into images processed by a lightweight, frozen vision encoder, while keeping proximal context as raw text tokens.\n\nTo enhance compression quality, the authors propose Probability-Informed Visual Enhancement (PVE), a training objective that masks high-frequency, low-information textual tokens and aligns visual features with the remaining semantically rich text embeddings. This enables the vision encoder to produce semantically dense visual tokens.\n\nExperiments on long-context language modeling demonstrate that VIST reduces the number of tokens by 2.3× compared to the original text, while also cutting FLOPs by 16% and memory usage by 50% relative to the text-encoder-based compression baseline CEPE. Additionally, VIST consistently outperforms CEPE on most in-context learning and open-domain question answering tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The idea of compressing long textual context by rendering them as images is novel and interesting.\n- VIST demonstrates strong performance across three key task categories—long-context language modeling, in-context learning, and open-domain question answering—consistently outperforming text-encoder-based compression methods while clearly reducing computational costs.\n- Ablation study demonstrates the effectiveness of the proposed PVE training strategy, and the analysis on the visual-text embedding distance provides valuable insights into the frequence-based masking strategy.\n### Weaknesses\n- The number of visual tokens in VIST depends on several factors, including the resolution of the rendered text images, font size, and the number of tokens mapped to each image. However, the current manuscript does not provide a systematic analysis comparing visual token counts to textual token counts under different settings of these factors. Such an analysis is essential to better understand the advantage of image rendering in compressing long textual context.\n- VIST employs an encoder-decoder architecture to handle long-context inputs. While effective in static settings, this design is ill-suited for tasks involving continuous or streaming inputs, such as multi-turn dialogue. In these scenarios, the entire context must be repeatedly re-encoded with each new input, leading to significant inefficiencies.\n- The choice of vision encoder is a critical component of the VIST framework, as it directly affects the quality of visual token representations. However, the current manuscript does not explore the impact of different vision encoders.\n\nQUESTIONS:\n- Have you considered how to deal with multi-turn dialog using VIST?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe inference cost of LLMs directly scales with the number of input tokens. This has propelled a lot of work around ways to compress the input tokens or process the input tokens intelligently, while undergoing minimum performance loss. This paper proposes an interesting idea of parsing long context input in LLMs as multiple text images and passing them through a vision encoder which has OCR capabilities. THe vision encoder output can then be projected to the LLM embedding layer, and processed with some text input (finegrained input) to get the overall prediction. While the approach in novel and interesting, the empirical results are not that convincing as I detail in my review.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea is very interesting, and quite different from the standard attention and other thresholding based token compression works that I have read.\n- I really like the motivation of the approach around how humans read: finegrained + coarse attention. \n- The end to end approach makes sense, along with various sub-components like frequency based masking.\n\nWeakness:\nMy key issue with the paper is that the empirical results show it’s limited success in practice, while being an interesting idea.\n\n- First, in Table 1, the authors evaluate perplexity on the last 256 tokens in a long context input. Note that the preceding 2k tokens (T_d) are processed without any compression i.e. via standard LLM forward pass. Now, it is not clear whether the T_e tokens play an important role in determining the perplexity, especially given that the immediately preceding tokens are anyways being processed via standard forward pass. This issue can even be seen in the fact that as T_e i.e. the context is being increased (from 2048 to 14336), the final perplexity doesn't seem to improve that much. \n- Moreover, in Table1, the gains over the CEPE text encoder based baseline doesn’t seem to be significant enough, both in terms of flops reduction or performance.\n- Table 2: Authors seem to work with small in-context learning tasks, that too for text classification (like SST2). It would be better to try out modern long context  analysis tasks, needle in hay stack tasks, etc. Nonetheless, even in the setting the authors considered for Table 2, the performance drop with token reduction is indeed significant (like from 57 to 46). This really raises questions about efficacy of the approach or the claims. \n- Table 3 is a bit hard to digest. If the input context to standard model (TinyLlama) is with 15 passages, why does EM score drop below 1. Shouldn’t your implementation ignore the earlier input if the context length is limited. \n- Similarly, in Table 3, on NQ and PopQA, it looks like the additional set of passaages (k_e) do not really bring significant performance improvement beyond that of what you get with just processing recent past input (k_d).\n\nQUESTIONS:\nSee the weakness section. \n\nOverall, while the proposed ideas are interesting, it looks like unfortunately, the empirical observations are not strong enough,\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a vision-centric token compression method for large language models (LLMs), named VIST, which renders long textual contexts as images and employs a lightweight vision encoder for compression. In addition, a probability-informed visual enhancement (PVE) strategy is introduced to guide the compression process toward semantically rich regions, thereby improving the alignment between visual and textual features. Experimental results demonstrate the effectiveness of the proposed approach across multiple benchmarks and tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper is well-written and effectively presents the motivation and method in a concise and organized manner.\n2.\tThe idea of rendering text as images for long-context compression is both novel and intuitively appealing.\n3.\tThe experimental results are comprehensive and convincingly demonstrate the effectiveness of the proposed vision-centric compression strategy.\n\nWeaknesses: \n\n1.\tThe scalability and generative capabilities of the proposed approach across different-sized LLMs are not sufficiently validated. We suggest including experiments with LLaMA-2 models at multiple scales, particularly since LLaMA-2 serves as the baseline in CEPE. This would provide a more meaningful comparison and better demonstrate the generalization ability of the proposed method, beyond the current results with only TinyLLaMA-1.1B and Mistral-7B. \n2.\tThe manuscript briefly mentions the Perceiver Sampler but does not provide a detailed explanation of its architectural design. Including a more thorough description of its structure would enhance clarity and help readers better understand its role and details.\n3.\tThe training pipeline of VIST lacks sufficient clarity. Specifically, it remains ambiguous whether the Perceiver Resampler is pre-trained offline using the loss $L_{PVE}$ or jointly optimized with the LLM during tuning the cross-attentions. We suggest explicitly stating the training strategy to avoid confusion and to improve reproducibility.\n\nQUESTIONS:\n1.\tThe authors mention that a lower compression ratio does not always lead to better results. Could this be due to potential side effects introduced by the frequency-based masking strategy?\n2.\tDoes adopting frequency masking with different ratios contribute to improved performance?\n3.\tWould the proposed approach remain effective if only the fast path were applied directly to processed the whole rendered text as an image\n4.\tAs mentioned in the weaknesses, can the proposed method scale to large-scale LMMs? In addition, is the Perceiver Resampler pre-trained offline or jointly trained with the LLM in an end-to-end manner?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}