Title: Glance2Gaze: Efficient Vision-Language Models from Glance Fusion to Gaze Compression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal large language models; Vision-Language Models;
Abstract: Vision-language models heavily rely on visual representations, yet ensuring its efficiency remains a critical challenge. Most existing approaches focus on reducing visual tokens either at the visual encoder phase or during the LLM decoder stage. Inspired by human visual cognition, where an initial global glance precedes focused attention on semantically salient regions, we introduce Glance2Gaze, a cognitively inspired framework that mimics the human two-stage attention process. The framework consists of two key components: the Glance Fusion module, which integrates multi-layer vision transformer features with text-aware attention to generate a semantically enriched global representation, and the Gaze Compression module, which utilizes a novel query-guided mechanism to selectively compress visual tokens based on their semantic relevance. Experimental results on widely adopted benchmarks demonstrate that Glance2Gaze outperforms existing methods, achieving superior performance with equal or lower computational cost. Furthermore, it generalizes well to high-resolution and video scenarios, showcasing robust and scalable efficiency improvements in VLMs.


Peer reviews:
comments: SUMMARY:
This work points out that the existing approaches focus on reducing visual tokens at the visual encoder phase or the LLM decoder stage. However, this work suggests that there should be an initial global glance that precedes. This work has two key components: 1) Glance Fusion module: integrates a multi-layer vision transformer; 2) Gaze compression module: selectively compresses visual tokens based on the semantic relevance.

STRENGTHS AND WEAKNESSES:
Strength:
* The paper is well-written and clear presentation.
* The motivation is clear, presented in Figure 1. And the proposed method is based on the Motivation.
* The proposed method is clear and presented in Figure 2.
* The experiment supports the proposed methods.

Weakness:
* Provide the code in Appendix for reproduction.
* In Table 3, the Glance2Gaze  achieves better performance than naive Video-LLaVA on TGIF. However, the Galance2Gaze should achieve lower performance by employing compression.
* In Table 5, it seems that the effectiveness of the proposed fusion method gradually decrease, with the model size increase. What is the performance of the proposed fusion method with a larger model size?

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method to **compress visual tokens** in LLaVA-style multimodal LLMs (where a ViT vision encoder has its vision tokens projected to prefixed input tokens to the autoregressive LLM). The method \"Glance2Gaze\" consists of two modules:\n\nFirst, a **Glance Fusion module** computes $\\mathbf{V}_Q$ based on *both* vision tokens from multiple layers and text tokens from the instruction. The idea here is to extract information from multiple levels and based on the instruction, similar to the initial fixation stage in the human visual system.\n\nSecond, a **Gaze Compression** module replaces the full set of visual tokens with a subset that shrinks in progressive layers of the LLM transformer, to reach a certain target reduction in FLOPs. This set is computed via cross-attention using learned queries and $\\mathbf{V}_Q$ from the Glance Fusion module as keys and values.\n\nGlance2Gaze is compared to recent methods for visual compression in VLMs (FastV, SparseVLM, PDrop, VisionZip) on some common VLM and video benchmarks, reducing the FLOPs in the LLM to 33%/22%/11%, and shows a **consistent improvement**.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper introduces an **innovative** biologically-inspired method for visual token compression in LaVA-style models. The Glance Fusion module leads to a set of vision tokens that is better suited for answering a question by incorporating the question in the selection process. The Gaze Compression mechanism still achieves good results at a high token compression level.\n\n1. The method works with both images and videos and achieves a **consistent performance gain** over recent methods like VisionZip.\n\n1. The paper is **well-written** and effectively illustrated, making the somewhat complex methodology easy to understand.\n\n**Weaknesses**\n\n1. The paper does not report the overall cost of training with Glance2Gaze. The **FLOPs computations** for inference are incomplete (e.g. the vision encoder is ignored), and there is only a single measurement of inference performance (single architecture, single FLOPs reduction setting).\n\n1. **Clarity** of some of the equations and FLOPs computations **could be improved**, see details in section \"Questions\" below.\n\nQUESTIONS:\n1. In Formula (9) the symbol $\\mathbf{V}^l_Q$ appears both on the left side and the right side. I think on the right side it should instead be $\\mathbf{V}^{l-1}_Q$. I find it a bit confusing that there are \"task-enhanced visual tokens\" $\\mathbf{V}_Q$ in Equation (8) and then there is a new symbol \"visual tokens\" $\\mathbf{V}^l_Q$ that re-uses the same symbol but is different (the former being the integrated tokens from the Glance Fusion stage, and the latter being the consecutively compressed visual tokens from the Gaze Compression stage).\n\n1. When computing the FLOPs it seems that only the LLM is taken into consideration? What fraction of the overall FLOPs are due to the vision encoder? Note that in Table 4 a 20x fold of FLOPs reduction is reported, but the speedup is 6.39x.\n\n1. FLOPs computations in the appendix in Equation 2 and lines 20-27: Why would the visual token sequence compression FLOPs (first part of the formula) be included in \"gaze compression\"? I find that naming a bit confusing because the first part is simply the cost of the forward pass in the decoder. Still with the first part of the equation, why is $n$ constant throughout the decoder? I would have expected $n$ to be constant up to layer $r-1$, but then be replaced with $n_\\text{ins} + p_l$ from the gaze compression, otherwise Gaze Compression would only add FLOPs to the sum (in the second part). As for the Glance Fusion FLOPs computation (appendix lines 18-19), it seems that the projections are missing from Equation 1.\n\n1. In Figure 1, how are the tokens pruned? Are the residuals of pruned tokens still available to layers with a higher index?\n\n1. In Figure 4, is it correct that the \"Layer-specific Query\" bars include the Glance Fusion stage and the only difference to the full Glance2Gaze is that the query embeddings are not shared?\n\nSmall comments\n\n1. Line 95: \"it resizes\"\n1. Line 108: missing space between the reference list and \"Resampler\".\n1. Line 255: missing space between \"Bind\" and reference\n1. Line 279: \"but\" is a bit misleading: one would usually expect the first part of the phrase to be positive (e.g. \"reduces parameters\") and the second part to be negative (e.g. \"reduces performance\") – but in this case both changes are negative\n1. Line 637: where exactly is the training compute mentioned in the paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the efficiency of MLLMs, with one idea that to observe the image information, it is necessary to first conduct a preliminary global browsing, and then focus on semantically prominent regions. A two-stage framework, namely Glance2Gaze, is proposed. The glance fusion module integrates multi-layer vision transformer features with text-aware attention to generate semantically global representations, and the gaze compression module utilizes a query-guided mechanism to selectively compress visual tokens by semantic relevance. Experiments are conducted in a series of VQA benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n[+] The manuscript is well written, with clear logics and sufficient formulations.\n\n[+] Efficient MLLMs have great practicality for the community, thus this work is on the right path.\n\n[-] Many bad cases from the cognitive perspective. For some simple scenes, from glance to gaze is feasible. While in many complex scenes, a single image has multiple focuses, and sometimes, these focuses may overlap. Such complex scenes may lead to failure of Glance2Gaze. \n\n[-] Unclear Efficiency. In Sec 3.3, this paper shows some cases of LLaVA-1.5-7B in terms of FLOPs to prove the efficiency. However, this is not comprehensive and requires more evaluations, such as lantency and throughput. Please report more quantitative results.\n\n[-] How to deal with image-text unpatching? As one generalized model, MLLM also encounters many special inputs, e.g., the text inputs are unrelated to the image inputs. In this case, glance fusion will be misled, causing gaze compression to fail.\n\n[-] Insufficient Ablations. How to prove the effectiveness of the glance fusion module?  For gaze compression, will learnable queries be better than directly calculating the similarity of image-text?\n\nQUESTIONS:\n[-] Some Surprising Results. In Table 1, on the benchmarks of SQA, MMVet, and SEED, Glance2Gaze even outperforms LLaVA-1.5-7B, which means that compressing tokens for efficiency actually enhances performance. Please provide more discussions to explain these phenomena.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Glance2Gaze, a cognitively-inspired framework for efficient vision-language models that mimics human eye’s movement patterns. The framework consists of two components: Glance Fusion, which integrates multi-layer ViT features with text-aware attention, and Gaze Compression, which progressively reduces visual tokens inside of LLMs using a shared query pool. It achieves good performance while pruning the input tokens that could significantly reduce computation cost.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strength\n(1) Extensive evaluation on 10 image understanding and 4 video understanding benchmarks demonstrates consistent improvements, with particularly impressive results maintaining good performance at 33%,22% and 11% FLOPs.\n\n(2)The paper includes reasonable ablation studies, visualizations of the compression process, and detailed efficiency comparisons including both FLOPs and prefill latency measurements.\n\n(3)The text-aware attention mechanism in Glance Fusion and the shared query pool design in Gaze Compression are novel contributions that effectively balance efficiency and performance.\n\n \n \nPaper Weakness\n(1) The choice of ViT layers for Glance Fusion and compression sequence for Gaze Compression seems arbitrary. For Glance Fusion, it lacks discussion why choosing selected layers L={7,13,9,23} to enhance semantic information is necessary. Similarly, the paper uses exponential decay (e.g., [256, 128, 64, ..., 2]) for compression but doesn't provide in-depth analysis.\n\n(2) Section 4.2 claims Glances2Gaze achieves inference speedup up to 6.39x compared to vanilla method, whereas table 4 only provides latency comparison of prefilling time. The overall inference time comparison is untested.\n\n(3) Training of Glance2Gaze requires finetuning the whole language model, compared to the training-free method such as VisionZip, it introduces additional training costs despite performance gain.\n\n(4) While the cognitive inspiration is compelling, the paper lacks rigorous theoretical analysis explaining why Glance2Gaze could outperform previous work such as VisionZip.\n\nQUESTIONS:\n(1) How were the specific layer indices chosen for Glance Fusion? Were other layer indices tested for Glance Fusion?\n(2) Can you provide a breakdown of the computational overhead with the mentioned \"0.17%\" in section 3.3? For the computation cost of the language model, does it differ between prefill and inference?\n\nLIMITATIONS:\nHere’s one suggestion to improve the insight of the paper. The idea of Gaze Compression is impressive. Similar to table 5, adding the comparison of Gaze Compression to other methods would further improve the necessity, e.g. comparing it with token pruning using only attention score with the same compression ratio in each layer.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}