Glance2Gaze: Efficient Vision-Language Models from Glance Fusion to Gaze Compression
Authors: Juan Chen
Honglin liu
Yingying Ao
Ting Zhang
Yan Huang
Xudong Liu
Biao Li
Jintao Fang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal large language models; Vision-Language Models;
Primary area: deep_learning

Abstract: Vision-language models heavily rely on visual representations, yet ensuring its efficiency remains a critical challenge. Most existing approaches focus on reducing visual tokens either at the visual encoder phase or during the LLM decoder stage. Inspired by human visual cognition, where an initial global glance precedes focused attention on semantically salient regions, we introduce Glance2Gaze, a cognitively inspired framework that mimics the human two-stage attention process. The framework consists of two key components: the Glance Fusion module, which integrates multi-layer vision transformer features with text-aware attention to generate a semantically enriched global representation, and the Gaze Compression module, which utilizes a novel query-guided mechanism to selectively compress visual tokens based on their semantic relevance. Experimental results on widely adopted benchmarks demonstrate that Glance2Gaze outperforms existing methods, achieving superior performance with equal or lower computational cost. Furthermore, it generalizes well to high-resolution and video scenarios, showcasing robust and scalable efficiency improvements in VLMs.

Glance2Gaze: Efficient Vision-Language Models from Glance Fusion to Gaze Compression
Juan Chen 2, Honglin Liu?, Yingying Ao?, Ting Zhang Yan Huang
Xudong Liu", Biao Li Jintao Fang"
School of Computer Science and Engineering, South China University of Technology Meituan Inc. School of Artificial Intelligence, Beijing Normal University cs esjchenmmail.. scut. edu. en, tingzhang@bnu. edu. en, aihuangy@scut. edu. cn (liuhonglin03, aoyingying, 1iiglloouoounn yaowenyuan fangj (ntaoemmeiii tammeittancoom
Abstract
Vision-language models heavily rely visual representations, yet ensuring its efficiency remains critical challenge. Most existing approaches focus on reduc- ing visual tokens either at the visual encoder phase during the LLM decoder stage. Inspired by human visual cognition, where an initial global glance precedes focused attention semantically salient regions, introduce Glance2Gaze, cognitively inspired framework that mimics the human two-stage attention process. The framework consists of two key components: the Glance Fusion module, which integrates multi-layer vision transformer features with text-aware attention to gen- erate a semantically enriched global representation, and the Gaze Compression module, which utilizes a novel query-guided mechanism to selectively compress visual tokens based on their semantic relevance. Experimental results widely adopted benchmarks demonstrate that Glance2Gaze outperforms existing methods, achieving superior performance with equal or lower computational cost. Further- more, it generalizes well to high-resolution and video scenarios, showcasing robust and scalable efficiency improvements in VLMs.
Introduction
Large Vision-Languaae Models (VLMs) 66 have made significant advances image captioning. visual question answering, and multimodal dialogue understanding driven by the rapid progress of Large Language Models (LLMMs) 7-1.. typical VLM architecture consists of three key components: (i) pre-trained visual encoder responsible for extracting dense image representations; (ii) a cross-modal projector that aligns visual features with textual embeddings; and (iii) a language model backbone that processes the combined visual-textual tokens to generate task-specific outputs. Central to this pipeline are visual tokens, whose quality critically impacts model performance and efficiency, making them core focus in VLM design.
Visual tokens are conventionally extracted from images using pre-trained vision backbones such as CLIP-Vi ViT [2], EVA 13 or InternImage [14, which generate fixed-length sequences of visual embeddings for downstream multimodal tasks. Recent advancements in large VL.Ms have shown that increasing the number of visual tokens can significantly improve performance by capturing finer visual details
15] [6]. However, denser visual token representations pose substantial computational challenges due to the quadratic complexity of attention mechanisms in LLMss, where
"This work done when Juan Chen was intern Mcituan Inc. Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inage Question:
title book?
TextVQA
Accuracy
55
50-
45-
60%
Baseline
KOMERAAN SNTH Compen
40
8
12
16
20
24
28
31
Layer Index
GQA
SA G
65
KOMEED
60-
Accuracy
55
50-
45
60%
Biscline
40
8
12
16
20
24
28
31
Layer Index
Figure I: Left: Cross-modal attention heatmaps between instructions and visual tokens from different ViT layers, with numerical annotations indicating layer indices. Right: The pruning ratio-performance trade-off curve under LLa
computational cost scales with the square of the total token count. Consequently, increasing token density exacerbates inference latency, memory consumption, and scalability issues.
In response to increasing complexity, substantial research has explored visual token selection, an approach that retains only the most informative tokens while discarding redundant or less relevant ones. This strategy is driven by the insight that not all visual tokens contribute equally to downstream tasks 23 Existing methods can be broadly classified into two paradigms based on the stage at which pruning is applied. The first focuses on encoder-stage pruning (17 19 20
22 24, where token selection occurs within the visual encoder before fed into the language model. The second involves
LLM-stage pruning [88 21 23 25) where selection performed alongside the language model decoder, often leveraging the full attention matrix to assess token importance. By selecting a subset of salient tokens, both paradigms aim to reduce token count while maintaining task performance, thereby enhancing computational efficiency. This paper also focus on efficient visual representation, providing novel insight from a cognitive perspective.
In this paper, we propose a novel framework for efficient vision-language models inspired by the plance-to-gaze mechanism observed in human visual cognition. Our work draws on the human eye movements that typically exhibit a two-stage pattern: initial fixations are broad and exploratory in nature, serving rapidly acquire a global understanding of a scene, while subsequent fixations become increasingly focused (28). Intuitively, human first perform a glance to capture the global layout and salient structures, followed by a gaze that zooms in on regions of interest for detailed inspection (29. However, existing methods often aim to improve efficiency by selecting a subset of salient tokens, overlooking the hierarchical nature of human attention.
Motivated by this insight, we propose Glance2Gaze, a hierarchical visual token processing frame- work that emulates the dual-phase mechanism of rapid glancing followed by focused gazing. In contrast, our approach begins with a lightweight global sean to identify candidate regions and then progressively refines attention toward semantically rich subregions, enabling deeper cross-modal interaction. Specifically, Glance2Gaze comprises two complementary modules: Glance Fusion and Gaze Compression. Glance Fusion enables fast and holistic perception by dynamically aggregating multi-layer features rather than solely the penultimate ViT layer used in conventional approaches. We employ a text-aware attention integrate hierarchical representations, enriching global semantics without increasing token count. As shown in Figure [ (left), attention heatmaps across VIT layers indicate that each layer contributes distinctively to cross-modal understanding, highlighting the benefit of hierarchical feature aggregation for global context modeling. Building on the initial global understanding, Gaze Compression shifts the focus toward localized visual cues by progressively condensing visual tokens within the language model. This module is informed by the observation that token redundancy varies with decoding depth: shallow decoder layers are sensitive to pruning, whereas deeper layers remain robust under significant token reduction, shown in Figure [ (right).

---

We introduce a novel query seeereggided mechanism to selectively compress visual tokens based on their semantic relevance, facilitating a smooth transition from global to fine-graing see-ggraied visual reasoning.
We conduct extensive experiments to evaluate the effectiveness and efficiency of our proposed frame- work. Empirical results demonstrate that it consistently outperforms state-of-the-art baselines both image and video understanding tasks, while maintaining comparable computational efficiency on the LLaVA VA backbone series. To ensure comprchensive evaluation, we include a detailed analysis of computational cost, including inference latency comparison. Beyond empirical performance, our framework is grounded in cognitive principles drawn from human visual perception. By align- ing model behavior with established cognitive mechanisms, our approach provides a principled, biologically inspired pathway toward more efficient rision-language models.
In summary, our key contributions are:
We introduce a Glance2Gaze, a cognitively motivated two-stage visual token processing framework that mirrors human eye movement patterns, initial global glancing followed by focused gazing, to enhance efficiency in vision-language models.
Building on the insight, we design a Glance Fusion module that aggregates multi-layer VIT features using text-aware attention, enriching global semantic understanding without increasing visual token count and a Gaze Compression module, an efficient iterative visual token compression method that uses shared query pool to condense visual tokens, emulating the visual system's 's gaze process.
We present in-depth analysis about each component of the proposed framework. Empiri- cally we demonstrate our approach delivers superior performance comparing with state-of- the-art baselines both image understanding and video understanding tasks.
2 Related Work
Vision-Language models. Recent advances LLMS like GPT 7 and LLaMA (9] have driven the emergence of VLMs [3) VLMs aim to process and understand information across multiple data modalities, including text, images, videos. LLaVA [2) pioneered open-source VLMs by introducing visual-Language instruction dataset and establishing robust framework through the integration of -IT---xX-- [1] visual encoder with the Vicuna [ language model, thereby laying the groundwork for future architectures. However, it resizes each image into fixed number of visual tokens, hindering its performance in complex tasks. To enhance visual tokens, several methods [ 10 [5]
have increased the number of image tokens to enhance model detail comprehension and reduce hallucinations. LLaVA-NeXT [1] dynamically divides images into patches, allowing the visual encoder to process them independently before concatenating them form the visual representation. Meanwhile, Qwen2-VL. (5) introduces dynamic resolution to accommodate high-resolution inputs and trains the visual encoder from scratch using extensive image-text pairs, resulting in improved performance on to enhance video understanding, resulting in a substantial increase token count. While increasing visual tokens aids in managing complex scenarios, the excessive volume can significantly hinder the practical application of VLMs in real-world settings. Recently, GG-Transformer (36 adopts a glance-and-- -gaze concept to improve the modeling of long-range dependencies and local contexts within ViTs. However, its objective differs fundamentally from ours, as it focuses on enhancing unimodal representational capacity in ViT, whereas our goal is to improve efficiency in VLM.
tasks. Furthermore, video- based VLMs [34 35) extract multiple frames
Visual compression for VLMs. Recent methods have concentrated on reducing the visual token count while aiming to preserve model performance. Prior works have explored compressing visual tokens before interfacing with the LLM [17
Resampler [39] utilizes a Iearnable query embedding to compress visual tokens from the vision encoder, while MQT [38] introduces random selection of queries to adapt token count to task needs. VisionZip 2 employs a training- free pruning strategy using encoder attention to identify and remove redundant tokens, similar to LLa Prumerge 20). These methods face challenges in optimizing performance and efficiency; excessive token retention leads to redundancy, whereas aggressive compression compromises accuracy. Another type of approaches [T8 21 23 25| 27 reduce token count LLM decoders using attention matrices. F [18] proposes halving visual tokens post-second LLM layer, while Sparse 2 employs a training-free method to prune tokens based on text relevance scores. These rely on full attention
20 22
24 33
37 38
matrices, making them incompatible with FlashAttn [40]. PDrop 21 attempts to address this by
3

---

Large Language Model
Glance Fusion
kn
Caze Compression Layer
Vision Encoder
KET Layer
Instraction
Eaptr-aagetoe weight
exponential dacay
Decoder Layer
Correlattion
Laper
Gaze Compression Layer
Decoder Layer
Gaze Compression
Gaze Compression Layer
Decoder Layer
pecode oyper
Compreectio Layer
Compress
Buwker Lag
GW
Attentice
Altentio
ALkeniiion
Glance Fusion
Projector
victon Encoder
Tokerdze De3
Image
systew
Instruction
Figure 2: Diagram of the proposed Glance2Gaze framework. Glance Fusion combines hierarchical VIT features, guided by instruction, to produce enriched text-aware visual representations. Gaze Compression iteratively reduces visual tokens using a shared query embedding pool Q..
introducing a lightweight similarity calculation before attention. However, these methods miss the human-like glance-to-gaze transition, which highlights the importance of capturing a global visual context before compression a crucial aspect often neglected.
Unlike these methods, the proposed Glance2Gaze framework is inspired by cognitive principles and consists two steps: Glance and Gaze, to efficiently compress visual tokens. The Glance step enhances overall image perception by providing additional detail, while the Gaze step gradually focuses on more relevant local areas with a novel query-guided compression.
3 Method
The Glance2Gaze architecture is depicted in Figure 2 before the LLM and embeds the Gaze Compression module within it. Glance Fusion enhances text-aware visual representations by integrating ViT layers prior to token compression, while Gaze Compression iteratively reduces visual tokens starting at specific decoder layer, transitioning focus from global scenes to local regions.
Our design places the Glance Fusion module
Revisiting VLMs. In VLMs, an image input I E | x3 is divided into N discrete tokens via patch convolution and refined through K transformer layers, yielding visual tokens V E x xxd, where d denotes the dimension of visual tokens. Typically, tokens from the penultimate layer are selected for input the projector and subsequently the LLM. R denotes the total number of decoder layers in the LLM.
3.1 Glance Fusion
Building on the preceding analysis, introduce Glance Fusion enhance global context understand- ing. While prior studies 43 explored improving visual features using pre-trained visual encoders, these methods overlook the crucial interplay between textual and visual signals, thus limiting their performance gains. Dense Connector [41] divides ViT layers into groups, averages tokens within each group, and concatenates them across channels. MMFusee [2 utilizes deep ViT features as queries to retrieve missing details from shallow features. In contrast, Glance Fusion employs text-conditioned attention dynamically integrates VIT's multi-level features, enabling task-aware feature weighting without incurring significant computational overhead or added token count.

---

iisual-isttruction correlation. We first partition the VIT into distinct hierarchical stages to miti- gate computational overhead while preserving representational diversity. We strategically sample intermediate layers from VIT. Let I. = {4. l....... Ls) denote the predefined set of layer indices (e.g.,4, 1 E [1, 24] for 24-layer ViT), where
|T_ specifies the number of selected layers. From RgxxNxd we extract tokens corresponding L. yielding lightweight hierarchical representation V IRSxNxd These tokens are then projected into the LLM's embedding space via the projector Pv. yielding Vq:
the full-layer image tokens
concat(Vi,):s E. VQ= = (V(V) RSNNxd.
(1)
(2)
where d, denotes the dimension of text tokens.
Let Time denotes the input instruction tokens, we first generate its textual embedding Eis E MM using the LLM's native embedding layer, ensuring parameter-sharing consistency with linguistic processing, M equals to the number of instruction tokens. To adaptively align textual semantics with hierarchical visual features, we introduce layer-specific projection layer (P:)s= where Pa
transforms Eine into a subspace tailored for the Vq- Formally:
E ims P7 (Eins) E Xfx, 1..,...
For each layer L, the projected text embedding E is paired with its corresponding ViT feature V/ to compute cross-modal correlation score, enabling granularity-awar fusion of visual semantics. Specifically, we take Vă query and E as key to compute scaled dot-product correlation score, resulting in an attention matrix A NR.. We calculate the row-win now-wise average to form the vector Pi2: R.N, which represents the correlation of each image token in the 1,-th VIT layer with all instruction tokens. Then we concatenate all Fize to form Gu2t R SxN:
A Vi
Vd. E
ERNXM 8 1,2, ...., S,
(4)
F: = avg( 22 E ERN, , =11=122 ..,....,S. S
(5)
..2 concat S
(6)
Instruction-oriented integration. To optimally extract visual features that enhance instruction com- prehension, we apply Softmax to Gu2r along the column dimension, yielding normalized correlation scores for each visual token across S different layers. These scores act as dynamic weights to produce task-enhanced visual tokens Vq. resulting in:
Go2: = softmar (G dim = 0),
Vo= = -. e=1 Vi o G &.Nx
(8)
where O denotes clement-wise multiplication broadcasted along the token dimension. This method employs the correlation between instructions and ViT layers weights fuse their outputs, yielding task-enhanced visual features. These enhanced features, Vq. are then concatenated with text tokens for processing by the LLM.
3.2 Gaze Compression
We propose Gaze Compression, a parameter-efficient mechanism that iteratively reduces visual token count across decoder layers via shared learnable query pool. Building previous observation that shallow decoder layers are sensitive to pruning, we retain all visual tokens in the shallow decoder layers to fully understand them. Therefore, we decide to initiate the Gaze Compression from a predefined layer r. By retaining the full token count at shallow layers, we enable the LLM to thoroughly understand visual tokens and enhance overall comprehension, thus laying solid foundation for subsequent compression.
Progressive gaze compression. To achieve token reduction, we define monotonically decreasing sequence = [Pr, (Pr.Pr+1,.... with p, 2 Pr+1 2 ... 2 Pn. where pl (r S 1 S R) represents the number of compressed visual tokens at the I-th decoder layer, which is less than the number of initial
5

---

visual tokens, namely Pr < N. Starting from layer r. query-based compression attention operation compresses visual tokens before they are inputted to each decoder layer. At each 1-th layer in the I.LM, learnable query Q E RRexde is initialized, with visual tokens from 1-1-th layer H serving as key and value for cross-attention computation. The attention output yields compressed visual tokens for the decoder layer's processing. However, distinct queries across layers increase parameter count and complicate optimization for discrete visual token extraction. To mitigate this, a learnable query embedding Qs R is shared across all pertinent decoder layers, Consequently, at the 1-th layer, the first pt queries Q¹ E IRpi are drawn from Qa. and visual tokens are compressed as follows:
H = Fo(ffttmar (. (H,(H vde
(9)
where Fofa/k/v denotes four distinct linear projection layers, and PE refers to the 2D Rotary Po- sitional Embedding [10 applied to Q' for spatial information perception. The resulting output H; E IR replaces the input - yielding compressed visual tokens. By continually compress- ing beyond layer visual tokens are reduced, lowering computational demands in the decoder layers. This progressive reduction mimics the fine gazing process, focusing visual tokens on critical image regions.
3.3 Efficiency Analysis
Following prior work accounts for less than 2.72% of the total FLOPs, which therefore negligible compared to the computational cost incurred by visual token processing within LLM. We thus provide additional details in the supplementary material and here exclude it from further analysis.
we report the FLOPs of the image token component. Glance Fusion
Consequently, we focus solely on the FLOPs associated with the LLM's processing of visual tokens, evaluating efficiency from two perspectives: the FLOPs required process visual token sequences through decoder layers, and 2) the overhead introduced by Gaze Compression, as outlined in Eq. 9
For Vicuna [8] (featuring 3-layer MLP following attention), the FLOPs per layer is 4njdg - 2n de 3ndd,dm. where n, denotes the number of visual tokens (n4 = N before layer r, ny = Pr thereafter) at 1-th decoder layer and dm is the hidden dimension of FFN. The compression overhead in Eq. includes attention computation and four linear layers, resulting in 2(pd + PI- + P1p:-1d4) FLOPs, assuming Pt = N when = Then the overall computational cost is:
R
R
C=
Σ
And + 2n + 3rdd,dm + 2(pdd + +p--d + 1dP)
(10)
For instance, by setting r to 9 and using an exponentially decaying sequence P = 1256..... . LLaVA-1.5-7B, we maintain only 33.2% of the FLOPs while minimally affecting performance across several benchmarks. To manage different compression ratios, pn is always set to 2, with adjustments made and pr- Additional details are provided in the supplementary material.
2] in
4 Experiments
4.1 Image Understanding Tasks
Benchmarks. To validate the effectiveness of our method in image understanding tasks, we con- ducted experiments ten mainstream datasets, including TextVQA [44) POPE [45), GQA [46).
VQAv2 47. SEEDBench 48), MMBench [4], MME [50], ScienceQA-IMe [5T], MMVet [52) and
-LaVA-Bench-in-taee wild (2).
Implementation details. We applied the proposed method both LLaVA-1.5-7B B and the high- resolution backbone, LLaVA-NeXTIS
[15], which increases the input image resolution to 4x more pixels. For LLaVA, the vision encoder was frozen while the remaining parameters were fine-tuned using the LLLVA-665H 3 dataset, adhering to the original training settings. For LLAVA-NeXT, all parameters were unfrozen during fine-tuning. Given its proprietary code and training data, we used the Open-LLaVA-NexT (53, an open-source replication, for training, following PDrop [21.. In the

---

Table The performance of Glance2Gaze at various compression configurations on ALLVA-11.5-7B with FLOPs and the final column indicating the relative proportion of visual token computation and performance compared the original model. Bold for best and underline for second-best performers.
Method
FLOPs|POPE
SQA MME GQA SEED MMB VQAv2 TexiVQA MMVel LLaVA-B
Avg.
LLaVA-1.5-7B
100%
85.9
69.5
1862
61.9
58.6
64.7
78.5
58.2
31.1
66.8
100%
FastV
33%
64.8
67.3
1612
52.7
57.1
61.2
67.1
52.5
27.7
49.4
87.5%
LL
33%
85.3
68.7
1787
59.5
58.7
64.1
75.6
57.8
33.1
66.1
99.0%
PDrop
33%
82.3
70.2
1766
57.1
54.7
63.2
56.1
30.5
96.2%
VisionZip
33%
84.9
68.2
1834
60.1
57.1
63.4
77.4
57.8
32.6
66.7
99.1%
Glance2Gaze
33%
85.5
70.4
1812
61.5
58.7
64.5
77.6
57.2
32.7
66.4
99.9%
FastV
22%
59.6
60.2
1490
49.6
$5.9
61.8
50.6
28.1
52
83.2%
VL.
22%
85.0
68.6
1746
58.4
58.2
64.5
73.8
56.7
29.0
62.7
96.3%
PDrop
22%
82.3
69.9
1664
56.0
53.3
61.1
55.1
30.8
94.4%
VisionZip
22%
83.7
68.3
1823
58.9
55.8
62.6
76.6
57.0
32.9
64.8
97.9%
Glance2Gaze
22%
70.2 84.5
1794
59.3
56.6
63.4
77.4
56.8
33.1
65.4
98.7% %
FastV
11 %
48.0
51.1
1256
46.1
51.9
48.0
55.0
47.8
25.8
46.1
73.8%
Sparse VLM
11%
77.5
69.8
1589
53.8
52.2
60.1
68.2
53.4
24.9
57.5
89.0%
PDrop 11%
55.9
69.2
1092
41.9
40.0
33.3
45.9
30.7
73.5%
VisionZip
11%
80.9
68.8
1756
57.0
53.4
61.5
74.2
56.0
30.2
63.6
94.9%
Glance2Gaze
11%
83.1
69.1
1722
56.9
53.9
61.7
74.9
55.7
31.4
64.1
95.6% %
3
324
255
0: wha year VOS this clOCk MadE?
A: 1755
Q: what brand the renote control?
A: KICKER
ground?
Figure 3: Visualization of different query embedding counts for compressing visual tokens from Tex 44). Number within each image denotes pt.
Glance Fusion module, L is set to (7, 13, 19, 23). We implement the Gaze Compression strategy under different compression ratios. All experiments conducted 8 NVIDIA-A100-SOG GPUs. Please refer to the supplementary material for more implementation details.
Results on LLaVA. Table I presents the performance of Glance2Gaze on LLaVA-1.5-7B B
benchmarked against state-of-the th-the-art approaches FastV [18] Sparse VLM 23 PDrop (21), and VisionZip [22] across three FLOPs configurations, 33%. 22% and 11% indicating the computational ratio of visual processing compared to full token retention. Overall, Glance2Gaze consistently achieves top results at all compression levels and ranks as the best second-best performer across nearly all datasets, showcasing its strong competitiveness. At 33% FLOPs, Glance2Gaze outperforms PDrop by 3.7% and VisionZip by 0.8%. Moreover, at 22% FL.OSs, Sparse experience 2.7% drop in accuracy, whereas Glance2Gaze only loses 1.2%. surpassing VisionZip by 0.8%. At the extreme compression ratio of 1 11%, Glance2Gaze retains 95.6% performance, exceeding VisionZip by 0.7% and SparseVLM by 6.6%, demonstrating its robust performance across various compression requirements.
Results on LLaVA-NEXT. LLaVA-NeXT extends LLaVA to handle high figh-resolution scenarios by dividing images into up to five patches, creating up to 2880 image tokens. Glance Fusion and Gaze Compression are applied independently to each sub-image. As shown in Table Glance2Gaze ex- hibits greater advantages in high-resolution scenarios compared to standard resolutions. Glance2Gaze surpasses the second-best performer, VisionZip, with an average improvement of 1.2-1.3% across three FLOPs setting. Furthermore, Glance2Gaze surpasses VisionZip on nearly all datasets, demon-
7

---

Table The performance of Glance2Gaze on LLaVA-NeXT:S
Method
FLOPs
TextVQa
POPE
SQA
MME
GQA
MMB
VQAv2
Avg.
LLaVA-NeXT-7B
100%
65.8
86.6
69.2
1801
63.8
67.2
80.5
100%
Sparse VLM
22%
57.8
67.7
1772
60.3
65.7
77.1
95.4%
VisionZip
22%
60.8
87.6
67.9
1778
62.4
65.9
79.9
97.9%
Glance2Gaze
22%
62.1
89.2
69.9
1784
63.1
66.4
80.1
99.2%
SL
11%
55.9
67.3
1694
57.7
64.3
73.4
92.3%
VisionZip
11%
59.3
86.2
67.5
1770
61.0
64.4
78.4
96.3%
Glanee2Gaze 11%
60.5
87.5
69.9
1771
62.0
64.3
78.8
97.6%
SLM 5%
46.4
67.5
1542
51.2
63.1
66.3
85.0%
VisionZip 5%
57.3
83.4
67.5
1699
58.2
63.9
75.6
93.6%
Glance2Gaze
5%
58.6
84.4
68.8
1718
58.9
64.0
76.4
94.8%
Table 3: The performance Glance2Gaze on Video-LLaVA at 6% FLOPs.
Method
TGIF MSVD MSRVTT Activity Net
Avg.
Video-LLaVA
47.1
69.8
56.7
43.1
100%
FastV
23.1
38.0
19.3
30.6
52.1%
SparseVLM VLM
44.7
68.2
31.0
42.6
86.5%
VisionZip
42.4
63.5
52.1
43.0
93.2%
Glance2Gaze
45.0
66.5
52.4
43.8
96.2%
strating its robustness in preserving essential information during compression, especially in intricate OCR tasks such TextVQA and VQAv2. The superior performance of Glance2Gaze high- resolution scenarios likely stems from its ability to focus on spatially crucial details through the glance-to-gaze process, which becomes increasingly important as image complexity escalates.
4.2 Video Understanding Tasks
Benchmarks. Beyond image understanding, we applied the proposed method to video comprehension tasks, evaluating on four widely adopted video-based question answering benchmarks: TGIF [54] MSVD [55), MSRVTT (55) and ActivityNet [56].
34 train video VLM, which use Language Bind [57] as the vision encoder, extracts 8 frames from a video, encoding each into 256 visual tokens, culminating 2048 image tokens in the end. To make fair comparison with VisionZip and Sparse and pr are set to 3 and 10 respectively create a model with FLOPs reduced to 6% of the original. Please refer to supplementary material for more training details.
Implementation details. We use Video-LLaVA VA
Performance on Vidco-LLaVa VA. In video understanding tasks, a substantial degree of frame-to- frame redundancy presents opportunities for computational efficiency without significantly compromis- ing performance. Our method effectively capitalizes this redundancy, achieving 96.2% of the performance using only 6% of FLOPs. whereas FastV suffers a 47.9% performance drop. Notably, Glance2Gaze outperforms VisionZip by 3% on average and ranks first on 3 out of 4 benchmarks, making it particularly suitable for real-time and large-scale video analysis applications.
4.3 Efficiency Comparison
Table 4 analyyes the computational costs, cuda time (i.e., the time required to generate the first token), overall latency (i.e., the total time to complete the entire sequence), and throughput on the TextVQA dataset, evaluated using NVIDIA-A1001 -00--- GPU, consistent across all compared baselines.
The results demonstrate that, under the same TFLOPs reduction ratio, Glance2Gaze not only delivers superior performance but also achieves the fastest inference speed, with 6.39x acceleration in pre- filling time, a 15x speedup in overall latency, and a 3.1 3.14x improvement throughput, effectively balancing efficiency and accuracy.

---

Table 4: Computational cost analysis comparison on LLaVA-NeXT-7.
Method
TFLOPs Cuda Time (ms) Latency (ms)
Throughput
TextVQa
LLAVA-NeXT
53.83
313
475
10.884
65.8
FastV
2.76
170 (L.8x)
358 (1.33x) 12.272 (1.13x)
Sparse 2.76
183 (1.71x)
379 (1.25 x) 13.233 (1,22x)
46.4
VisionZip
2.76
57 (5.49x)
192 (2.47x) 26.983 (2.48x)
57.3
Glance2Gaze
2.67
49 (6.39x)
151 (3.15x) 34.121 (3.14x)
58.6
Layer-speeifie Query
Gaze Compression only
Glanec2Gaze
SEED
GQA
POPE
SQA
MMB
LLaVA-B
VQAv2
TexiVQA
83.2 84.0 84.5
Accuracy
68.4 69.2 70.2
70.4
65.0
62.4
63.4
58.7
59.3
60.0
59.7
55.8
56.6
55.8
56.8
53.3
50.7
Figure 4: Ablation study evaluating the effect of shared query pool in Gaze Compression comparing layer-specific query and the effect of Glance Fusion.
4.4 Analysis and Discussion
Visualization of Gaze Compression. Figure 3 illlustrates the visual features learned by the query embedding at 33% FLOPs on TexiVQA. Initially, with more visual tokens retained, the query captures broader global information. As compression progresses and fewer visual tokens are kept, focus shifts to regions containing the answer, demonstrating how the query learns to localize important areas through the gaze compression procedure, such as those involving answers.
The effect of Gaze Compression compared to attention-based strategies. To rigorously assess the attention- based pruning, evaluate both methods under
advantage of Gaze Compression identical compression ratio. As shown in Table attention- based pruning results notable perfor- mance degradation. Even with subsequent fine-tuning, the baseline remains inferior Glance2Gaze, underscoring that static pruning guided solely by local attention scores lacks the adaptability required for task-specific compression.
The effect of a shared query pool. We investigate the effects of using layer-specific queries in Gaze Compression, shown in Figure Using independent queries for each layer increases parameters and significantly damages performance across all datasets. This may be because the shared queries enhance pattern extraction from the same image, implicitly constraining the query's 's optimization domain. More ablations about Gaze Compression is available in the supplementary material.
The effect of Glance Fusion. Figure 4 demonstrates the impact of applying Gaze Compression alone and with Glance Fusion at 22% FLOPs on V...5--..-7B, resulting in substantial performance enhancements across all datasets. Notably, Glance Fusion improves accuracy by 1.0 and 1.6 points on TextVQA and VQAv2, respectively. where fine-grained image details are crucial for text understand- ing tasks. We emphasize that the additional computational overhead introduced by Glance Fusion is minimal, yet its effectiveness is substantial. More ablations about Glance Fusion is available in the supplementary material.
Comparison with other fusion strategies. We compare Glance Fusion with two representative fusion strategies, Dense Connector [4]] and MMFuser 42 on LLaVA [ with different scales, as summarized Table 6 Dense Connector was not evaluated on LLa -A-55--.3 and therefore excluded from that setting. Glance Fusion consistently outperforms both baselines across model scales. On LLaVA-1,5-7B, it exceeds Dense Connector by 0.1-1.1 points, despite using fewer ViT layers (4 vs. 24), and surpasses MMFuser, which uses five layers, by 0.5-1.9 points. Similar

---

Table Comparison of Gaze Compression with attention- based pruning methods.
Method
POPE
SQA
MME GQA
SEED
MMB
VQAv2
TexiVQA
MMVei LLaVA-B
Avg.
w/o finetune
83.4
69.1
1775
59.7
54.7
63.0
74.6
55.8
30.6
62.0
96. 16.1%
w finetune
84.7
69.1
1793
60.8
56.8
63.2
75.7
56.4
31.4
65.2
97.9%
Glance2Gaze
85.5
70.4
1812
61.5
58.7
64.5
77.6
57.2
32.7
66.4
99.9%
Table Exploring the potential of Glance Fusion compared with other fusion strategies.
Method
GQA
VQAv2
SQA
TextVQA
POPE
LLaVA-7B
62.0
78.5
66.8
58.2
85.9
+Dense Connector
63.81.81
79.5
69.52.71
59.21.01
86.6
+MMFuser
62.80.87
79.1 79.1
68.71.90
58.8
86.39.47
Glannee Fusion
64.1 64.1a.17
79.6
70.6*.8*
59.4
87.2
I.L.VA-1BB
63.3
80.0
71.6
61.3
85.9
+MMFuser
63.40.11
80.1 80.1
71.2
59,90.44
87.5
+Glance Fusion
64.5
80.4
71.90.37
87.2 87.2
improvements are observed on L.LVA-1.5-13B. These results highlight Glance Fusion's 's superiority over task-agnostic fusion methods.
5 Conclusion and Limitation
This paper introduces novel cognitive-inspired visual token compression through two-stage glance-to-gazz approach, First, Glance Fusion dynamically merges multi-layer ViT features using text-aware attention strategy to enhance image understanding. Next, Gaze Compression employs a query-based approach using a shared query pool to selectively compress visual tokens within LLM, mimicking the gaze process by progressively focusing on detailed local regions. Experimental results show that Glance2Gaze surpasses existing methods in performance with equal or reduced computational cost. While the overall framework is cognitively inspired, the internal mechanisms, particularly the fusion and compression strategies, remain largely heuristic and lack explicit modeling of cognitive processes. Future work could explore tighter integration with cognitive theories to further enhance both interpretability and performance.
10

---

Acknowledgments
This work was supported by the National Key Research and Development Program of China un- der Grant 2024YFE0105400, the Guangdong Provincial Key Field R&D Program under Grant 2024B0101 and the Guangzhou Science and Technology Plan Project - Key R&D Plan under Grant 2024B01 W0007. This work was jointly supported by the Fundamental Research Funds for the Central Universities under Grant 2243 2243100004 and 2253500001.
References
[1] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang- Xizhou Zhu, Lewei Lu, et al. Intervvl: Scaling up vision foundation models and aligning for generic lisual-linguistic tasks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185-24198, 2024.
[2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing systems, 36:34892-344 -4916, 2023.
[3] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22396, 2024.
[4] Gemini Team, Rohan Anil, Sebastian Borgeaud, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint urXiv:2312. 18805 2023.
[5] Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing aision-languaee model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[6] Deyao Zhu, Jun Chen, Xiaoqian Shen. Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:230.110592, 2023.
[7] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXin:2303.08774, 2023.
[8] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 with 90% * chatgpt quality, march 2023. URL htps://tmsss org/hlog/2023--3-390-
-3-30v-uuaa 3(5), 2023.
[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[10] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:24/2. 151 15, 2024.
[11] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 220i:2200500068. 2022.
[12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PmLR, 2021.
[13] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 233.v 15389, 2023.
[14] Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li. et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14419, 2023.
11

---

[15] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[16] Pan Zhang. Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang. Zhiyuan Zhao, Haodong Duan, Songyang Zhang. Shuangrui Ding. et al. Internim Intemin-xcomposer: vision- languaee large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 15/ 2023.
[17] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprini arXiv:2210.09461, 2022.
[18] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Boug-and-play inference acceleration for large vision-languee models. In European Conference on Computer Vision, pages 19-35. Springer, 2024.
[19] Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.1000, 2024.
[20] Yuzhang Shang. Mu Cai, Bingxin Xu. Yong Jae Lee, and Yan Yan. plava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXhvv2403.35330 2024.
[21] Long Xing. Qidong Huang. Xiaoyi Dong. Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint vrXiv:2410.1224: 2024.
[22] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang. Jingyao Li. Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04400 2024.
[23] Yuan Zhang, Chun-Kan Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang. Kuan Cheng. Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:24 10Xiv:2410.04417, 2024.
[24] Shaolei Zhang, Qingkai Fang. Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.0388 2025.
[25] Jieneng Chen, Luoxin Ye. Ju He, Zhao- Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi-modal models via visual context compression. In The Tiiht Annual Conference on Neural Information Processing Systems, 2024.
[26] Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. Dynamie-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876 2024.
[27] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-frce visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, number 21, pages 22128-22136 2025.
[28] Alfred I. Yarbus. Eye movements and vision. Springer, 2013.
[29] Binh X Nguyen. Tuong Do, Huy Tran, Erman Tjiputra, Quang D Tran, and Anh Nguyen. Coarse-to-fine reasoning for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1558-4566, 2022.
[30] EAB Over, ITC Hooge, BNS Vlaskamp, and CI17617434 Erkelens. Coarse-to-fine eye movement strategy in visual search. Vision Research, 47(17)2272-22880 2007.
[31] Kirsten Petras, Sanne Ten Oever, Christianne Jacobs, and Valerie Goffaux. Coarse-to-fine information integration in human vision. Neurolmage, 186:103-112, 2019.
[32] Watt. Scanning from coarse to fine spatial scales in the human visual system after the onset of stimulus. Journal of the Optical Society of America A, 4( 4(10):2006-202) 1987.
[33] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19740-19742. PMLR, 2023.
[34] Bin Lin, Bin Zhu Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arX11:2311.10122, 2023.
12

---

[35] Bin Lin. Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning, et al. Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 15947, 2024.
[36] Qihang Yu, Yingda Xia, Yutong Bai, Yongyi Lu, Alan I. Yuille, and Wei Shen. Glance-and-gaze vision transformer. Advances in Neural Information Processing Systems, 34: 14:92292-000 13003, 2021.
[37] Wentong Li, Yugian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal Ilm. arXiv preprint urXiv:2407.0239p 2024.
[38] Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng. and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37:50168-50188. 2024.
[39] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo; a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23783 2022.
[40] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
[41] Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. Advances in Neural Information Processing Syxtems, 37:33108-33140, 2024.
[42] Yue Cao, Yangzhou Liu, Zhe Chen, Guangchen Shi, Wenhai Wang, Danhuai Zhao, and Tong Lu. Mmfuser: Multimodal multi-layer feature fuser for fine-grained vision-language understanding. arXiv preprint arXiv:2410.1 1829, 2024.
[43] Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nic. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 26540-26550, 2024.
[44] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[45] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ri-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[46] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 6700-6709, 2019.
[47] Yash Goyal, Tejas Khot, Douglas Summers- Staye Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 0914-691,2222 2017. 2017.
[48] Bohao Li, Rui Wang. Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench Benchmarking multimodal lims with generative comprehension. arXiv preprint 2307. 16/25, 2023.
[49] Yuan Liu, Haodong Duan. Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[50] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu. and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[51] Pan Lu, Swaroop Mishra. Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances Neural Information Processing Systems, 35:2507-25 -2511, 2022.
[52] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308, 02490, 2023,
13

---

[53] Lin Chen and Long Xing. Open-llava-next: An o nexv series for facilitat- ing the large multi-modal model community, http:////wtaataaoooo//////aaaaaa NeXT 2024.
[54] Yunseok Jang. Yale Song. Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. Proceedings of IEEE conference on computer vision and pattern recognition, pages 7758-2766, 2017.
[55] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. Proceedings of the 25th ACM international conference on Multimedia, pages
2017.
[56] Fabian Caba Heilbron, Victor Escorcia, Bemard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale vidco benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961-979199 961-970, 2015,
[57] Bin Zhu, Bin Lin, Munan Ning, Yang Yan. Jiaxi Cui, HongFa Wang. Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, al. Languagehind: Extending video-langue Iidoo-language pretraining to n-modality by language- based semantic alignment. arXiv preprint arXiv:23/0.006: 2023.
[58] Ruipu Luo, Ziwang Zhao, Min Yang. Junwei Dong. Da Li, Pengeheng Lu, Tao Wang. Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley; Video assistant with large language model enhanced ability, arXiv preprint arXiv:2306.008 07207, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims are consistent with the contributions and scope detailed in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical claim.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The experimental setup and datasets are detailed in Section 4 while the
proposed architecture clearly outlined in Section The supplementary material includes comprehensive training parameters and recipes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data we used open benchmarks which can be downloaded from their websites. Our code will be released upon paper's 's acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ Soaeleese/CooeSSSmmmsisiooooee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//ddeeeSbmimaia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed experimental settings are provided in Section 4 and supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

http://wwrrrasccoooooil///c/taaa The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper specifies the hardware used (8 NVIDIA-A A100-8000000 GPUs) in Section 4 FLOPs and inference latency per test samples is provided, allowing for estimation of computational costs.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Both positive and negative societal impacts discussed in supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release high-risk data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites the sources for existing methods and benchmarks and respect their CC-BY-SA Licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper describes the usage of Vicuna g as foundational component of the core method in this research, acknowledging its importance and originality. The authors have obtained the necessary authorization to use this model, ensuring compliance with relevant guidelines and respecting intellectual property rights. Hence, the use of Vicuna is integral to the methodology presented and is well-documented within the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (bttps://neur oatp:/uppp...o/ooooeeeemee////// for what should or should be described.
21

---

A Supplementary Material
A.1 Broader Impact
The proposed Glance2Gize advances vision-language models by introducing a cognitively inspired framework that harmonizes efficiency and performance through its two-stage attention mechanism. By mimicking human visual cognition- ffrst capturing global context and then focusing salient details-the method significantly reduces computational overhead while maintaining or improving accuracy across diverse tasks. This innovation broadens accessibility to advanced visual- visual-language systems, enabling deployment in resource-limited settings such edge devices real-time appli- cations. Its scalability to high-resolution and video imputs further extends practical utility in fields like medical imaging, autonomous systems, and multimedia analysis. Environmentally. the reduced computational demand aligns with sustainable AI development by lowering energy consumption. As a generalizable paradigm, Glance2Gaze also inspires future research biologically inspired attention mechanisms, fostering interdisciplinary advancements in efficient multimodal learning.
A.2 Efficiency Analysis
typically measure efficiency by the computational cost of visual tokens in LLMs, a method we also adopt. Additionally, we analyze the extra computational load introduced
Previous works [8 21 23]
by Glance Fusion.
FLOPs within Glance Fusion. Considering the computational cost incurred by Glance Fusion, account for the FLOPs arising from all projection operations of both visual (Eq. 2 and textual
(Eq. 3 inputs, as well as the overhead introduced by the visual-text correlation computation (Eq, 4 Specifically, the visual mapping introduces SN d,,d, FLOPs, while the text mapping contributes SMd, d FLOPs. Furthermore, the visual-text correlation involves S matrix multiplications, resulting in SN Md, FLOPs. Overall, the total FLOPs within the Glance Fusion module can be expressed as:
Cylance SNd.d + SMd, + SN
(11)
FLOPs within LLM (Visual Tokens). We assess FLOPs of visual tokens within the LLM by examining two factors: (i) the standard forward pass through the decoder, and (ii) the additional cost introduced by the Gaze Compression module. For the visual token processing in Vicuna-7B (8) using a 3-layer MLP as FFN and 32 decoder layers, each layer incurs 4nddf + 2nf + 3n FLOPs, where ny and d, denote the number of visual tokens and the hidden dimension of the FFN, respectively. In our framework, LLM layer index. The FLOPs from Eq. incclude 2 matrix multiplications and linear projections, totaling 2p:- IP + 2p--1d + 2p Therefore, the total FLOPs within LLM are:
is 576 before layer and pj thereafter, with I representing the
R
R
Cum = Σ And + =22 de + 3rdddmm + Σ 2(pi- + Pi + pid)). 1=r
(12)
We found Cgtance to be negligible compared to Cum- For instance, in LLaVA-1.5-7B. with r set to 3 and pr 100, Cum amounts to 1.IT FLOOPs while Cglance is only 0.03T FLOPs, yielding proportion of 0.0272. Given its insignificance, C Cglance can be omitted from detailed computational analysis, allowing us concentrate on the more impactful Cum for performance assessment.
A.3
More Implementation Details
A.3.1 Image Understanding Tasks
Training Recipes. Following LLaVA (3), we employ a two-stage training strategy for Glance2Gaze. In the first stage, we align image-text pairs by retaining the LLaVA architecture and training only the projector on the LLaVA-558K dataset for one epoch with a batch size of 256 and a learning rate of le-3. In the second stage, we incorporate Glance Fusion and Gaze Compression into LLaVA, training all parameters except the visual encoder for one epoch with a batch size of 128 and learning rate of 2e-5.
To train LLAVA-NeXT-7B 15], we employ Open-LLaaVe VeX- 533, an open-source replication, due to proprictary restrictions the original code and training sets. Our approach involves two-stage
22

---

Table 7: Compression configurations employed LLaVA-1.5-7B.
FLOPs
P
[256, 209, 170, 139, 114, 93, 76, 62, 50, 41, 33, 27, 22, 15, 12, 1,,36,63333
33%
[121, 103, 88, 75, 64, 29, 24.
22%
21, 18, 15, 13, 1,,
[100, 87, 77, 67, 59, 52, 45, 40, 35, 30,
11%
23, 20, 18, 16, 14, 12, 10, 9, 8. 7, 3...9.77.6,,,,4433333
Table 8: Compression configurations employed to LLa
FLOPs
P
[121, 103, 88, 75, 64, 54, 46, 40, 29, 24, 21, 18, 15, 13, 11,9, 5,
22%
[100, 77, 59, 45, 40, 35, 30, 27, 23, 20, 18, 16, 14, 12, 10, 8, 6,,,,,,,,,,
11%
[25, 23, 21, 19, 18, 16, 15, 14, 13, 12, 11,
5%
2
10, 9, 7, 7, 6, 6, 5, 5, 4. 4, 3, 3, 2,
training process. Similar to LLaVA-1.5-7BB the proposed Glance2Gaze is integrated only in the second stage. In the first stage, we focus solely on training the projector with a batch size of 256 and a learning rate of le-3 for epoch using the LLaVA-558K dataset. In the second stage, Glance2Gaze embedded into the ILLVV--NeT architecture, with all parameters including the visual encoder unfrozen for training over two epochs with batch size of 128 and a learning rate of 2e-5, utilizing the same dataset as in [53]
Implementation Details. In Glance Fusion, I is consistently detailed features. For Gaze Compression, we adjust r and pr to manage different computation configurations, detailed in Table
{7, 13, 19, 23) capture fine-
LLaVA-NeXX processes high high-resolution images by dividing them into multiple sub-images (up to S) for individual handling by the visual encoder, which are subsequently concatenated within the LLM. Both Glance Fusion and Gaze Compression are applied to each sub-image independently.
A.3.2 Video Understanding Tasks
Training Recipes. Following Video-LLaVA 33), we employ a two-stage training strategy. In the first stage, only the projector is trained for one epoch with batch size of 256 and learning rate of le-3, using the LLLVV--55K dataset and subset of Valley [58], while retaining the original architecture. In the second stage, we integrate Glance Fusion and Gaze Compression into Vieo--LLVV,, freeze the vision encoder, and finetune all other parameters with batch size of 128 and a learning rate of 2e-5, using the LLaVA-665K dataset and 100k video-text instructions from Video-ChatGPT [34).
Implementation Details. In Video-I.LaVA 34 frames sampled from each video, with each frame encoded into 256 tokens, resulting in 2048 tokens per video. For Glance Fusion, T. is set to {7, 13, 19, 23) by default. Gaze Compression is applied each frame, with r and p. 10, yielding model that operates at 6% of the FLOPs required for full token retention.
A.3.3 More Ablation Studies
More ablation on the effect of a shared query pool. Figure presents additional ablation results comparing layer-specific query embeddings with shared query pool in Gaze Compression. While the layer-specific approach increases parameters, it significantly degrades performance across all datasets.
More ablation on the effect of Glance Fusion. assess the importance of global glance fusion prior to gaze compression, compare Gaze Compression alone with the full Glance2Gaze in
23

---

Layer-specific Query
Gaze Compression only
Glance2Gaze
SEED
SQA
POPE
VQAv2
50.7
563
688
$4.0
84.5
7014
LLaVA-B
17686
1563
27.9
Figure 5: More ablation results on significance of shared query pool and the effect of Glance Fusion.
Table 9: Ablation studies and pr. evaluated on VQAv2 dataset.
p,=256
r=9
Method
r=1
r=3
r=9
r=17
r=25
p,=32
p,=128
p,=256
p,=529
FL.OPs
12%
17%
33%
56%
78%
24%
28%
33%
42%
Acc
70.2
74.6
77.6
78.1
78.4
74.7
75.4
77.6
77.9
Figure 5 Adding Glance Fusion before Gaze Compression significantly improves performance across all datasets, with notable gains on fine-grained OCR tasks like TextVQA and VQAv2.
Ablation study on r and p.. In Gaze Compression, the initial compression layer index r and the size of query embedding pr together shape the compression ratio. To evaluate the performance impact of each variable independently, we varied one while holding the other constant, facilitating a comparative analysis of their effects, as shown in Table reveals a direct proportional relationship between accuracy and the starting compression layer when is constant; larger corresponds to higher accuracy. This is logical, as premature compression may prevent the LLM from fully processing visual tokens. Similarly, opting for a larger query embedding size yields benefits, as it allows richer information capture during compression.
Exploration on number layers in Glance Fusion. We fuse different numbers of ViT layers, with results displayed in Figure 6 (a). Interestingly, adding more VIT layers does not always result in proportional performance gains. This aligns with observations in Figure 1, suggesting significant redundancy across layers and certain intermediate layers in VIT not contributing to instruction comprehension. To balance performance across all datasets, we selected layers for fusion in this study.
Necessity of Instruction in Glanee Fusion. To demonstrate the significance of instruction Glance Fuse, we compared it with straightforward feature averaging across layers. As illustrated in Figure (b), indiscriminate fusion of ViT layers results minimal improvement, whereas instruction-guidee fusion significantly enhances features and boosts performance.
A.3.4 More efficiency comparison
We provide comprehensive tables summarizing the training cost, inference latency, prefilling time, and throughput across different frameworks and compression configurations, offering clear overview of the computational efficiency, as shown Table 10
24

---

75
70
11AVA-15-78
wil
iteeiicalol
Accuracy
Accuracy
65
TextVQA
SQA
60
MMB
17
Number of VIT Layers
50
GQA
SQA
MMB
(a) Exploration on number of VIT layers
(b) Importance of Instraction-oriented integration
Figure Ablation studies on Glance Fusion module.
Table 10: Comparison of computational efficiency between Glance2Gaze and baseline models.
Metrics include training GPU hours, inference TFLOPs, latency, CUDA time, and throughput,
Method
Training GPU Hours
Inference TFLOPs
Latency CUDA Time (ms)
Throughput
115VA-1.5-8 family
LLaVA-1.5-BB
104
10.07
185.9
115
28.754
Glanee2Gaze (33% FLOPS)
3.36
133.2
40.022
Glance2Gaze (22% FLOPS)
60
2.22
109.2
32
51.351
Glance2Gaze (11% FLOPP)
LLaVA-Next-7B famify
LLaVA-Nex-7BB
366
53.83
475,2
313
10,884
Glance2Gaze (22% FLOPs)
242
11.84
262.0
19.741
Glange2Gaze (11% FLOPs)
5.53
191.9
30.719
Glance2Gaze (5% FLOPS)
87
2.67
151.2
49
34.121
Video-LLaVA fomill
Video-LLaVA
297
37.38
739.6
342.2
21.599
Glance2Gare (6% FI.OPs)
151
2.24
484.3
125.1
35.095
25


Reviews:
[{"review_id": "D4rYGMLGwb", "reviewer": "Reviewer_kUtD", "summary": "This paper introduces Glance2Gaze, a cognitively-inspired framework for efficient vision-language models that mimics human eye’s movement patterns. The framework consists of two components: Glance Fusion, which integrates multi-layer ViT features with text-aware attention, and Gaze Compression, which progressively reduces visual tokens inside of LLMs using a shared query pool. It achieves good performance while pruning the input tokens that could significantly reduce computation cost.", "questions": "(1) How were the specific layer indices chosen for Glance Fusion? Were other layer indices tested for Glance Fusion?\n(2) Can you provide a breakdown of the computational overhead with the mentioned \"0.17%\" in section 3.3? For the computation cost of the language model, does it differ between prefill and inference?", "limitations": "Here’s one suggestion to improve the insight of the paper. The idea of Gaze Compression is impressive. Similar to table 5, adding the comparison of Gaze Compression to other methods would further improve the necessity, e.g. comparing it with token pruning using only attention score with the same compression ratio in each layer.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Paper Strength\n(1) Extensive evaluation on 10 image understanding and 4 video understanding benchmarks demonstrates consistent improvements, with particularly impressive results maintaining good performance at 33%,22% and 11% FLOPs.\n\n(2)The paper includes reasonable ablation studies, visualizations of the compression process, and detailed efficiency comparisons including both FLOPs and prefill latency measurements.\n\n(3)The text-aware attention mechanism in Glance Fusion and the shared query pool design in Gaze Compression are novel contributions that effectively balance efficiency and performance.\n\n \n \nPaper Weakness\n(1) The choice of ViT layers for Glance Fusion and compression sequence for Gaze Compression seems arbitrary. For Glance Fusion, it lacks discussion why choosing selected layers L={7,13,9,23} to enhance semantic information is necessary. Similarly, the paper uses exponential decay (e.g., [256, 128, 64, ..., 2]) for compression but doesn't provide in-depth analysis.\n\n(2) Section 4.2 claims Glances2Gaze achieves inference speedup up to 6.39x compared to vanilla method, whereas table 4 only provides latency comparison of prefilling time. The overall inference time comparison is untested.\n\n(3) Training of Glance2Gaze requires finetuning the whole language model, compared to the training-free method such as VisionZip, it introduces additional training costs despite performance gain.\n\n(4) While the cognitive inspiration is compelling, the paper lacks rigorous theoretical analysis explaining why Glance2Gaze could outperform previous work such as VisionZip.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\nHere we address the points you mentioned in the weaknesses part.\n\n 1. We appreciate the reviewer’s attention to the choice of ViT layers and compression sequence.\n \n    1).  Our selection of layers {7,13,19,23} was guided by both visual insights and empirical evidence. As illustrated in Figure 1 (left) in the main paper, while ViT layers exhibit varied responses to instruction tokens, adjacent layers yield highly correlated representations. For empirical evidence, we computed average attentional responses of text tokens across 24 ViT layers using the TextVQA and ScienceQA datasets, resulting in the following normalized distribution:\n \n    | Layer Range | Values                                                     \n    |-------------|------------------------------------------------------------------------|\n    | 1–7         | 0.0393, 0.0393, 0.0395, 0.0403, 0.0405, 0.0416, 0.0442                  |\n    | 8–17        | 0.0400, 0.0401, 0.0398, 0.0398, 0.0397, 0.0408, 0.0394, 0.0392, 0.0395, 0.0395 |\n    | 18–20       | 0.0406, 0.0432, 0.0412                                                 |\n    | 21–23       | 0.0627, 0.0634, 0.0658                                                 |\n\n    Within each range, we selected the layer exhibiting the highest response correlation to instruction tokens, yielding the set {7,13,19,23}. For further ablation, we tested the second-best layers in each range, {6,9,20,22}. As shown in the table below, both sets significantly outperformed the baseline, but the original selection gave marginally higher gains.\n\n    | Method                | GQA         | VQA v2      | SQA         | TextVQA     | POPE        |\n    |:-----------------------|:-------------|:-------------|:-------------|-------------|:-------------|\n    | LLaVA-1.5-7B          | 62.0        | 78.5        | 66.8        | 58.2        | 85.9        |\n    | + Glance Fusion \\([7, 13, 19, 23]\\) | 64.1 (+2.1) | 79.6 (+1.1) | 70.6 (+3.8) | 59.4 (+1.2) | 87.2 (+1.3) |\n    | + Glance Fusion \\([6, 9, 20, 22]\\)  | 64.1 (+2.1) | 79.4 (+0.9) | 70.5 (+3.7) | 59.4 (+1.2) | 86.9 (+1.0) |\n\n    2). An exponential decay schedule (the function is ${\\frac 2 {p_r}}^{1/(R-r)}$, specific number for each layer are given in Table 1 in the supplementary) was adopted in Gaze Compression to enable stable and progressive compression of visual information. We here additionally evaluated two variant schedules: fast-to-slow decay and slow-to-fast decay. Both resulted in performance degradation, with the fast-to-slow strategy yielding the most significant drop. We hypothesize that aggressive early compression leads to substantial information loss, causing irreversible harm. Results are presented in the table below.\n\n    | Strategy               | POPE  | SQA   | MME  | GQA  | SEED | MMB  | VQAv2 | TextVQA | MMVet | LLaVA-B | Avg. |\n    |:-----------------------|:-----:|:-----:|:----:|:----:|:----:|:----:|:-----:|:-------:|:-----:|:-------:| :-------:|\n    | Exponential Decay (Adopted) | 83.1  | 69.1  | 1722 | 56.9 | 53.9 | 61.7 | 74.9  | 55.7   | 31.4  | 64.1   | 95.6%|\n    | Fast->Slow             | 82.7  | 68.4  | 1686 | 55.4 | 53.8 | 59.7 | 72.6  | 54.8   | 30.7  | 63.3   | 93.9%|\n    | Slow->Fast             | 83.0  | 69.2  | 1712 | 56.0 | 53.2 | 61.2 | 73.8  | 55.2   | 31.3  | 63.7  | 94.9% |\n\n2. Thank you for pointing this out. We now included a comparison of overall latency in below table on the TextVQA dataset. Glance2Gaze achieves a **3.15× speedup in overall inference latency** and a **6.39× speedup in prefilling time** compared to the vanilla model. These results confirm that our method offers substantial end-to-end efficiency gains, not just in the prefilling phase.\n\n    | Method      | TFLOPs | Cuda Time (ms) | Latency (ms) |\n    |:-----------:|:------:|:--------------:|:------------:|\n    | LLaVA-NeXT  | 53.83  |      313       |     475      |\n    | FastV       | 2.76   | 170 (1.84 $\\times$) | 358 (1.33 $\\times$) |\n    | SparseVLM   | 2.76   | 183 (1.71 $\\times$) | 379 (1.25 $\\times$) |\n    | VisionZip   | 2.76   |  57 (5.49 $\\times$) | 192 (2.47 $\\times$) |\n    | Glance2Gaze | 2.67   |  49 (6.39 $\\times$) | 151 (3.15 $\\times$) |\n\n3. We acknowledge that Glance2Gaze requires model finetuning, unlike training-free approaches. However, the results reported for VisionZip correspond to its fine-tuned variant (as noted in their paper), ensuring a fair comparison. Importantly, despite requiring training, Glance2Gaze significantly reduces overall training cost: as shown in our response to Reviewer CY4r (W1), it achieves a **76.2% reduction in LLaVA-Next-7B training time**, operating at only **5% of full-model FLOPs**, while retaining **94.8%** of the original model’s performance. This exceeds VisionZip’s fine-tuned performance (**93.6%**) under comparable compression, demonstrating that the modest training overhead yields superior efficiency–performance trade-offs and positions Glance2Gaze as a highly competitive alternative.\n\n4. We thank the reviewer for highlighting the cognitive motivation behind Glance2Gaze. Conceptually, Glance2Gaze is motivated by a well-established cognitive principle, it explicitly separates what to attend to from how to resolve it. This two-stage abstraction enhances spatial reasoning and compositional generalization, contributing to improved performance. Our current goal is to  demonstrate the proposed approach's effectiveness through extensive experiments. We acknowledge that a stronger theoretical foundation, e.g., via information-theoretic or attention-based modeling, would further strengthen the work and we consider it an important direction for future research.\n\nThen here are the answers to the Questions part.\n\n 1. Please see the answers to Weakness one.\n 2. We thank the reviewer for raising this point.\n\n    1). As mentioned in lines 200–201 of the main paper, the breakdown is provided in Supplementary A.2. However, we would like to acknowledge that Equation (1) in the Glance Fusion section of the supplementary initially omitted the projection operations. After incorporating all relevant projection layers, we recalculated the overhead introduced by Glance Fusion to be approximately 0.03 TFLOPs. Given that the language model requires around 1.1 TFLOPs to process visual tokens during the prefilling stage, the updated overhead ratio is 0.0272. We will revise both the supplementary and main text to reflect this corrected value for transparency and accuracy.\n\n    2). The primary distinction between the prefilling and decoding (inference) stages of LLMs lies in token processing. During prefilling, all input tokens, including instruction text and image tokens, are processed in parallel. In contrast, decoding omits image token computation, as their hidden states are cached in the KV store. Consequently, the FLOPs associated with image tokens pertain exclusively to the prefilling stage.\n\nThen here are the answers to the Limitations part.\n\n1. We appreciate the reviewer’s valuable suggestion to compare Gaze Compression with attention-based token pruning at a fixed compression ratio. Accordingly, we implemented a baseline that removes tokens solely based on instruction-to-image attention scores starting from the $r$-th decoder layer. To ensure fairness, we fixed $r=9$ and $p_r=256$, replicating the exponential token retention schedule used in Glance2Gaze. At each layer from $r$ onward, visual tokens were ranked by their attention importance, and the lowest-ranked tokens were pruned according to the predetermined count. As shown in the table below, this attention-based pruning causes substantial performance degradation. Even after fine-tuning (The visual encoder is frozen while all other parameters are fine-tuned for 1 epoch, using identical training configurations as Glance2Gaze), the baseline fails to match Glance2Gaze’s performance, confirming that static pruning based on local attention scores lacks the adaptability required for task-specific compression.\n\n    | Method   | POPE | SQA  | MME  | GQA  | SEED | MMB  | VQA v2 | TextVQA | MMVet | LLaVA-B | Avg.|\n    |:--------------:|:----:|:----:|:----:|:----:|:----:|:----:|:------:|:-------:|:-----:|:-------:|:-------:|\n    | w/o finetune     | 83.4 | 69.1 | 1775 | 59.7 | 54.7 | 63.0 | 74.6   | 55.8    | 30.6  | 62.0    | 96.1% |\n    | w finetune     | 84.7 | 69.1 | 1793 | 60.8 | 56.8 | 63.2 | 75.7   | 56.4    | 31.4  | 65.2    |97.9% |\n    | Glance2Gaze    | 85.5 | 70.4 | 1812 | 61.5 | 58.7 | 64.5 | 77.6   | 57.2    | 32.7  | 66.4    |99.9% |"}, {"review_id": "tuidoDXM71", "reviewer": "Reviewer_TFE2", "summary": "This paper focuses on the efficiency of MLLMs, with one idea that to observe the image information, it is necessary to first conduct a preliminary global browsing, and then focus on semantically prominent regions. A two-stage framework, namely Glance2Gaze, is proposed. The glance fusion module integrates multi-layer vision transformer features with text-aware attention to generate semantically global representations, and the gaze compression module utilizes a query-guided mechanism to selectively compress visual tokens by semantic relevance. Experiments are conducted in a series of VQA benchmarks.", "questions": "[-] Some Surprising Results. In Table 1, on the benchmarks of SQA, MMVet, and SEED, Glance2Gaze even outperforms LLaVA-1.5-7B, which means that compressing tokens for efficiency actually enhances performance. Please provide more discussions to explain these phenomena.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[+] The manuscript is well written, with clear logics and sufficient formulations.\n\n[+] Efficient MLLMs have great practicality for the community, thus this work is on the right path.\n\n[-] Many bad cases from the cognitive perspective. For some simple scenes, from glance to gaze is feasible. While in many complex scenes, a single image has multiple focuses, and sometimes, these focuses may overlap. Such complex scenes may lead to failure of Glance2Gaze. \n\n[-] Unclear Efficiency. In Sec 3.3, this paper shows some cases of LLaVA-1.5-7B in terms of FLOPs to prove the efficiency. However, this is not comprehensive and requires more evaluations, such as lantency and throughput. Please report more quantitative results.\n\n[-] How to deal with image-text unpatching? As one generalized model, MLLM also encounters many special inputs, e.g., the text inputs are unrelated to the image inputs. In this case, glance fusion will be misled, causing gaze compression to fail.\n\n[-] Insufficient Ablations. How to prove the effectiveness of the glance fusion module?  For gaze compression, will learnable queries be better than directly calculating the similarity of image-text?", "was_revised": true, "final_justification": "Firstly, I would like to thank the author for their rebuttal. During this period, the author provides detailed responses to my key concerns, which further enhances the quality of the method. Taking into account both innovation and practicality, I have decided to slightly increase my rating to borderline acceptance.", "rebuttal": "Thank you for taking the time to review our paper.\n\nHere we address the points you mentioned in the weaknesses part.\n\n1. Thank you for the insightful comment. We agree that cognitively complex scenes, particularly those with multiple and potentially overlapping focal points, pose challenges for attention-reduction mechanisms. To explore this, we evaluated Glance2Gaze on counting tasks from the VQA v2 dataset, which exemplify such complexity due to scattered and partially overlapping target objects requiring distributed attention. We compare with LLaVA-1.5-7B under various reduction ratios. As shown in the table below, Glance2Gaze consistently preserves 93.43%–99.20% of its full-retention accuracy, significantly outperforming VisionZip. These results suggest that Glance2Gaze does not concentrate on a single region, but instead maintains broad coverage of all relevant areas in complex visual scenes. We welcome further discussion if you were referring to other types of complex scenes.\n\n    | Method      | FLOPs | Acc          |\n    |:-----------|:-----|:------------|\n    | LLaVA-1.5-7B| 100%  | 62.4 (100%)  |\n    | VisionZip   | 33%   | 61.8 (99.04%)|\n    | Glance2Gaze | 33%   | 61.9 (99.20%)|\n    | VisionZip   | 22%   | 59.3 (95.03%)|\n    | Glance2Gaze | 22%   | 60.4 (96.79%)|\n    | VisionZip   | 11%   | 57.0 (91.35%)|\n    | Glance2Gaze | 11%   | 58.3 (93.43%)|\n\n2. In Section 3.3 of the main paper, we report FLOPs and also include CUDA prefilling time in Table 4 as a proxy for latency. We now included a comparison of overall latency and throughout in below table on the TextVQA dataset. Glance2Gaze achieves a **3.15× speedup in overall inference latency**, a **6.39× speedup in prefilling time** and a **3.135× increase in throughput** compared to the vanilla model. These results confirm that our method offers substantial end-to-end efficiency gains, not just in the prefilling phase.\n\n    | Method      | TFLOPs | Cuda Time (ms) | Latency (ms) | Throughput |\n    |:-----------:|:------:|:--------------:|:------------:|:------------:|\n    | LLaVA-NeXT  | 53.83  |      313       |     475      |    10.884 | |\n    | FastV       | 2.76   | 170 (1.84 $\\times$) | 358 (1.33 $\\times$) | 12.272 (1.128 $\\times$) |\n    | SparseVLM   | 2.76   | 183 (1.71 $\\times$) | 379 (1.25 $\\times$) | 13.233 (1.216 $\\times$) |\n    | VisionZip   | 2.76   |  57 (5.49 $\\times$) | 192 (2.47 $\\times$) | 26.983 (2.479 $\\times$) |\n    | Glance2Gaze | 2.67   |  49 (6.39 $\\times$) | 151 (3.15 $\\times$) | 34.121 (3.135 $\\times$) |\n\n3. Thank you for raising this point. While most standard VQA datasets contain well-aligned image-text pairs, we recognize the need for benchmarks that intentionally introduce image-text mismatch while still requiring visual grounding. Despite an extensive search, we were unable to identify publicly available datasets with these characteristics, and we would welcome recommendations for future study. As an alternative, we evaluated POPE, a benchmark for hallucination scenarios where textual descriptions reference non-existent objects. Results in Tables 1~2 in the main paper demonstrate Glance2Gaze's robustness across reduction ratios, significantly outperforming other pruning methods. \n\n4. We appreciate reviewer's attention on effectiveness of Glance Fusion module and learnable queries in Gaze Compression. \n\n    1). In the main paper, we conducted targeted ablations to evaluate the contribution of the Glance Fusion module. As shown in Figure 4, the “Gaze Compression Only” baseline disables Glance Fusion, revealing a notable performance drop discussed in lines 282–288. Additionally, Table 5 compares Glance Fusion as a plug-and-play component against alternative fusion strategies, with detailed analysis in lines 289–296. Both results consistently confirms Glance Fusion's effectiveness.\n\n    2). Accordingly, we implemented a baseline that removes tokens solely based on instruction-to-image attention scores starting from the $r$-th decoder layer. To ensure fairness, we fixed $r=9$ and $p_r=256$, replicating the exponential token retention schedule used in Glance2Gaze. At each layer from $r$ onward, visual tokens were ranked by their attention importance, and the lowest-ranked tokens were pruned according to the predetermined count. As shown in the table below, this attention-based pruning causes substantial performance degradation. Even after fine-tuning (The visual encoder is frozen while all other parameters are fine-tuned for 1 epoch, using identical training configurations as Glance2Gaze), the baseline fails to match Glance2Gaze’s performance, confirming that static pruning based on local attention scores lacks the adaptability required for task-specific compression.\n\n    | Method   | POPE | SQA  | MME  | GQA  | SEED | MMB  | VQA v2 | TextVQA | MMVet | LLaVA-B | Avg.|\n    |:--------------:|:----:|:----:|:----:|:----:|:----:|:----:|:------:|:-------:|:-----:|:-------:|:-------:|\n    | w/o finetune     | 83.4 | 69.1 | 1775 | 59.7 | 54.7 | 63.0 | 74.6   | 55.8    | 30.6  | 62.0    | 96.1% |\n    | w finetune     | 84.7 | 69.1 | 1793 | 60.8 | 56.8 | 63.2 | 75.7   | 56.4    | 31.4  | 65.2    |97.9% |\n    | Glance2Gaze    | 85.5 | 70.4 | 1812 | 61.5 | 58.7 | 64.5 | 77.6   | 57.2    | 32.7  | 66.4    |99.9% |\n\nThen here are the answers to the Questions part.\n\n1. While full retention theoretically preserves maximal information, it may introduce redundant or irrelevant features that degrade model performance, an effect similarly observed in early deep networks, where increasing depth did not always yield better results compared to shallower architectures. In contrast, our approach avoids naive compression by employing a structured fusion mechanism (e.g., Glance Fusion) that selectively integrates visual signals aligned with task-relevant semantics."}, {"review_id": "73VQKZzVT5", "reviewer": "Reviewer_CY4r", "summary": "The paper introduces a new method to **compress visual tokens** in LLaVA-style multimodal LLMs (where a ViT vision encoder has its vision tokens projected to prefixed input tokens to the autoregressive LLM). The method \"Glance2Gaze\" consists of two modules:\n\nFirst, a **Glance Fusion module** computes $\\mathbf{V}_Q$ based on *both* vision tokens from multiple layers and text tokens from the instruction. The idea here is to extract information from multiple levels and based on the instruction, similar to the initial fixation stage in the human visual system.\n\nSecond, a **Gaze Compression** module replaces the full set of visual tokens with a subset that shrinks in progressive layers of the LLM transformer, to reach a certain target reduction in FLOPs. This set is computed via cross-attention using learned queries and $\\mathbf{V}_Q$ from the Glance Fusion module as keys and values.\n\nGlance2Gaze is compared to recent methods for visual compression in VLMs (FastV, SparseVLM, PDrop, VisionZip) on some common VLM and video benchmarks, reducing the FLOPs in the LLM to 33%/22%/11%, and shows a **consistent improvement**.", "questions": "1. In Formula (9) the symbol $\\mathbf{V}^l_Q$ appears both on the left side and the right side. I think on the right side it should instead be $\\mathbf{V}^{l-1}_Q$. I find it a bit confusing that there are \"task-enhanced visual tokens\" $\\mathbf{V}_Q$ in Equation (8) and then there is a new symbol \"visual tokens\" $\\mathbf{V}^l_Q$ that re-uses the same symbol but is different (the former being the integrated tokens from the Glance Fusion stage, and the latter being the consecutively compressed visual tokens from the Gaze Compression stage).\n\n1. When computing the FLOPs it seems that only the LLM is taken into consideration? What fraction of the overall FLOPs are due to the vision encoder? Note that in Table 4 a 20x fold of FLOPs reduction is reported, but the speedup is 6.39x.\n\n1. FLOPs computations in the appendix in Equation 2 and lines 20-27: Why would the visual token sequence compression FLOPs (first part of the formula) be included in \"gaze compression\"? I find that naming a bit confusing because the first part is simply the cost of the forward pass in the decoder. Still with the first part of the equation, why is $n$ constant throughout the decoder? I would have expected $n$ to be constant up to layer $r-1$, but then be replaced with $n_\\text{ins} + p_l$ from the gaze compression, otherwise Gaze Compression would only add FLOPs to the sum (in the second part). As for the Glance Fusion FLOPs computation (appendix lines 18-19), it seems that the projections are missing from Equation 1.\n\n1. In Figure 1, how are the tokens pruned? Are the residuals of pruned tokens still available to layers with a higher index?\n\n1. In Figure 4, is it correct that the \"Layer-specific Query\" bars include the Glance Fusion stage and the only difference to the full Glance2Gaze is that the query embeddings are not shared?\n\nSmall comments\n\n1. Line 95: \"it resizes\"\n1. Line 108: missing space between the reference list and \"Resampler\".\n1. Line 255: missing space between \"Bind\" and reference\n1. Line 279: \"but\" is a bit misleading: one would usually expect the first part of the phrase to be positive (e.g. \"reduces parameters\") and the second part to be negative (e.g. \"reduces performance\") – but in this case both changes are negative\n1. Line 637: where exactly is the training compute mentioned in the paper?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper introduces an **innovative** biologically-inspired method for visual token compression in LaVA-style models. The Glance Fusion module leads to a set of vision tokens that is better suited for answering a question by incorporating the question in the selection process. The Gaze Compression mechanism still achieves good results at a high token compression level.\n\n1. The method works with both images and videos and achieves a **consistent performance gain** over recent methods like VisionZip.\n\n1. The paper is **well-written** and effectively illustrated, making the somewhat complex methodology easy to understand.\n\n**Weaknesses**\n\n1. The paper does not report the overall cost of training with Glance2Gaze. The **FLOPs computations** for inference are incomplete (e.g. the vision encoder is ignored), and there is only a single measurement of inference performance (single architecture, single FLOPs reduction setting).\n\n1. **Clarity** of some of the equations and FLOPs computations **could be improved**, see details in section \"Questions\" below.", "was_revised": true, "final_justification": "I maintain my rating of 5=accept. I think the paper presents an interesting method and shows a superior performance with previous sota in a fair comparison, so I find it valuable to share the work with the larger NeurIPS community.\n\nAs for the two weaknesses that I reported in my original review, I think the authors have addressed both of them satisfactorily:\n\n1. The authors expanded on their FLOPs computations, and added a table that not only lists theoretical FLOPs, but also measures throughput and prefill latency, and compares their method to various baseline methods at different sparsification settings.\n\n2. As for clarity, the authors have fixed some issues that I mentioned in my original review, and promised to improve the readability of some of the formulas that I found lacked a bit on clarity in the initial version of the manuscript.", "rebuttal": "Thank you for your thoughtful comments and suggestions.\n\nHere we address the points you mentioned in the weaknesses part.\n\n1. We appreciate reviewer's attention on training cost, FLOPs computation and inference performance. \n\n    1). Here we provide comprehensive tables presenting training cost, inference latency, prefilling time, and throughput across various frameworks and compression settings to offer a clear picture of computational costs.\n\n    | Method                | Training GPU Hours | Inference TFLOPs | Latency (ms) | Cuda Time (ms) | Throughput |\n    |:---------------------:|:------------------:|:----------------:|:------------:|:--------------:|:----------:|\n    | LLaVA-1.5-7B         | 104                | 10.07            | 185.9        | 115            | 28.754     |\n    | Glance2Gaze (33% FLOPs) | 72                 | 3.36             | 133.2        | 66             | 40.022     |\n    | Glance2Gaze (22% FLOPs) | 60                 | 2.22             | 109.2        | 32             | 51.351     |\n    | Glance2Gaze (11% FLOPs) | 43                 | 1.11             | 100.2        | 22             | 56.82      |\n\n    | Method                | Training GPU Hours | Inference TFLOPs | Latency (ms) | Cuda Time (ms) | Throughput |\n    |:---------------------:|:------------------:|:----------------:|:------------:|:--------------:|:----------:|\n    | LLaVA-Next-7B         | 366                | 53.83            | 475.2        | 313            | 10.884     |\n    | Glance2Gaze (22% FLOPs) | 242               | 11.84            | 262.0        | 100            | 19.741     |\n    | Glance2Gaze (11% FLOPs) | 173               | 5.53             | 191.9        | 90             | 30.719     |\n    | Glance2Gaze (5% FLOPs)  | 87                | 2.67             | 151.2        | 49             | 34.121     |\n\n    | Method                | Training GPU Hours | Inference TFLOPs | Latency (ms) | Cuda Time (ms) | Throughput |\n    |:---------------------:|:------------------:|:----------------:|:------------:|:--------------:|:----------:|\n    | Video-LLaVA           | 297                | 37.38            | 739.6        | 342.2          | 21.599     |\n    | Glance2Gaze (6% FLOPs) | 151                 | 2.24             | 484.3        | 125.1          | 35.095     |\n\n    2). Following common practice in visual pruning literature (e.g., VisionZip, Pdrop, SparseVLM), we focus FLOPs measurement primarily on the computation related to visual tokens within the LLM. This is because pruning techniques selectively reduce computation in the LLM's visual token processing, while the visual encoder’s computational cost remains fixed and thus does not affect the pruning evaluation. Here we also present a detailed breakdown of FLOPs across different components of the vision-language model, including the visual encoder.\n\n     |         Glance2Gaze (11% FLOPs)          | in Visual Encoder | in Glance Fusion | in LLM  |\n    |:-----------------:|:-----------------:|:----------------:|:-------:|\n    | FLOPs             | 0.38 T             | 0.03 T           | 1.1 T   |\n    | proportion        | 0.2517             | 0.0199           | 0.7285  |\n\n2. Thank you for the helpful suggestion. We will revise the relevant equations and FLOPs computations in the main text and supplementary to improve clarity and transparency.\n\nThen here are the answers to the Questions part.\n\n1. Thank you for pointing this out. We acknowledge that the reuse of the same symbol in Equation (9) as well as the reuse from Equation (8) could lead to confusion regarding the semantics of the visual tokens at different stages of the pipeline. We will carefully revise the notation in the final paper.\n\n2. Please see the answers to Weakness one for more comprehensive tables presenting training cost, inference latency, prefilling time, and throughput. Regarding the discrepancy between FLOPs reduction and wall-clock speedup, this stems from the well-established distinction between theoretical computational savings and practical runtime acceleration. FLOPs reductions do not translate linearly to real-world speedups due to factors such as memory access patterns and hardware constraints. Nonetheless, achieving a 6.39× speedup represents a substantial improvement in inference latency, underscoring the practical effectiveness of our approach.\n\n3. We thank the reviewer for highlighting this point and apologize for any confusion caused in FLOPs computation of visual tokens. \n\n    1). Equation (2) aims to compute the total computational cost of processing visual tokens within the decoder, which consists of two parts:\n    -   the standard forward pass through the decoder (first term), \n    -   the additional cost introduced by the Gaze Compression module (second term).  \n\n    We will revise the title to **“FLOPs within LLM (Visual Tokens)”**, to reflect that both standard forward and gaze compression contribute to the total cost.\n    \n    2). Regarding the variable $n$ in the first term of Equation (2), your interpretation is correct. As clarified in Line 206 of the main text, $n$ denotes a variable that remains constant prior to layer $r$ and takes the value of $p_l$ thereafter.\n\n    3). We appreciate the reviewer pointing out the omission. The FLOPs computation for Glance Fusion should indeed include the cost of projection layers for both text and visual inputs. In Glance Fusion, the computational cost should include $S$ projection layers for text mapping and $S$ projection layers for visual mapping, which bring $SNd_vd_t$ and $SMd_t^2$ FLOPs respectively. After correction, the FLOPs for the Glance Fusion stage be 0.03T. Thus, the ratio of computation between Glance Fusion and the LLM is $0.03/1.1=0.0272$. We will update Equation (1) and the related text accordingly.\n\n4. We compute visual token importance by averaging instruction-to-image attention scores extracted from the decoder's attention matrices. Specifically, we isolate the instruction-to-visual attention submatrix and calculate the mean attention each visual token receives from all instruction tokens, yielding a scalar importance score per token. Tokens are ranked accordingly, and those with the lowest scores are pruned based on a predefined ratio. Pruned tokens and their residuals are completely excluded from subsequent layers and computations, thus exerting no influence on downstream activations.\n\n5. Your understanding is correct. The \"Layer-specific Query\" bars in Figure 4 do include the Glance Fusion stage. The only difference from the full Glance2Gaze model is that the query embeddings are not shared across layers. \n\n6. Regarding the small comments, we thank the reviewer for pointing out grammatical and formatting errors, and we will address these in the final paper. Please refer to our response in Weakness one for clarification on training compute concerns."}, {"review_id": "jcUD4KbTAM", "reviewer": "Reviewer_j7mC", "summary": "This work points out that the existing approaches focus on reducing visual tokens at the visual encoder phase or the LLM decoder stage. However, this work suggests that there should be an initial global glance that precedes. This work has two key components: 1) Glance Fusion module: integrates a multi-layer vision transformer; 2) Gaze compression module: selectively compresses visual tokens based on the semantic relevance.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n* The paper is well-written and clear presentation.\n* The motivation is clear, presented in Figure 1. And the proposed method is based on the Motivation.\n* The proposed method is clear and presented in Figure 2.\n* The experiment supports the proposed methods.\n\nWeakness:\n* Provide the code in Appendix for reproduction.\n* In Table 3, the Glance2Gaze  achieves better performance than naive Video-LLaVA on TGIF. However, the Galance2Gaze should achieve lower performance by employing compression.\n* In Table 5, it seems that the effectiveness of the proposed fusion method gradually decrease, with the model size increase. What is the performance of the proposed fusion method with a larger model size?", "was_revised": true, "final_justification": "There is no concerns.", "rebuttal": "Thank you for your valuable feedback.\n\nHere we address the points you mentioned in the weaknesses part.\n\n1. We appreciate the reviewer’s emphasis on reproducibility and the interest in accessing our code. Our code will be released upon paper’s acceptance.\n\n2. We clarify that among the four video-QA datasets in Table 3 of the main text, only ActivityNet shows a slight performance gain for Glance2Gaze over naive Video-LLaVA. On TGIF and the other datasets, Glance2Gaze generally underperforms, aligning with the expectation that compression reduces available information and may impair performance. However, video data often contain significant redundancy and noise within visual tokens, which can hinder model comprehension. The compression applied by Glance2Gaze aims to distill salient visual information by removing redundant or noisy tokens. This simplification can yield a clearer, more focused representation, enabling better feature capture and, occasionally, improved performance on specific datasets or samples. Therefore, the occasional performance improvement post-compression is plausible and consistent with the data characteristics and model behavior.\n\n3. Thanks for your insightful comment. We conducted additional experiments incorporating a larger model following Dense Connector for fair comparison. Specifically, we employed SigLIP-so-384px as the visual encoder alongside the Yi-34B large language model to evaluate the effectiveness of our Glance Fusion method at scale. Due to resource limitations, we utilized the LoRA fine-tuning technique for training the projector and the LLM, with lora_r and lora_alpha set to 128 and 256, respectively, consistent with Dense Connector’s settings. Given that SigLIP comprises 27 ViT layers, we applied Glance Fusion at layers [7, 14, 21, 27]. The results, summarized in the table below, indicate that Glance Fusion significantly enhances performance in larger models, surpassing Dense Connector. \n\n    | Method                   | training | GQA  | VQA v2 | SQA  | TextVQA | POPE |\n    |:------------------------:|:--------:|:----:|:------:|:----:|:-------:|:----:|\n    | baseline (SigLip-Yi-34B) | LoRA     |   62.4   |    81.1    |   78.8   |    64.7     |    81.4  |\n    | + Dense Connector        | LoRA     | 63.9 (+1.5) |    -    | 80.5 (+1.7) | 66.7 (+2.0)    |   80.5 (-0.9)  |\n    | + Glance Fusion          | LoRA     |   64.3 (+1.9)   |    83.5 (+2.4)    |   81.9 (+3.1)   |    67.5 (+2.8)     |    82.6 (+1.2)  |"}]

Author rebuttal:

Meta-review: This paper makes a significant contribution to the important and practical problem of VLM efficiency. The proposed Glance2Gaze framework is novel, well-motivated, and empirically powerful. The paper received 3 borderline accepts and an accept. Authors have addressed all major reviewer concerns. Therefore, we have decided to accept the paper.
Decision: Accept (poster)