Rethinking Causal Mask Attention for Vision-Language Inference
Authors: Xiaohuan Pei
Tao Huang
Yanxiang Ma
Chang Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Inference
Casual Mask Attention
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We challenge the traditional causal mask design and comprehensively investigate its role in vision-language inference.

Abstract: Causal attention has become a foundational mechanism in autoregressive Vision-Language models (VLMs), unifying textual and visual inputs under a single generative framework. However, existing causal mask-based strategies are inherited from large language models (LLMs) where they are tailored for text-only decoding, and their adaptation to vision tokens is insufficiently addressed in the prefill stage. Strictly masking future positions for vision queries introduces overly rigid constraints, which hinder the model’s ability to leverage future context that often contains essential semantic cues for accurate inference.
In this work, we empirically investigate how different causal masking strategies affect vision-language inference and then propose a family of future-aware attentions tailored for this setting.
We first empirically analyze the effect of previewing future tokens for vision queries and demonstrate that rigid masking undermines the model’s capacity to capture useful contextual semantic representations. Based on these findings, we propose a lightweight attention family that aggregates future visual context into past representations via pooling, effectively preserving the autoregressive structure while enhancing cross-token dependencies.
We evaluate a range of causal masks across diverse vision-language inference settings and show that selectively compressing future semantic context into past representations benefits the inference.

Published as a conference paper at ICLR 2026
RETHINKING CAUSAL MASK ATTENTION FOR VISION- LANGUAGE INFERENCE
Xiaohuan Pei
Tao Huang
Department of Computer Science The University of Sydney, Australia x xiaohuan. pe p @sydney. edu. au
Department of Computer Science Shanghai Jiao Tong University, China t. t.huang@ss s wtw.juuuuu...
Yanxiang Ma
Chang Xu
Department of Computer Science The University of Sydney, Australia yanx iang. ma @ sydney. edu. au
Department of Computer Science The University of Sydney, Australia
C. xuê sydney. edu. au
ABSTRACT
Causal attention has become foundational mechanism in autoregressive Vision- Language models (VLMs), unifying textual and visual inputs under single gen- erative framework. However, existing causal mask-based based strategies are inherited from large language models (LL.Ms) where they are tailored for text-only decod- ing, and their adaptation to vision tokens is insufficiently addressed in the prefill stage. Strictly masking future positions for vision queries introduces overly rigid constraints, which hinder the model's s ability to leverage future context that often contains essential semantic cues for accurate inference. In this work, we empirically investigate how different causal masking strategies affect vision- lasgun-language inference and then propose a family of future-aware attentions tailored for this setting. We first empirically analyze the effect of previewing future tokens for vision queries and demonstrate that rigid masking undermines the model's capacity to capture useful contextual semantic representations. Based on these findings, we propose a lightweight attention l'milly that aggregates future visual context into past rep- resentations via pooling, effectively preserving the autoregressive structure while enhancing cross- toos-token dependencies. We evaluate a range of causal masks across diverse vision-language inference settings and show that selectively compressing future semantic context into past representations benefits the inference.
INTRODUCTION
In recent years, autoregressive large language models (LLMs) have achieved remarkable break- throughs in linguistic understanding by enforcing causal attention mechanisms that restrict each token to attend only to its preceding context (Achamm et al. (202)); ) Anil et al. (2023); ; Bai et al. (2023); ; Li al. (025): y Radford et al. (2019). This left-to-right causal masking effectively prevents information leakage from future tokens, aligning model predictions with the natural sequential structure of lan- guage. For instance, in the sentence "She very smart", predicting the token "very" based the token "smart" would violate causality, as the model could also generate "not smart" given the same prefix. By restricting attention past tokens, causal masking ensures consistent and contextually appropriate predictions.
2024a Bai et al. (2025): Chen et al. (2024b); Li et al. (2025); (Chu et al. (202)). which extend LLMs to multi-modal settings, adopt similar autoregressive framework by aligning and concatenating visual tokens with textual tokens. However, unlike text, visual information is inherently non-sequential, with regions processed holistically
langunel models (VLMs) Liu al. (2033
"This work was supported part by the Australian Research Council under Projects DP240101849 and FT230100549. 1 'Coresponding Author

---

Published as a conference paper at ICLR 2026
rather than strictly in order. Consequently, enforcing strict causal masking on visual tokens may unnecessarily restrict the model's capacity to leverage contextual cues from future tokens. Recent studies Yin et al. 2024) suggest that future semantic attention scores, typically masked in causal settings, can be exploited without violating causal logic. Furthermore, Qi et al. (2025) indicates that visual tokens may not benefit significantly from positional interactions, highlighting potential misalignment between the causal structure designed for text and the optimal structure for visual processing.
As a result, a critical question arises: Is causal attention truly a feasible mechanism for vision- language understanding? In this paper, we systematically explore the impacts of causal attention on textual and visual tokens and reveal a surprising finding: while breaking the causal masks between textual tokens significantly disrupts model predictions, relaxing the causal constraints on visual tokens unexpectedly improves performance, even though the model is trained causally (see Figure D.
To comprehensively investigate this phenomenon, we conduct an in-depth analysis of how relaxing future attention for visual tokens affects model behavior across diverse vision-language tasks, particularly those involving long contexts and multi- image reasoning. We propose three future-aware causal mask- ing strategies, each targeting distinct regions in the multi-modal attention matrix. By examining their task-specific advantages and limitations, garding the role of future visual context in enhancing inference accuracy. Our study aims address the following key ques- tions: (1.) Does the causal attention in LLM fits the visual tokens in the popular VLMs like LLaVA? (2.) How should the causal attentions be revised to fit the multi-modal situation?
Rouge
10
Break Visua
uncover a variety of intriguing insights re-
Break Tecrual Causal Aitertion
Before
After
Figure 1: Breaking the casual masks of LLaVA-7b on the AL- FRED benchmark [Shridhar et al. (200
(3.) What tokens should the vision tokens allowed to access in the causal mask? (4.) How does pre-seen visual semantic information impact tasks that heavily rely on visual reasoning? How about text-depent text-dependent tasks?
Based our findings, we conclude that allowing visual tokens to access future context significantly enhances VLM performance. However, directly breaking the causal masks between visual tokens substantially inereases computational cost. To effectively incorporate future information while maintaining computational efficiency, we propose a kernel pooling method that merges future semantic attention into past regions. Additionally, pronounced impact of merging future attention into attention sink regions, merging future into past even outperforms direct future access in some tasks, which notably alters VLM inference behavior.
uncover several intriguing insights, such the
As result, this paper revisits the overlooked role of causal attention for visual tokens VLMs and systematically investigates its limitations and alternatives. By proposing and evaluating family of future-aware masks along with lightweight merging techniques, it offers both empirical gains and conceptual insights that challenge the default fundamental autoregressive design inherited from text-only LLMs.
PRELIMINARY
2.1 CAUSAL ATTENTION MECHANISM.
For vision- languael models (VLMs), the input consists of m visual tokens X E Bx and textual tokens X E BXxxxHxD which are concatenated into unified sequence X X @ X BBLLxH I, m + n. The input X is then projected into queries, keys, values: Q, K. e BBXLLXxD During the prefill stage, causal attention is computed as:
A= Softmax
QKT Vd
+ +MM)V.
Mi
0, 00,
i otherwise,
where A E RBXHXLXD, L = =mm + n, with m and n being the number of visual and text tokens respectively. The mask M E LRL enforces autoregressive constraints across the entire sequence.
2

---

Published as a conference paper at ICLR 2026
For each query token i, the causal masked row M, will be initialized as
Mf Ma1. Ma....
Mis
"....,, , -204
(2)
v(past
L i (tuuue)
2.2
VISION LANGUAGE MODEL
Vision Language Models (VLMs) like LLaVA Liu et al. (2023) transfer the image input X into vision tokens x E RI. via a pretrained vision encoder g(X), where m is the number of vision tokens. The vision tokens are projected into text feature spaces, but contain the information from the images as
(5 22 -... -.,)) = =x"= = -x
In VLM, after the vision encoder, the image tokens are treated as if they were text tokens. Both image tokens x" and text tokens x² are input into LLM Se in sequence. Denote token i in the token sequence by Ig when there are m vision tokens and n text tokens, VLMs can be generally defined as,
To fo x2 by
x/ 22 ".,,)
(4)
where Ta the feature of the output token. In the LLM, the input tokens are sent into the causal attention layers, where the context feature between the tokens will significantly affect the predic- tion Yang et al. (202T);
; Pei et al. (2024). To be more precise, we denote the image and text token separately. Let Q" Q K and K denote the queries and keys for the x" and w', respectively, we define B(x", x²) = (Q" "Q""'""""""""""""""""""""
where @ is the concatenate function. Follow in VLM, the softmax attention can be defined as,
va
(x x²; M9) Softmax (B(x", x') + M/²)
(5)
Then the attention output can be redefined as A ho(x" x², M V. The distribution of the prediction with causal attention in VLM can be formulated as
exp (e(ar) ho(x) im x x Mc))
(= = x x x x x Ene exp (e(x²) he (x) xi..; Mc))
(6)
where 2/ is the entire output vocabulary, e(-) is the vector in attention. Eq. 6 shows that the context information in visual semantics is learned between vision tokens. However, Eq. 3 shows that the context information of vision semantics is fixed into the vision tokens x by the pre-trained vision encoder g(X). This means that the causal attention conflicts with the vision encoder in context information comprehension. Intuitively, we believe that in VLM, the image tokens have huge potential unrevealed by not applying the causal attention mechanism on the image tokens.
3 UNDERSTANDING OF CAUSAL ATTENTION
We aim to investigate and release the potential of future context causal attention for vision-language models (VLMs). We begin by conducting an empirical study that examines how visual tokens interact with future tokens under various causal masking strategies. This analysis reveals that letting visual tokens open access to future context has the potential to improve reasoning performance. Motivated by these findings, we further propose a lightweight mechanism that enables the model to benefit from future visual signals without breaking the autoregressive structure. Figure 2 presents an overview of our investigation. The future-aware causal mask allows vision tokens to preview attention scores from future tokens and selectively compresses valid future attention into past positions enhance efficiency while preserving autoregressive constraints.
3.1 FUTURE AWARE CAUSAL MASKS
The mainstream VLM backbone consists of a vision encoder to project the visual patches to visual tokens, and afterwards concatenate them with text tokens. Given set of visual tokens x" = {x", ......, xy, and text tokens x² = {#f. ..., }. flattened input sequence can be constructed as X x @ x². In this paper we focus the case where vision tokens are entered before text tokens. The subscripts related to x" is n [1, m). and to x' is Z n [m + 1, m + n], where Z is the set of integers. For simplicity, denote Z 2 ml by V. and Z n [m 1,m. + n) by T.
3

---

Published as a conference paper at ICLR 2026
mask Inforence
Carnent
Your Boodl Fight
Thurn and wvr the was riightet nrightstt nexer
Plick Frr ouf engirnstt
to
Kamip
LLOKA
K
Figure 2: An overview of our investigation into causal attention in vision-language inference. (a.)
Casual mask inference: enforces strict autoregressive decoding by blocking all future attention. (b.) Future-aware inference: enables visual tokens to preview future tokens in the upper-triangular region. (c.) Light future-aware inference: compresses future attentions into past visual positions.
Table 1: AP: Action Prediction Wu et al.
Detrctio
(2024) VN: Visual Navigation Krantz
2020 SC: State Change
A suact so
Mask
Temporal Multi Multi-Image Tasks
raae Your
AP
VN
SC
Ieea Langl's
M
39.8
31
30
Figure 3: An Example of Temporal Multi-Images Task, Visual Navigation
M
39.9(t)
32(t)
31.5(t)
This decomposition enables to design future-aware variants of causal attention that selectively relax constraints for vision tokens while preserving strict autoregressive decoding for text. The standard design of causal attention prevents each token from attending to future positions and this constraint originally designed for text decoding, can overly restrictive for visual tokens. examine this, we propose set of causal masking strategies that make the visual attention access the future sematic attention scores. As the future region of casual mask contains visual to visual (+2v) and visual to text (u2t), we define three future-aware variants mask strategy: Future-Award Full Mask MI, Future-Aware Visual-to-Visse Mask M and Future-Aware Visual-to-Textur Mask M
Definition 3.1 FFuture-Awar Full Mask), For any query position i E (i.e., visual token), the future-an aware full mask M. E R retains attention to all positions j. including future tokens in both visual and textual modalities:
0, M,, 1- otherwise
SJ iniEV)
Then the following holds:
Full upper-triangle is visible for visual queries.
Pust causal structure is preserved: Mr, = 0 for
When E T. standard causal mask is used.
Observation of M Accessing full future attention scores for visual query could be beneficial to temporal multi-image tasks. Allowing visual tokens to attend to the entire future context enhances tasks that rely on global temporal reasoning, as it enables each visual query to incorporate upcoming visual attentions that be crucial for accurate inference and decision-making.
Analysis. Figure 3 and Table demonstrate that applying the full future-aware mask Mf consistently improves performance across all temporal multi-image tasks. Specifically, Visual Navigation (VN) and State Change (SC) tasks, which require image sequences, M yields significant score gains over the standard causal mask. These tasks (e.g.
reasoning over temporally ordered
4

---

Published as a conference paper at ICLR 2026
Egocentric Navigation, Action Sequence Prediction, and Scene Transition) demand the model to interpret actions or spatial arrangements over time. The full future mask allows each visual query to access all subsequent visual and textual context during the prefill stage. This enables the model to aggregate temporally rich semantics that are not yet locally visible but are crucial for understanding object motion trajectories, navigation goals, state shifts. Such unrestricted future attention is particularly helpful in settings where key visual cues for inference (e.g., an agent reaching a door or an object changing color) appear later in the image sequence. reveals that the utility of full future-aware attention varies with task structure: it brings minimal benefit to static or single- inputs but becomes indispensable for robust temporal modeling in multi-imen aulti-image, temporally structured settings.
Definition 3.2 (Future-Aware Visual-to- Visual Mask). For any query i E V. the visual-to-vic to-viuaal fulure mask M. permits altending future visual tokens but masks future text tokens:
0, ifj ii iii.jEV)
M
(8)
-00; otherwise
Then the following holds:
Only future visual tokens are accessible to visual queries.
Future text tokens are masked with -o0.
When E reverts to standard causal masking.
Tax
Table 2: VCC: Visual Change Cap-
tion Thamtani & Berg-Kirkpatrice (2018). VRE: Visual Relation Expres-
sion Hosseinzadeh & Wang 2021
Mask
Visual Relation Tasks
VCC
VRE
Figure 4: Example of a risual-relational task. The model must reason over relationships among visual tokens rather than iso- lated appearances.
M
16.2
16.6
M
16.7(1)
18.1(t)
Observation M Allowing access to future visual tokens can benefit Visual Relation Inference tasks (e.g., Visual Change Captioning, Visual Relationship Expression), as it enables visual queries to capture inteructions with future visual content- essential component of reasoning about visual relationships.
Analysis. Figure and Table show that applying the iisual-tovisuall future-aware mask Mr leads to noticeable improvements on visual relation tasks such Visual Change Captioning (VCC) and Visual Relation Expression (VRE). These tasks involve identifying subtle differences or relationships between two related images, where the visual context is rich but the textual signal is limited. By allowing visual queries to access future visual tokens during the prefill stage, M enables the model better compare visual patches across frames and capture object interactions or appearance changes. As illustrated in the distribution gap, the attention distribution under v2 closely aligns with the original softmax distribution, indicating that this selective relaxation of the mask preserves natural attention behavior. The empirical results show that visual relational reasoming relies mainly on intra- intra-modal alignment rather than cross-mode fusion. It benefits from accessing future visual tokens while textual tokens remain strictly causal.
Definition 3.3 (Future-Aware Visual-to- Textual Mask). For any query i E V, the visual-to-textudl future mask Mf allows access to future text tokens while masking future visual tokens:
(0
insiiniiVjj iisiv
M
(9)
-00,
otherwise
Then the following holds:
Visual queries could preview future textual atlention scores.
5

---

Published as a conference paper at ICLR 2026
Future visual context is strictly masked.
When E T, attention follows standard left-to-right causality.
Table 3: Text-Rich Image QA Tasks:
OCR- VQA Mishra et al. (2019) and
TextVQA Kembhavi et al. 2017
you rysprobbiity to
Mask
Text-Rich Image QA Tasks
What GINGUT
VQR-VQA
TextVQA
M
22.5
32.0 38.5(t)
Figure 5: An Example of Text-Rich VQA Tasks
M
23.0(1)
Observation of 2 Enabling future access from visual tokens to textual tokens benefits Text-Rich Image QA tasks, it allows visual queries to anticipate and integrate critical textual cues embedded in images-ofier the key to accurate reasoning and answer generation.
Analysis. Figure and Table 3 show that the risual-to-textuer future- mask M yields notable improvements in Text- Rich Image QA tasks such as OCR- OCR-VQA Mishra et al. (2019) and Vex Kembhavi al. (0177) These benchmarks require extracting fine-grained textual informa- tion embedded in complex visual layouts-such as texthook diagrams or document images where visual cues often need to resolve or align with distant text regions. By allowing visual queries to attend to future textual tokens, 21 enables earlier visual patches to preemptively integrate relevant linguistic content during prefill, improving semantic alignment and grounding. Specifically, this attention mode avoids exposing future visual context, maintaining temporal consistency. The KDE distribution gap further indicates that attention distribution under M better aligned with the natural softmax pattern than other variants, supporting the hypothesis that selective cross-modal future access can improve answer accuracy in scenarios dominated by image-embedded text.
Based on our definition in Def. 3.1 and 3.3 the distribution of To in Eq. 6 be revised as
exp (e(x) he (x xi..i R))
Px = x x x
(10)
Σn exp (e(x') The (x( xf. x(m)
where μ is the modified mask strategies and j E (M M MM) are selected manually and fixed.
LIGHT FUTURE AWARE ATTENTION FAMILY
While granting visual tokens access to future context holds great potential for improving multimodal understanding, such full visibility comes at the cost of increased inference laancy-pprticclartt during the autoregressive decoding phase. Fortunately, the recent trend of separating prefill and decoding stages in VLMs allows us shift this overhead entirely into the prefill phase. Leveraging this separation, we propose a lightweight attention mechanism that compresses future visual information into past positions during prefill, enabling the model to benefit from future-aware context while preserving the original causal mask structure during decoding.
1N
16J
161
Figure 6: An overview of attention design for vision language inference. (a.) Casual Mask Attention. (b.) Future-Aware Full Attention. (c.) Future-Aware Visual-to- -iiuull Attention. (d.) Future-Aware Visual-to- -ex-uulll Attention. (e.) Light Future Aware Attention.

---

Published as a conference paper at ICLR 2026
Table Performance comparison across vision-language tasks using different future-aware causal masking strategies for visual queries. We evaluate the baseline causal mask (M*), three future-relaxed variants (M M M2), and their lightweight merge variants (prefix size 1).
ActionL
ActionP
ActionS
CLEVR
Order
Doc
Nav
Moving
OCRVQA
Object
SpoDDfff
State
TQA
Method
LLaVA-7b
0.230
0.515
0.445
0.166
0.245
0.450
0.310
0.490
0.225
0.485
0.162
0.300
0.320
0.250
0.495
0.435
0.181
0.250
0.445
0.320
0.490
0.230
0.495
0.165
0.305
0.385
Mr
0.255
0.515
0.440
0.177
0.250
0.430
0.325
0,515
0.220
0.500
0.167
0.325
0.385
Mr
0.505
0.400
Amerre
0.225
0,51
0.435
0.175
0.27
0.445
0.320
0.490
0,205
0.510
0.167
0.305
0.385
M +merge
0.245
0.495
0.435
0.180
0,250
0.445
0.320
0.490
0,230
0.495
0.164
0.305
0.375
M +mmerge
0.245
0.500
0.450
0,188
0.265
0,420
0.320
0.505
0.225
0.490
0.173
0.320
0.375
LLaVA-13b
0.500
0.455
0.360
M
0.245
0.455
0.495
0.156
0.445
0.460
0.270
0.500
0.415
0.475
0.120
0.360
0.51.
M
0.500
0.360
M
0,245
0.460
0.495
0.156
0.440
0,460
0,260
0.510
0.415
0.475
0.155
0.370
0.510
emerge
0.245
0.455
0.495
0.155
0.445
0.46
0.270
0.500
0.415
0.475
0.141
0.36
0.515
M -merge
0.245
0.455
0.495
0.155
0.445
0.46
0.270
0.500
0,415
0.475
0.119
0.360
0.510
M +merge
0.255
0.450
0.495
0.158
0.445
0.465
0.260
0.505
0.415
0,480
0,115
0.355
0.525
Motivated by the attention sink phenomenon observed in autoregressive models Gu et al. (002))
Xiao et I 2024); Yang et al. (2024) and its effectiveness in recent inference optimization studies Ge al. [2023 ; Liu et al. (2024b), we merge the compressed future information into the initial vertical past positions to enhance semantic propagation during prefill. We apply ID kernel pooling over the attention weights using kernel size k to aggregate visual semanties, and merge the resulting
summary score back into the past region j I i of the same row A, as:
= [0, j S ir = -00
M1r(1)
(11)
[1,
otherwise,
ET 0,
mB @
where j S and
C(B,1)
(12)
otherwise
where A; E RR denotes the attention distribution for query Qs Af represents its masked future segment, l is the kernel size, T aggregated semantic score via kernel pooling, and C(B, jt) is the attention row after merging. Then, the autoregressive generation process refines the token distribution from both the compressed future and the original past attention, producing predictions conditioned on enriched context representations:
i defines the maximum pooling range, AP is the
1x(x", x': p) (B(x", xx) + C(B. p) + M²)
(13)
Based on our definition in Def.
3.1 3.2 and 3.3 the distribution of x, in Eq. Glaan be revised as
exp (e(x:) (1)))) (xf ximi R))
Po = x x xf.n)
(14)
Σ. exp (e(x') ) Thg (x"xx.m xi:ni
where μ E (M M M) selected manually and fixed, and hn ximm xf :4) is the modified mask attention family equipped with merged semantic future attentions.
The compressed method ensures that the final attention pattern remains strictly causal (lower- triangular) while still benefiting from future visual semantics aggregated during prefill.
Analysis of I. Lightweight Attention Results. Table 4hows that the proposed lightweight attention strategy, which merges compressed future scores into fixed prefix token, achieves competitive performance while preserving the standard causal structure. Across both 7B and 13B models, future- aware masks with prefix merging (such as A +merge and M mmerge) perform par with or better than their unmerged counterparts on tasks involving temporal reasoning, visual relations, and text-rich understanding. This indicates that full access to future tokens is not always necessary during decoding. Instead, summarizing future information into a small prefix, even a single token, provides sufficient global context for accurate generation.
7

---

Published as a conference paper at ICLR 2026
Table 5: Effectiveness of different future-aware masking strategies v: Consistent performance improvement across all benchmarks for the task. x: Performance degradation across all benchmarks. "_": Mixed results, some benchmarks improve while others degrade.
Mask
Temporal Multi-Image Tasks
Sematic Multi-
Image Tasks
Needle In a Haytack
T-1
T-2
T-3
T-4
S-1
S-2
S-3
S-4
S-5
N-1
N-2
M
X
X
M
/
M
Figure 7: Performance comparison of three causal masks (M', M M and their lightweight merged variants across different model architectures (v1.5-Vieuna, v1.6-Mistral) and sizes (7b, 13b).
DISCUSSION
In this section, we address the questions posed in Section mental results. We further provide insights into these issues and explain how future-aware semantic design can support the development of vision-language models (VLMs).
I throuuhh empirical analysis and experi-
1. Causal attention from LLMs may not align well with vision tokens in VLMs and limits their contextual capacity. Table 5 shows that relaxing the standard causal mask with future-aware strategies (Definitions 3.1 3.2 yields selective improvements across benchmarks, rather than uniform gains. Temporal likely because they require modeling event sequences, spatial localization, and counterfactual changes over time. In these tasks, allowing visual queries to access future visual cues helps encode scene dynamics and long-term dependencies. Similarly, visual-relation tasks like S-2 and S-3 show gains under v and M suggesting that self-attention among visual tokens (e.g., for spotting subtle differences) or previewing embedded text (e.g., for reading labels in diagrams) enhances fine-grained reasoning. However, text-dominant tasks (S-5, IR), these relaxed masks often degrade performance, confirming that strict autoregressive masking remains essential for textual alignment and matching.
tasks (T-1 to T-4) consistently benefit from M and M
2. Causal attention could be revised by selectively relaxing future masking for vision tokens. Definitions 3.1 3.2 3.3 introduce new masking strategies that modify the upper-triangular part of the causal mask. Instead of blocking all future tokens, these strategies allow visual queries to access selected future tokens: M keeps all future tokens visible, M/220 keeps only future visual tokens, and 22 keeps only future text tokens. As shown in Table these changes improve performance in tasks that involve visual reasoning or temporal understanding. The results suggest that strict causal masking, designed for text, may be too limiting for vision. Allowing future attention in a controlled way helps vision tokens gather important context early, and better aligns the attention pattern with how visual information is structured.
221
3. Vision tokens could attend to either or both visual and textual tokens based on task needs. The three masking strategies deñned Definitions 3.1 3.2, and 3.3 specify which types of future tokens visual queries may attend to. The expermental results in Table 4 and Figure reeeeal that the optimal access pattern depends on the nature of the task. For visual relation inference (e.g., visual change caption, visual rela- tion expression), v2 performs best, reasoning relies capturing spatial or temporal relationships between future visual observations. In contrast, text-centric tasks like OCR-VQA and TextVQA benefit more from 2 where visual tokens preview future textual content to interpret embedded text. Mcanwhile, M enables broad access to both modalities and helps in temporally grounded multi-image tasks. These findings suggest that causal masking could be
8

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of future-awar attention strategies with and without merging. Prefill Valid Attentions counts mon-masked attention scores. L: The length of the prefil attention. m/n: the number of visual/textual tokens.
Attention Type
Prefill Valid Attentions
Decoding Latency
M
LLL 1)/2 + mL - m(m + +1/22
83,1783 ms/token
M +merge
11/+
26.5362 ms/token
M
L(L + 1)/2 + m(m - 1)/2
64. ms/token
2 +merge
L(L + +1)/2
26.4037 ms/token
M
L(L + 1)/2 + mn L(L + 1)/2
43,0362 ms/token
M 22 +merge
26. 1051 ms/token
flexibly adapted: vision tokens could be granted selective access to future visual or textual information based on the modality relevance of the downstream task.
4. Pre-seen visual semantics show task-dependent benefits. Allowing visual tokens to preview future content helps in tasks that rely heavily on intra-visual reasoning (e.g.. Visual Change Captioning and Visual Relation Expression), while future text access proves more beneficial for text-dominant tasks (e.g.. OCR-VQA, Tex(VQA). As shown in Table 2 and Figure relaxing visual-to-visual constraints using 2 leads to notable gains in visual relation benchmarks, where understanding visual relationships across multiple frames regions is essential. Conversely, Table Figure demonstrate that M significantly boosts performance in text-rich visual QA tasks by letting visual tokens access future text cues early in the decoding process.
Building on the findings from future-aware masking strategies, we further provide some insights on the method of merging future attention into the past region.
1. Merging future attention in the prefill stage could enjoy a trade-off' of performance and latency in the latter decoding stage. Figure 7 shows that merging pooled future attention into early prefix tokens retains most of the performance benefits offered by future-aware masks. Meanwhile, Table quaaticaivie demonstrates that merging significantly reduces decoding latency (As breaking the structure precludes computing attention scores via concatenation over past keys/values, ate performance without KV cache for the no-merge variant. ). Compared to the unmerged versions, applying merge leads to a reduction from 83.18 ms/token (MF) to 26.53 ms/token (M /merge)) from 64. 13 ms/token (2 to 26.40 ms/token ( +merge), and from 43.04 ms/token (M"2) to 26.10 ms/token 2M merrge). The 2-3x speedup stems from the fact that merged models rely solely on standard causal decoding, avoiding the overhead of computing extra future attention.
evalu-
2. Future semantics be utilized by merging them into attention sink regions in the past. To evaluate this,
40.5
40.0
we define the prefix ratio as prefix size/ L, where L is the total attention length, and the prefix size refers to the num-
39.5
39.0
38.5
ber of past tokens into which the pooled future attention scores are merged. Figure SouureBsshor showss that as the prefix ratio
7B Ar
38.0
gm
37.5
increases, attention to future tokens decreases, indicating that future information can be compressed into earlier to- kens through pooling. This preserves the autoregressive structure while enabling the model to access future seman- ties indirectly. The prefix acts as an attention sink that gathers and retains useful signals for subsequent genera- tion. Interestingly, we find that merging the pooled future scores into just the first token already leads strong results, suggesting that single well-positioned sink token is often sufficient absorb and propagate future context effectively.
37.0
36.5
138
36.0
10
20
30
50
Prefix Ratio (%)
Figure 8: Effecc of prefix ratio of Light future aware attentions.
6 CONCLUSION
In this work, we revisit causal attention in lasion-language models (VLMs) and show that the standard left-to-right masking used in language models often misaligns with the characteristics of visual inputs.

---

Published as a conference paper at ICLR 2026
We conduct detailed empirical study across 15 multimodal tasks and propose three future-aware causal masking strategies that selectively expose future tokens to visual queries. These strategies lead to clear improvements tasks requiring temporal, relational, or text-based reasoning. We also introduce a lightweight attention mechanism that compresses future attention into prefix tokens during prefill, preserving decoding efficiency while enhancing context modeling. We further analyze the root cause of the misalignment and provide insights that improve the understanding and design of modality aware causal attention in vision-language models.
ETHICS STATEMENT
This work studies inference-time causal masking for decoder-only vision-language models. The research does not involve human subjects, annotation efforts, newly collected data. All tasks used in our evaluation come directly from MILEBench, a publicly available long-context multimodal benchmark suite. The benchmark aggregates existing datasets, including VQAv2, GQA, TextVQA, OCRBench, and Spot-the-Diff, under unified evaluation protocols. We use these datasets strictly under their respective licenses, and to our knowledge, none contain sensitive personal information. Our method modifies only the inference-time attention mask of open-source pretrained models without introducing new training or supervision. We encourage responsible and privacy-aware usage of this work and discourage applications that violate safety, faimesss ethical norms.
REPRODUCIBILITY STATEMENT
We aim to make this study fully reproducible. The proposed masking mechanisms, including M1 2 M222, merge-based variants, and baseline causal masks, are defined in the main text and appendix. All experiments rely on public MILEBench tasks, which provide standardized long-context and multimodal evaluation settings consolidated from VQAv2, GQA, TextVQA, OCRBench, and Spot-the-Diff. Dataset preprocessing, multimodal tokenization, prompt formats, and evaluation metrics specified in the main paper.
All models are evaluated using the Hugging Face trans forers rmers library version 4.34.7, which is compatible with CUDA 12.4 and avoids conflicts with F alashhtteent inn kemelss. We publicly released checkpoints of L.LVV L-BBB and LLaVA-13B without any additional training. Hyperparame- ters such as decoding settings, maximum context length, and mask-application rules remain fixed across all experiments and are reported in the appendix. Since our method is inference-only and preserves the original architecture, all results can be reproduced by applying our mask implementation to the released checkpoints. We will release our full implementation, configuration files, and scripts upon publication to support independent verification.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally your phone, arXiv preprint arXiv:2404.1422 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar 2rXiv:2303.0877 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502: 2025.
10

---

Published as a conference paper at ICLR 2026
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishman, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1161-116311, 2020.
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In European conference on computer vision, pp. 170-188. Springer, 2022.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 16495-16504, 2022.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with sounce suites. Science China Information Sciences, 67(12):220101, 2024a.
The Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie visual- visual-linguistic tasks. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 24185-24198, 2024b.
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang. Bo Zhang, Xiaolin Wei, et al. MobilevIm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint 1.. 2023.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01800 2023.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint 20X11:2410.10781, 2024.
Mehrdad Hosseinzadeh and Yang Wang, Image change captioning by learning from an auxiliary task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2725-2734, 2021.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 709-727. Springer, 2020.
Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning describe differences between pairs of similar images. arXiv preprint arXiv: 1058,4 2018.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 4999-5007, 2017.
Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, and Marinka Zitmik. Token reduction should go beyond efficiency in generative models-from vision, language multimodality. arXiv preprint arXiv:2505. /8227, 2025.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruy Batra, and Stefan Lee. Beyond the nav nav-graph: Vision-and aad--anddlanggg languge navigation in continuous environments. In Computer Vision-ECC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pp. 104-120. Springer, 2020.
11

---

Published as a conference paper at ICLR 2026
Chia- Men--enneee Kuo, Sijie Zhu, Fan Chen, Xiaohui Shen, and Longyin Wen. D-attn: Decomposed attention for large vision-and-languu model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 23935 229444.. 2025.
Aonian Li, Bangwei Gong, Bo Yang. Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint wrX.v:2501.08 13, 2025.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Xingyuan Li, Yang Zou, Jinyuan Liu, Zhiying Jiang, Long Ma, Xin Fan, and Risheng Liu. From text to pixels: a context-aware semantic synergy solution for infrared and visible image fusion. arXiv preprint wwXiv:2401.0042// 2023b.
Xingyuan Li, Jinyuan Liu, Zhixin Chen, Yang Zou, Long Ma. Xin Fan, and Risheng Liu. Contourlet residual for prompt learning enhanced infrared image super-resolution. In European Conference Computer Viston, pp. 270-288 Springer, 2024.
Yongqi Li, Wenjie Li, and Liqiang Nie. Mmcoqa: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 4220-4222 1200,4231,200 2022.
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26689-26699 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2629662660660 220066 2024a.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.01244 2024b.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 2200-2209, 2021.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
Xiaohuan Pei, Tao Huang, and Chang Xu. Cross-self kv cache pruning for efficient vision-language inference. arXiv preprint arXiv:2412.04622 2024.
Jianing Qi. Jiawei Liu, Hao Tang, and Zhigang Zhu. Beyond semantics: Rediscovering spatial awareness in vision-language models. arXiv preprint arXiv:2503. 17349, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
12

---

Published as a conference paper at ICLR 2026
Konstantin Schall, Kai Uwe Barthel, Nico Hezel, and Klaus Jung. Gpr1200: a benchmark for general- purpose content- based image retrieval. In International Conference on Multimedia Modeling, pp- 205-216. Springer, 2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 15638- 15650, 2022.
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milms in long context. arXiv preprint arXiv:2404, 18532, 2024.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.060) 2021.
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13636-13645, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023.
Feng Wang, Timing Yang. Yaodong Yu, Sucheng Ren, Guoyizhe Wei, Angtian Wang. Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Adventurer: Optimizing vision mamba architecture designs for efficiency. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30157-30166, 2025.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. arXiv preprint arXiv:2405.09711, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv. 0rg/abs/2309.174555 pp. 1, 2024.
Shuai Yang. Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Sced-story: Multimodal long story generation with large language model. arXiv preprint arXiv.2407.08683, 2024.
Xixuan Yang, Xin Huang, Chiming Duan, Tong Jia, Shandong Dong, Ying Li, and Gang Huang. Enhancing web service anomaly detection via fine-grained multi-modal association and frequency domain analysis. arXiv preprint arXiv:2501. 16875, 2025.
Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for rision-languase tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9847-9857, June 2021.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive eogguage-imag pre-training. arXiv preprint arXiv:2 1.07783, 2021.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint /9Xiv:1910.0144 1900.01442, 2019.
Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. Sta- blemask: Refining causal masking in decoder-only transformer. arXiv preprint arXiv:2402.04779, 2024.
13

---

Published as a conference paper at ICLR 2026
Chulhee Yun, Srinadh Bhojamapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077, 2019.
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision- lasion-laguuag-acttion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1702-1713, 2025.
Yang Zou, Zhixin Chen, Zhipeng Zhang, Xingyuan Li, Long Ma, Jinyuan Liu, Peng Wang, and Yan- ning Zhang. Contourlet refinement gate framework for thermal spectrum distribution regularized infrared image super-resolution. arXiv preprint arXiv:2411.12530, 2024.
14

---

Published as a conference paper at ICLR 2026
A TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
A.1 EXPERIMENTAL SETUP
All experiments were conducted on NVIDIA A100 GPUs using the official implementation of LLa VA series Liu et al. (2024a), with FlashAttention-2.6.22 inteerraee for efficient attention computation. The context length was set to 4096 tokens, and all generations were performed using greedy decoding with fixed temperature of 0 to ensure deterministic outputs. Task definitions and groupings follow the standard taxonomy established by MILEBench [Song et al. (2024), covering a diverse spectrum of 29 multimodal benchmarks. To accommodate the memory and sequence length variability across datasets, batch sizes were dynamically adjusted: a batch size of I was used for long-context datasets such as MMCoQA [i et al. (2022) and GPRI200 Schall et al. (022), while batch size of 24 was adopted for the remaining tasks. For tasks with highly imbalanced attention patterns, we applied kernel-base attention merging strategies with top-k region ratios calibrated per dataset. We further incorporated minor task ssecifp biases- foreees-foraae example, a fixed bias of 0.5 in EgocentricNavigaatonn Krantz et al.) (2020) and 1.5 in VQA Tanaka et al. (2023) retaining default configurations elsewhere. All preprocessing and evaluation followed the official MILEBench protocol to ensure fair and reproducible comparisons.
Table 7: Detailed Tasks inherited from MILEBench Song al. (024).
Campury
Localaation Acticn Prediction
N0N
Activ Unskentzraling and Pradiction (T-1)
STA
200
Teuppeal Malii imar:
Actore Seppenee
Unpeat ESca1sess
CLEYZ 116
STAM Wirer
21033
Ohtjas Unukestarding (T-2)
Ofject Aarlone
CLEVER
200
Anarray
Cbredt Staffie
Pacepion You [uinnen
2004
Visual Novigation and Spotial Localzation (T-3)
Esweercx Narqaaors
Mostng Tirrctus
VLATS Syptiesa FR
CLIVVITT
200
Accuracy
Conttrfactual Reaonring and Stote Change
Claracis Order
Suze
Percepion Tes (1000n0----
Tca
R004
200
Stzre Terecioon
MinsaNe (Tc b120
Wabpage CA
Watuy (122 2818 K3264
Kevosedgy QA (5-1)
QA
Andrray
Semariic image
Long Toe with Imuuzs
Maad
Tou-Rich Images QA (5-2)
OCR
OUR vo Listir hiu
200
Avauaay
Dreument
DecVON SranAAAaaarerrra -11 PA1
Visaal Relatio Luference (S:3)
Varal Belannddig Rxpposdnng
Spo-dic
ROUGE- 200
Dallgaee (5-4)
Corvestiied Eajodded Dialogse
Meltinztl Dalngg
ALIREE Sh Thoorrss in posll
Malt in maa sele
200
Acarracy
Space Unddermnnng
BuScuus ae soort
200
TeaN
Diggoonic Exaltation
Nesile Hopanck (N
Hasstok
330
Iruge Retileal (1-1)
Trage Retional
GPRI27 Schall et (022
600
Aocuraus
A.2
RELATED WORK
With the success of decoder-only large language models (LLMs) Achiam et al. (2023) ; Bai et al. (0023); ; Li et al. ): [ouvron et al. (2023): Abdin et al. (2024), recent advances have extended their capabilities to the multimodal domain, giving rise to Vision-Language Models (VLMs), Early frameworks such as LLaVA Liu et al. (2023), VnternVL Chen et al. (2024b), and Qwen- VL Bai et al. (2025) demonstrate that instruction tuning can be adapted to handle flatted textual and visual inputs, enabling strong performance across tasks such as visual reasoning, captioning, and instruction following. Additionally, most multi-modality wre-trained work [Ln et al. (2024); y |Bii et al. (2025);
ei al. (2023b 2024); Zou et al. (2024); Yang et al. (2025) also inherit the causal masking design from LLMs, which, while crucial for token generation, may unnecessarily constrain specific modality token (e.g. In our paper it refers to visual tokens). These pre-trained models typically flatten visual and textual tokens into single sequence and feed them into an decoder-only. which may overlooking modality-specific attention patterns. To mitigate these limitations, recent studies have explored resolution-aware vision encoders CCai et al. (2022); [Cen et al. (2024a), multi-modk alignment modules (iighh et al. (2022). and fine-grained token interaction strategies Yao et al.) (2021) aiming to better adapt LLsebased decoder-only architecture to the visual perception reasoning. And the potential usage of the future tokens has been shown in LLMs architecture Yin et al. (2024). Beyond these lines of research, recent studies such as D-Attn Kuo et al. (2025), BLIP-2 et al. (2023a) and Adventurer Wang et al. 2025) explore architectural or encoder-based based mechanisms to relax modality
https://githiee cow.b//.oomaaaaa All.-All.a///aa alahhaatthattttion
15

---

Published as a conference paper at ICLR 2026
interactions, whereas our approach differs by keeping the standard decoder-only backbone unchanged and introducing training-free free, aware masking to systematically study how visual tokens should access future context during autoregressive inference. To accelerate inference, dynamic pruning has also been widely adopted in Vasion-language and multimodal large models, as discussed in recent surveys of token-reduction techniques Kong al. (2025). In the robot Vision-Languagee-Actii (VLA) domain, models such as CoT-VLA Zhao et al. (025)) further demonstrate that adjusting causal masks for action tokens can enhance long-horizon control, highlighting the broader importance of modalittyspeciif causal relaxation. However, the impact of ALM--inherited causal masking on visual token processing remains underexplored and despite its potential misalignment with the non- sequential nature of many visual reasoning tasks, which motivating the core investigation in our work.
A.3 ABLATION UNDER INTERLEAVED IMAGE-TEXT TOKEN PATTERNS
Beyond the conventional image text input pattern, some embodied benchmarks adopt interleaved token structures such as image + text + image + text. To examine whether our future-aware masks generalize these settings, we evaluate several mask variants the ALFRED /Shridhar al. (2020) benchmark, whose latent sequence composition exhibits such mixed ordering. In this configuration, visual tokens can still access their future textual or visual context through M f A222 and M 22t because our design only depends on each token's 's modality identity and its causal location within the sequence. The results are summarized in Table All future-aware variants outperform the original causal baseline, and the kernel- based future mask yields the largest improvement, suggesting that smooth future aggregation is beneficial for multi-step, temporally structured reasoning tasks.
Table 8: Future-aware masks under interleaved image-text sequences (Rouge-L Fl x100).
Mask Type
Open Tokens for Vision
Rouge-L F1
A vs Original
Variants
Future-only M
vision future text/vision future text only
15.52
40.23
improved grounding from future cues
Future V2T 2
15.61
+0.32
strongest non-kemel variant best temporal aggregation
Kernel Future
kernel-smoothed future access
17.11
+1.82
A.4 TUNING POTENTIAL OF FUTURE-AWARE MASKING
Our main objective is to analyze how causal masking influences visual tokens in decoder-only architectures. For this reason, most of our experiments focus on inference-only mechanisms, allowing us to isolate the effect of future visibility without altering optimization dynamics. Nevertheless, to examine whether the benefits also extend to the training stage, we additionally evaluate two lightweight tuning experiments that reflect the reviewer's 's concern.
Fine-tuning a Decoder-only VLA Model. We fine-tune a decoder-only laaion-languag--cttino action model (OpenVLA-OFT) on LIBERO-Spatial for 30k steps under both causal and future-aware masking. As shown in Table 9 the future-aware model achieves higher success rate (85.1% % vs. 84.7%) and its query-key correlations exhibit the same upper- ppier-tria structure observed in our inference-one analyses. This demonstrates that the benefit of granting visual tokens controlled access to future information not restricted to probing, but also manifests during training.
Table 9: Fine-tuning results on LIBERO-SE -Sptial (success rate).
Method
SR (1)
Diffusion Policy (scratch)
78.3 1: 1.1%
Octo fine-tuned
78.9 1: 1.0%
Open VLA + Causal
84.7 0.9%
Open + Future-aware
85.1 = 0.9% %
Learnable Mask Selection via Adapter. We further test simple learnable controller that selects masking strategies using lightweight MLP-basee adapter. Although this introduces additional FI. FLOPs, the adapter consistently improves performance across CLEVR, Nay, Object, and SpotDiff
16

---

Published as a conference paper at ICLR 2026
These results indicate that future-aware masking be integrated into differentiable routing modules and remains compatible with training- time adaptation.
benchmarks (Table 10
Table 10: Learnable adapter for mask selection (aceuracy).
Method
CLEVR
Nav
Object
SpotDiff
Original Mask
0.166
0.310
0.225
0.162
Adapter
0.188
0.320
0.240
0.173
Discussion. Together, these tuning experiments show that future-aware masking is not only effective as an inference mechanism but also benefits models when training is allowed. While our paper centers on analyzing causal behavior in existing LVLMs, these results highlight the potential of training- time designs as an exciting direction for future work.
A.S DISTRIBUTION ANALYSIS FOR FUTURE- AWARE ATTENTION.
To better understand the limitations of causal masking in vision-languaa models (VLMs), analyze the predictive uncertainty from an information-theone examine the mutual information between the model output and the observed context under different masking strategies. Let x" {-ī .... x and x = { x. denote the visual and textual tokens respectively, and let X x x" be the unified input sequence of total length L m + The autoregressive model predicts the output token z, based on masked prefix Si.. The mutual information between the output and its visible prefix context is T(X <x; = H (Iro) H Xxs),, where (-) denotes the entropy. Depending on the specific causal mask, the prefix Xgi may include different subsets of visual and textual tokens. For example,
perspective, following the previous work Yin
et al. (024)). In particular,
[xx:: xi.; U Ux(,z) H (xx) H ( x xf.)).
(15)
which isolates the contributions of each modality. As shown in our empirical study in Section and 4 breaking the visual- based causal inference procedure by exposing future tokens leads to distribution shift because of the rich semantic information in the masked future region.
Theoretical Properties. Based on the preceding information -oommation-thooretic derivations in Yin et al. 2024 Yun a (219)), and assuming the vision language models induce causally isotropic intermediate representations, we further derive the following properties of mutual information:
Property A.1. For any S L, the mutual information between the output token xo and the I-th layer (1) intermediate representation w Si is upper-bounded by the mutual information from the raw prefix input X ssi
(1 < <1<x::
This follows from the data processing inequality and reflects that internal representations cannot increase information about the larget beyond what available from the inpul.
Property A.2. If the VLM decoder is contextual, then its intermediate representation preserves all information in the input:
T(x:; ( T X
This implies that the decoder faithfully encodes the entire causal context without losing predictive power.
Property A.3. If the input distribution is causally isotropic and w is uniquely determined by X then the representation retains no more information than the original prefix:
I ()) & [xxo: X for all L.
This reinforces that isotropic settings do not amplify mutual information through intermediate computation.
Property A.4. If both the decoder is contextual and the data distribution is causally isotropic, then the mutual information is exactly preserved:
Ixx ( (er.).
This guarantees loss of information between the raw prefix and the intermediate representation.
17

---

Published as a conference paper at ICLR 2026
Property A.5 SUpper-Trianuoe Future Visibility in Multimodal Masking). For any visual query position E V and ground truth output Zo. the ratio of mutual information satisfies:
H(xx) (
S To)
(16)
I(XcLiIo)
H(I,) H (2. X X
I(2 :o)
(1)
where S2 represents the intermediate representation at layer I computed from prefix X This inequality quantifies how future-aware visual masking contributes to reducing uncertainty of the output, and suggests that semantically rich upper-triangle access allows earlier layers to preserve more predictive information.
Property A.5 establishes that, under future-aware masking strategy, visual queries that access upper-triangular regions (i.e., future tokens) can retain a higher fraction of mutual information with the output token To compared to standard causal masking. This suggests that even partial access to semantically informative future tokens allows intermediate representations to encode more predictive context. The inequality further implies that the proportion of retained information in early layers is lower bounded by the proportion of information retained in their corresponding representations (2()), In practice, this supports the design of selective future access in vision Asionnlanguage inference, where relaxing strict causality on visual queries can effectively enhance downstream prediction without fully compromising autoregressive generation.
A.6 FUTURE-AWARE
To efficiently support our future-aware causal masking strategies defined in Section 3 we integrate them with the FlashAttention framework for scalable inference. As detailed in A Algorithm we implement our masking logic by applying the selected future-aware mask ju E {MJ M AMM,M22: M 22t directly into the attention score computation, replacing the standard causal mask. During runtime, both the queries and key-value pairs are processed blocks to fit within on-chip memory, and attention scores computed with fused softmax operations to ensure numerical stability and memory efficiency. The masked attention scores are exponentiated and normalized via a log-sum-exp trick, and aggregated token-wiss to produce final outputs. This fusion enables proposed vision- language attention design to retain the efficiency advantages of Fltsttntention while supporting flexible, modality-aware causal constraints.
A.7 ROBUSTNESS OF THE KERNEL POOLING MERGING
We conduct further ablations to evaluate the robustness of the ID kernel pooling module. The results show that the module behaves consistently across a wide range of kernel sizes. As summarized below, kernel sizes of 1, and 7 yield identical performance, while extreme kernel size of 25 introduces only a minor fluctuation within +.0.1. This indicates that the method does not rely on shortcut patterns and preserves stable reasoning behavior.
OCR (LLaVA v1.5 v1.6)
VRE (LLaVA v1.5
Pool Size
v1.5
v1.6
v1.5
v1.6
23.0
21.5
17.9
15.5
3
23.0
21.5
17.9
15.5
7
23.0
21.5
17.9
15.5
25
23.0
21.4
17.9
15.4
We additionally compare mean pooling and max pooling under the same configuration. Both yield comparable results, with mean pooling showing only negligible differences (< 0.2) on LLaVA v1.6 variants. This further confirms that the ID kernel pooling module insensitive to the choice of pooling strategy. These results demonstrate that the kernel pooling mechanism is both reproducible and robust across kernel sizes and pooling types. We adopt max pooling in the main paper due to its consistently stable performance.
18

---

Published as a conference paper at ICLR 2026
Algorithm Future-Aware Mask equipped with FlashAttention
Require: Matrices Q, K, V,M E gixd future-aware mask E {M M 22 _M22J, block sizes
B,, Be
1: Divide Q into Т, = - B, blocks QI...., Qr.
2: Divide K, V,
l , into T
blocks K2, v My of size Be each E R.d and L E R
3: Initialize output
4: for = 1 to T, do
5: Load Q; into on-chip SRAM
6: Initialize ( 0, (0)
O, m (0) 000
7: for j = to T. do
8: Load K,. j Hy into on-chip SRAM
9:
Compute masked attention score:
5 Q,K, /vd+ Mi
Normalize: 8( exp(s!) -m (())
Update max: m (5) max m (5-1) mmx (1) dim 1))
11:
Update sum: g(0) = exp(m (1-1) (
m o e إ + 55(
12:
13:
Output partial result:
0 exp(m, (j-1) m ( 0 1ー + + v,
14:
Final output:
0 = o (T)) (T)))
Logsumexp: L, m (T.)
+ log (o (2a)
15:
Store O,, L to global memory
17:
return L,
OCR
VRE
SpotDiff
Method
v1.5
v1.6
v1.5
v1.6
v1.5
v1.6
Max
23.0
21.5
17.9
15.0
16.5
15.3
Mean
22.8
21.4
17.9
15.0
16.5
15.2
B THE USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we employed Large Language Model (LLM) writing assistant to refine the clarity and presentation of the text. Its use was strictly confined to linguistic enhancement rather than substantive content generation. Specifically, the LLM was used to:
Rephrase sentences and paragraphs for greater readability, conciseness, and academic formality.
Correct grammar, spelling, and punctuation errors.
Improve logical flow and transitions between sentences.
Limitation. Our analysis primarily follows the image - text token ordering commonly adopted in decoder-only LVLMs. While this setting covers the majority of existing architectures, does not explicitly evaluate alternative tokenization patterns such pure text - image sequences. This mismatch may limit the generality our observations cases where visual tokens always appear after textual tokens. Nevertheless, the applicability of our future-aware masks is not tied to fixed modality order. For mixed or interleaved patterns (e.g., text-image), any visual token that followed by subsequent tokens still benefits from the same visibility logic, since the mask operates solely on modality identity and causal position rather than on a specific interleaving template.
19


Reviews:
[{"review_id": "xjjc1mLjSE", "reviewer": "Reviewer_TbBr", "summary": "This paper presents a systematic investigation into the role of causal attention in VLM. The authors identify a key mismatch: the strict left-to-right causal masking inherited from text-only models constrains visual tokens, which naturally lack sequential order. To address this, they introduce a family of future-aware causal masks that selectively relax causal constraints, enabling visual queries to attend to future visual (M^{v2v}), textual (M^{v2t}), or both types of tokens (M^f). The paper further proposes a lightweight variant that encodes future-attention information into a prefix during prefill, balancing performance gains with efficient autoregressive decoding. Extensive experiments on 15 multimodal benchmarks show consistent, task-dependent improvements, providing new insights into the design of modality-aware attention mechanisms for VLMs.", "questions": "1. For the first weakness, could the authors further clarify the main novelty of this proposed work compared with other existing studies? \n2. In Table 4, the M^f variant, which combines both M^{v2v} and M^{v2t}, does not consistently achieve the best performance. Could the authors elaborate on why integrating both future visual and textual signals does not yield additive benefits? Some discussion on possible interference or redundancy effects would be valuable.\n3. Following Question 2, if M^f or M^f_merge does not consistently achieve the best performance across all tasks, how should one decide which variant to use during inference when the task type is unknown? Is there a possibility for the model to automatically adapt or select the most suitable masking strategy based on the input modality or context?\n4. It would be interesting to understand how the proposed future-aware masking mechanism generalizes to video-like or multi-frame visual inputs, where temporal order becomes more prominent. Would the same masking strategies still apply, or would additional temporal constraints need to be considered?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies a fundamental problem that the misalignment between text-oriented causal attention and the non-sequential nature of visual processing in VLMs. \n2.  The authors conduct a large-scale, systematic analysis of multiple future-aware causal masking strategies (M^f, M^{v2v}, M^{v2t}) across diverse multimodal benchmarks. The clear, task-dependent findings (e.g., M^{v2v} for visual reasoning, M^{v2t} for text-rich QA) provide concrete and interpretable insights.\n3. The paper is well-written and logically structured, with precise definitions, clear experimental setups, and insightful discussions that effectively connect results back to the core research question.", "weaknesses": "1. It would be helpful for the authors to clarify how the proposed future-aware masking strategy differs from existing approaches that also leverage bidirectional attention or cross-attention in vision-language or multimodal models. Specifically, how does this method compare conceptually and practically to (1) prior works that implement fully bidirectional attention over visual tokens[1], or (2) models that achieve modality alignment primarily through cross-attention mechanisms[2]? A discussion or empirical comparison would strengthen the contribution and highlight the unique aspects of the proposed approach.\n\n2. The proposed method primarily evaluates on interleaved text-image input sequences. However, in practice, many multimodal inputs follow different ordering patterns, such as text → image → text, or multiple consecutive text tokens followed by an image. It would be valuable for the authors to discuss how sensitive the future-aware masking strategies are to such variations in input order, and whether the same masking mechanisms can generalize effectively to these alternative sequence structures.\n\n3. The proposed \"light\" method uses kernel pooling to compress future attention and merges it into the first token based on the \"attention sink\" phenomenon. This mechanism is somewhat heuristic. The paper could be strengthened by exploring more sophisticated or learnable compression techniques to dynamically determine what context is most important to merge and where to merge it.\n\n\n[1] Kuo, Chia-Wen, et al. \"D-Attn: Decomposed Attention for Large Vision-and-Language Model.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.\n[2] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" International conference on machine learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ainx50znmj", "reviewer": "Reviewer_DpXd", "summary": "This paper revisits the role of causal masking in autoregressive Vision-Language Models (VLMs). While causal attention ensures sequential generation for text, the authors argue that its direct adoption for visual tokens is suboptimal because visual information is inherently non-sequential. Through systematic analysis, they demonstrate that relaxing causal masks for vision tokens—allowing selective access to future context—can improve performance in temporal reasoning, relational understanding, and text-rich visual tasks. They propose three future-aware masking variants (visual-to-visual, visual-to-textual, and full), along with a lightweight merging mechanism that compresses future information into past tokens to preserve autoregressive efficiency. Extensive experiments on 15 multimodal benchmarks show consistent gains across tasks, with improved reasoning accuracy and minimal latency overhead.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers a fresh and well-motivated perspective on how causal masking—originally designed for textual decoding—may be suboptimal for vision tokens. This conceptual rethinking addresses a fundamental assumption in current VLMs and opens up a new line of research on modality-aware causality.\n\n2. The proposed light future-aware attention introduces future context compression without retraining or architectural changes, adding negligible latency while delivering consistent gains.\n\n3. Experimental results show consistent accuracy boosts across 15+ benchmarks.", "weaknesses": "Actually I really like the insight this paper focuses on — questioning how VLMs can break free from the traditional causal attention inherited from language models. The idea is intuitively sound and genuinely interesting. However, given the paper’s current state, I cannot yet recommend acceptance. **I strongly encourage the authors to carefully revise the work, as it has great potential**. My main concerns are as follows:\n\n1. The paper currently provides an investigation and an inference-only solution, which reveals some of the limitations of causal masking but does not fundamentally resolve them. To convincingly verify the effectiveness of a non-causal design, the model should be adjusted (or at least fine-tuned) from the training stage, not just modified at inference.\n\n2. The results are not yet compelling, even though I believe the proposed approach can work. For instance, in Table 4, the overall performance of different LLaVA-7B and LLaVA-13B variants is quite close — no single attention structure consistently outperforms all others, which makes it difficult to draw firm conclusions.\n\n3. The idea of expanding receptive fields by merging future information into early tokens is not entirely novel; similar strategies have appeared in recent pure vision models [1].\n\n4. The paper should also discuss the connection between attention sinks [2] and the proposed approach. Prior studies have shown that language models naturally allocate disproportionate attention to early tokens. Could the proposed merging mechanism be implicitly benefiting from this effect? A more explicit analysis of this interaction would greatly strengthen the work.\n\n[1] Wang F, Yang T, Yu Y, et al. Causal image modeling for efficient visual understanding. 2024.\n\n[2] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with\nattention sinks,” arXiv preprint arXiv:2309.17453, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFzs1avM6j", "reviewer": "Reviewer_3gm7", "summary": "This paper investigates the suitability of standard causal attention masks, inherited from text-only LLMs, for autoregressive Vision-Language Models (VLMs). The authors argue that the strict left-to-right masking applied to visual tokens is an overly rigid constraint that prevents the model from leveraging valuable future context essential for visual reasoning. Through empirical analysis, the paper demonstrates that relaxing these causal constraints for visual queries can surprisingly improve performance. Based on this, the authors propose a family of future-aware causal masks that selectively allow visual tokens to access future context. To mitigate the computational cost of this relaxation, they also introduce a lightweight pooling mechanism to compress and merge future semantic information into past representations during the prefill stage, preserving the efficient autoregressive structure for decoding.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper compellingly identifies a fundamental misalignment between the sequential, autoregressive nature of LLM-native causal masks and the more holistic, non-sequential nature of visual information processing. I like the motivation.\n- The authors systematically evaluate different masking strategies ($M^f$, $M^{v2v}$, $M^{v2t}$) and connect their benefits to specific categories of vision-language tasks (e.g., temporal reasoning, visual relation, text-rich QA), providing a nuanced understanding of when and why future context is beneficial.\n- The findings are well-supported by experiments across several benchmarks, demonstrating consistent performance gains from the proposed future-aware masks and their lightweight variants.", "weaknesses": "- Lack the Ethics statement and Reproducibility statement in the main text.\n- The paper demonstrates that different future-aware masks (e.g., $M^{v2v}$ vs. $M^{v2t}$) are optimal for different tasks. This raises a practical question: how would a single general model choose the correct mask without a priori knowledge of the downstream task? The paper does not propose a dynamic or learned mechanism for this selection.\n- The experiments are conducted by modifying the LLaVA. It doesn't explore how these masking strategies might affect the model, such as Qwen-VL series or VILA.\n- The details of the ID kernel pooling and its integration could be explained in greater detail to ensure the reproducibility.\n- The paper's core idea of rethinking causal masks for specific modalities shares a conceptual similarity with other recent work CoT-VLA [1], which adapts attention for action tokens. This general direction of moving beyond rigid, text-native causal masking is necessary and very meaningful for the development of MLLMs. I am open for rasing my score if the authors can address my questions.\n\n[1] Zhao, Qingqing, et al. \"Cot-vla: Visual chain-of-thought reasoning for vision-language-action models.\" CVPR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBlb9AzeOb", "reviewer": "Reviewer_11Kz", "summary": "This paper challenges the default use of standard causal attention masks in VLMs. The paper empirically demonstrates that relaxing causal constraints for visual tokens can improve performance, whereas relaxing them for textual tokens severely degrades it. The authors then systematically investigate a family of \"future-aware\" causal masks: \n1. $M^{f}$ (Full Future): Allows visual tokens to see all future visual and textual tokens.\n2. $M^{v2v}$ (Visual-to-Visual): Allows visual tokens to see future visual tokens only.\n3. $M^{v2t}$ (Visual-to-Textual): Allows visual tokens to see future textual tokens only.\n\nTheir analysis reveals that these relaxed masks provide task-specific benefits: $M^{f}$ excels at temporal multi-image tasks, $M^{v2v}$ at visual relation inference, and $M^{v2t}$ at text-rich image QA. Recognizing that these masks introduce significant computational overhead during decoding, the authors propose a \"Light Future Aware Attention\" mechanism. This method computes the future-aware attention only during the prefill stage, then uses kernel pooling to compress this future semantic information and \"merge\" it into the initial past token representations. This approach allows the model to benefit from future context while retaining the standard, efficient causal mask during the autoregressive decoding phase. Experiments across 15+ benchmarks show this lightweight method achieves performance on par with or even exceeding the full future-aware masks, but with a 2-3x speedup in decoding latency.", "questions": "1. How do you train your VLMs? Do all settings follow LLaVA's 1.5 training process?\n2. Why don't you use an additional token, similar to an attention sink?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a simple, intuitive, yet largely overlooked problem: the fundamental mismatch between the strictly sequential causal mask and the non-sequential nature of visual data.\n2. The paper provides a thorough and systematic breakdown of the problem. The graphs and tables are clear.", "weaknesses": "1. The proposed method focuses on processing input tokens during the prefill stage. It's unclear how, or if, this concept could be extended to multi-turn dialogue where the future is truly unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)