Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
Authors: Dwip Dalal
Gautam Vashishtha
Utkarsh Mishra
Jeonghwan Kim
Madhav Kanda
Hyeonjeong Ha
Svetlana Lazebnik
Heng Ji
Unnat Jain
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal LLMs
Vision-Language Models
Fine Grained Visual Grounding
Image Warping
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We warp images using the model’s own attention so it “looks closer” at important parts, boosting accuracy without changing the model.

Abstract: Multimodal large language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perceptual grounding. We introduce AttWarp, a lightweight method that allocates more resolution to query-relevant content while compressing less informative areas, all while pre- serving global context. At test time, AttWarp closes a simple self-correction loop: the MLLM first produces cross-modal attention on the original image, which we use to rectilinearly warp the input and re-run the same frozen model, reallocating resolution toward regions it deems important without changing weights or architecture. This attention-guided warping preserves all original image information but redistributes it non-uniformly, so small objects and subtle relationships become easier for the same model to read while the global layout remains intact. Across nine benchmarks (TextVQA, GQA, DocVQA, POPE, MMMU, MIA- Bench, MMVP, RealWorldQA, BLINK) and four MLLMs (LLaVA, Qwen-VL, InternVL, and InstructBLIP), AttWarp consistently improves accuracy, strengthens compositional reasoning, and reduces hallucinations, outperforming four competitive baselines that manipulate raw images at test time. Together, these results show that attention-guided warping prioritizes information relevant to the query while preserving context, and that the same MLLMs perform better when given such warped inputs. The code and demos are available on the project page: https://dwipddalal.github.io/Attwarp/

Published as a conference paper at ICLR 2026
CONSTRUCTIVE DISTORTION: IMPROVING MLLMs WITH ATTENTION-GUIDED IMAGE WARPING
Dwip Dalal', Gautam Vashishtha?,
Utkarsh Mishra Svetlana Lazebnik',
Jeonghwan Kim',
Madhay Kanda', Hyeonjeong Ha²
Heng Ji
Unnat Jain'
1 University of Illinois Champaannn 4 University of California, Irvine
2 Skan Al
3 Texas A&M University
dvip2@i1linois. edu
ABSTRACT
Multimodal large language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors fine-grained perceptual grounding. We introduce AttWarp, a lightweight method that allocates more resolution query-relevant content while compressing less informative areas, all while pre- serving global context. At test time, AttWarp closes a simple self-correction loop: the MLLM first produces cross-modal attention on the original image, which we use rectilinearly warp the input and re-run the same frozen model, reallocat- ing resolution toward regions it deems important without changing weights or architecture. This attention- -uided warping preserves all original image informa- tion but redistributes it non-uniformly, so small objects and subtle relationships become easier for the same model to read while the global layout remains in- tact. Across nine benchmarks (TexIVQA, GQA, VQA, POPE, MMMU, MIA- Bench, MMVP, Real WorldQA, BLINK) and four MLLMS (LLaVVA, Qwen-VL. Intern VL. and InstructBLIP), AttWarp consistently improves accuracy, strength- ens compositional reasoning, and reduces hallucinations, outperforming four com- petitive baselines that manipulate images test time. Together, these sults show that attention-gun guided warping prioritizes information relevant to the query while preserving context, and that the same MLLMs perform better when given such warped inputs. The code and demos are available on the project page: https: https://dwipddalll .icallgiieee github. io/ Atttwap//
INTRODUCTION
Humans perceive certain regions of a scene by dynamically allocating high-resolution resources to areas of interest. This behavior is described as the interplay between foveal vision, which provides detailed perception at the center of attention, and peripheral vision, which rapidly scans the broader scene in lower resolution (Carrasco, 2011), 1). This warped way of perceiving our surroundings is dynamic and dependent on task demands. As Gibson argued (Gibson, 1966), perceptual systems actively restructure their input, sampling dense information where it's most needed. This introduces a form of distortion, not to obscure, but to enhance relevance.
While advanced deep learning models incorporate some aspect of this through attention mecha- nisms, they leave significant issues in fine- amn-ggrined perceptual grounding. Multimodal LLMs often fail to identify small details, distinguish between similar objects, and understand complex spatial relationships cluttered scenes, leading to misclassification and incorrect reasoning (Yang et al., 2024b; He et al., 2025; Kim & Ji, 2024). In this work, we investigate the benefits of this principle of warped perception in the context of modern multimodal LLMs. Particularly, we investigate the research questions: what is an effective method for warping images that preserve global context while expanding task-relevant regions? Would existing MLLMs perform better with warped images?
To answer the first research question, devise lightweight recipe for warping images that preserves Attention-Guided global context while expanding task-relevant regions. Our method, Attention Guided Image Warping (AttWarp), operates alug-and-play enhancement, requiring modifications the underlying

---

Published as a conference paper at ICLR 2026
"On the right desk; what left the laytop?"
Queestion:
Marginal Profks
Questions "Onn right desk, what to the left of laplop??
Attenttoo Aggregation from Language Decoder
Rectilinear
Warping
Original Imagee
Inttial Grid
Atteation Soore Matrix
W Warped Grid
Warped Image
LLaVi *Io the bookss deel.""
Marginalized Attention Scores
AttWoopp 'There the left
Figure 1: AttWarp overview. Given a query, our method extracts cross- modal attention maps from the MLLM's language decoder, aggregates them into marginal attention profiles, and uses rectilincar warping to expand high-attention regions while compressing low-attention areas. The warped image is then processed by the same MLLM, which now produces the correct answer.
MLLM architecture. As illustrated in Fig. 1, given an input image and query. we first extract cross- modal attention maps from the MLLM's language decoder. These attention maps are aggregated into Atlention Score Matrix and further condensed into ID marginal attention profiles along both horizontal and vertical axes (Fig. 1, middle). These profiles quantify the importance of each row and column in the image, with taller bars indicating regions that should receive more visual real estate. These marginal profiles guide our novel rectilinear warping process, which non-uniformly resamples the image grid, expanding high-attention regions and compressing low-attention areas, as shown by the red-to-blue gradient in Fig. (right). Crucially, our choice of warping ensures that all original image information is preserved, maintaining global context, unlike methods that crop or mask parts of the image. In the warped image, task-relevant objects such as the lamp and laptop highlighted by the green bounding box are visually enlarged, making fine-grained details and spatial relationships more accessible to the MLLM. Towards the second research question, we find that such warped images, when processed by the same MILMM, lead improved performance across nice multimodal benchmarks, and the idea generalizes to multiple MILM backbones. We empirically validate that our rectilincar design is crucial to this improvement without changing a single parameter of the MILM. We extend this framework in two directions: repeating this attention- suuiidd warp as an iterative self-correction step yields further Wtttaar Chain), and we learn distilled model to directly predict marginal profiles instead of estimating them from MLLM attention maps -Dittipp
Conceptually, AttWarp utilizes attention to modify the input image itself, different from the typical use of attention to reweight latent features. Importantly, our contribution is complementary to research that improves attention mechanisms in MILMss, such refining attention heads (Bi et al., 2024; Kang et al., 2025), adding auxiliary objectives (Yan et al., 2024), or redesigning cross-mode modal fusion layers (He et al., 2025). Finally, note that we intervene before feature extraction, while the above methods operate after the image has already been encoded, often from features that have already lost critical spatial detail (Pantazopoulos et al., 2024).
In summary, our key contributions are: 1) A lightweight method AttWarp that addresses the fine- grained perceptual grounding challenges identified multimodal L.LMs by modifying input images before feature extraction, requiring no MILM finetuning. 2) Consistent empirical gains over four competitive baselines, across nine standard vision-language benchmarks that test diverse capabilities such as fine-grained multimodal understanding, compositional reasoning, and hallucination mitigation. 3) Demonstrating generalization across multiple MLLM backbones and attention sources, underscor- ing its plug-an -nd---dd-pll compatibility. 4) Rigorous analysis validating that our warps indeed expand task-relevant regions and AttWarp's 's rectilinear design helps preserve the original data distribution.
RELATED WORK
Perception challenges in MILMS. MLLMs such as Flamingo (Alayrac et al., 2022), LLaVA(Liu al., 2023b), Qwen (Yang al., 2024a), MiniGPT-4 (Zhu al., 2023), and GPT-4V (OpenAI, 2023)
2

---

Published as a conference paper at ICLR 2026
have advanced image-gronee grounded dialogue and reasoning (Mitra et al., 2024; Dong al., 2024; Zhang et al., 2025a; 2024b). Yet they still struggle fine-graiieeed perception missing small attributes (e.g. object (Zhang et al., 2024a)), misclassifying sub-categories (Geigle al., 2024; Kim & Ji, 2024; Yu et al., 2025), and confusing geometric primitives (Zhang et al., 2025b). These limitations motivate our work, which aims enhance the fine-grained perception of the MILMS by improving query-specific spatial resolution prior to feature extraction.
Different Approaches for Fine-Grained Visual Understanding. We group prior efforts into the following categories: (i) Bounding-box methods (Peng et al., 2023; Chen et al., 2023a; Zhang et al.; Lu et al., 2024) steer attention by feeding cropped regions obtained from bounding boxes. (ii) Mask- based methods (Chen et al., 2024b; Yuan et al., 2024; You et al., 2023; Zhang et al., 2024b) supply (Kirillov et al., 2023). (iii) Cascade methods
pixel-accurate masks- often from Aegnentt-Anyhingg (He et al., 2025; Yang et al., 2023a; Yu et al., 2024) overlay detector cues or saliency heatmaps to bias the input. (iv) Reasoning methods (Surís et al., 2023; Wei et al., 2022) decompose queries into low-level visual steps. Our approach achieves stronger grounding while avoiding extra detectors, masks, or multi-step reasoning chains.
Pixel-Level Warping Techniques for Saliency Emphasis. Classical work includes seam carving (Rubinstein et al., 2010), saliency-aware warps (Wolf et al., 2007: Recasens et al., 2018), energy minimisation (Karni et al., 2009), finite-element grids (Kaufmann al., 2013), mesh parametrisation (Guo et al., 2009), and seam & scale methods (Zhang et al., 2009). Learning- based variants explore adaptive resizing (Talebi & Milanfar, 2021), saliency enhancement (Ghosh al., 2019; Miangoleh et al., 2023), and domain adaptation (Zheng et al., 2025), with contemporaneous magnification work (Mao et al., 2025). Many existing approaches, such as those employing energy minimization or seam carving, are optimization- based. Consequently, processing each input sample can take several minutes. In contrast, our proposed method leverages a single forward pass of a Cumulative Distribution Function (CDF), enabling near-instantaneour processing. Here, we build on saliency- aware sampling by introducing cuerr-condiiione,, rectilinear warp that preserves the image's regular grid structure, ensuring compatibility with the MLLM's vision encoder.
ATTENTION-GUID
IMAGE WARPING
We propose a simple and effective test-time technique to improve visual grounding of MLLMs. Instead of feeding the original image directly, we apply a spatial transformation guided by the model's internal attention, reshaping image regions based on their relevance to the query. Below, we provide high-level overview of AttWarp, followed by detailed description of its components.
Overview, As illustrated in Fig. 1, AttWarp uses cross- cross-moda modal attention maps from deeper layers of the MLLM (see Sec. 3.4) to guide
cwwpributioonpreeeeviie non-uniform resampling of the original image. This resampling operation, termed rectilinear warping (Sec. 3.1), redistributes pixel density across the image: regions with high attention are spatially expanded, while less relevant regions are compressed. Relevance is always defined with respect to the specific query, making the warping adaptive to task semantics. Crucially, the warped image retains a regular grid structure, ensuring compatibility with standard vision encoders. Next. in Sec. 3.2, we introduce extension of AttWarp that iteratively applies multiple warps, grounding each step in the model's evolving attention and improving performance on complex queries. Finally, in Sec. 3.3 we introduce EttWarp-Distill a computationally efficient version optimized for inference speed, which runs 3x faster than prior methods by shifting additional computation to training time through learned warping functions.
SetWarp-Chaile an
3.1 RECTILINEAR IMAGE WARPING
Given an input image I E H WX3 x3 and an attention score matrix A E H (from See. 3.4), our goal is obtain a function F that transforms I a warped image W- F A). The warping function F is designed to magnify important regions (high attention) and compress less relevant ones.
First, we compute marginal attention profiles (PDFs) along rows and columns decompose the 2D attention matrix into ID score vectors:
Horizontal Attention Profile: m.(
Σ
Aij Vertical Attention Profile: my(1)
Σ j=1
Agg. (1)
3

---

Published as a conference paper at ICLR 2026
Here, E (1,2... H) and (1, .2..... W). This decomposition facilitates rectilinear warping, enabling independent transformations along the horizontal and vertical axes while preserving the grid structure. Subsequently, we convert these marginals into cumulative distribution functions (CDFs):
Ek=1 mz(k) (b) M,()) Σk=1 mz(()) (k)
Ei= mk(k)
M₂ Σk= my(k)
These resulting cumulative functions (Ma, My) are monotonically increasing and therefore invertible. We define the warping functions using their inverses, known as the Inverse Distribution Functions:
Warp (i) W MF'(//W) [x
(P" H (i/H)
where i E (1,2.. (1.
H) and j E (1, W). Together, these inverse mappings Waap and Waap constitute the overall warping transformation F. yielding the warped image. The final warped image is computed through bilinear sampling, applied along all three channels:
W[i,j] = Interpolate(I, Bilinear) Warp (x),xx ()). ()))
3.2 ITERATIVE IMAGE WARPING (ATTWARP-CHATN)
The optimal degree of warping depends on the query and the image. Queries focusing small details benefits from strong warping, while broader scene may require minimal warping. A naive way is to use superlinear (or sublinear) transformation over the attention score matrix to upweigh (or downweigh) the attention-guided warp. In this section, we introduce a more intuitive and nuanced scheme that performs better called SttWarp-Chain.
We build an iterative warping method based on an empirical observation i.e., warping improves MLLMs attention (See 4.3) and enhanced attention maps subsequently yield better warping. Leverag- ing this insight, we develop AttWarp- Chrann which after each iteration, extracts an updated attention map, and progressively refines the warp applied in the previous iteration. Formally,
Initialization: (()) =I I, Chain step: W(d) F(W(d-1); A(d-)))
(5)
Here, 1d denotes the attention map computed from the warped visual input 1 practical question left answer is when terminate the chain of iterative warping steps? Instead of encoding this as hyperparameter, we propose a more adaptive route. As the relevant query-specific region expands, the attention map spreads more uniformly over the image. Eventually, the attention distribution stabilizes, indicated by minimal changes between successive attention maps. We quantify this stability through the following stopping criterion:
DKL. (p(d) (ptd)ppl-)) < <EK..
(6)
where p(d) and p(d-1) are normalized attention probability distributions from iterations d and d respectively. This termination ensures AttWarp- Charnn--hainnn achieves optimal spatial emphasis while mitigating the risks associated with noisy or overly aggressive warping. We quantitatively show effectiveness of AttWarp- Chain and termination criteria in App. D.3.
3.3 LEARNING TO PREDICT MARGINAL ATTENTION PROFILES
Many applications (e.g., edge AR and embodied agents) need fast, precise grounding. Masking, cropping. or re-rumning attention adds latency; even AttWarp-Chaal requires multiple passes. We therefore learn a single-pass predictor that outputs the horizontal and vertical marginals, mz(() and (i),), directly from an image-text pair. This neural functional approximation removes attention retrieval at inference and keeps the warping semantics of Sec. 3.1.
Teacher: generating marginal targets. We use the base MILM and the attention-extraction pipeline from Sec. 3.4 to produce training targets. For each image-text pair (I, text) we compute the attention score matrix A, derive axis-wise marginals via equation 1, and normalize them to unit mass, yielding (mg, m.y). This defines the dataset D {[I,, text,, My. My.n) J. Constructing these targets is intentionally done once offline to train the student; at inference the student replaces this pipeline, amortizing cost and enabling single-pass usage.
4

---

Published as a conference paper at ICLR 2026
Student: -ittilp Using these offline targets, train compact network to predict (mg, my) from an image-text text pair (architecture in Fig. 2). We encode the image with CLIP ViT-L/14 to obtain vision tokens Z and obtain text tokens from tokenizer E, applied to the query. Text conditions the vision tokens through Feature-wise Linear Modulation (FiLM) (Perez et al., 2018). 8). A small MLP maps the text tokens q to per-channel scale and shift parameters (a, b), and applies them channel-wise to obtain the modulated tokens Z a O Z + b. We then upsample Z to (H, W) and average along one axis at a time to obtain two ID summaries (horizontal and vertical). Two light ConvlD heads turn these summaries into logits, which SoftMax converts valid marginals (vi my Training minimizes the expected L, discrepancy over DD i.e., the average of [|m, mz + I, - mylli across samples.
Single-pass, fast inference. At inference, given (I, text), AttWarp-Distill outputs (Mi my in one forward pass. We convert them to CDFs (M2, My) via Eq. equa- tion 2, invert coordinates using Eq. equation 3, and bilinearly sample the image as in Eq. equation 4 to obtain W. This retains the semantics of Sec. 3.1 while reducing cost. Training details appear in App. G.3.
SoftMax
SoftMax
ConvID(5,5.1)
Conv1D(5,5,1)
(onv1D(5,5,1)
(555,,))
ax b
a
3.4 ATTENTION SCORE MATRIX IMPLEMENTATION
FLLM Condition
x
Linear
Here. we describe general recipe for constructing the attention score matrix A for any base MLLM. The proce- dure has two steps: (1) Attention retrieval, which reads raw cross-attention weights, and (2) Attention aggregation, which collapses those weights into single spatial map for the given image-text pair.
Couv 2D (3x3,
Linear
[1x1, 1)
Vivie
Thd Tokm
Attention map retrieval. The image I is tokenized into ammg = brea Wleat vision tokens on Meat x Wheat grid, After processing and the text, the MILM produces Hont output tokens. From selected decoder layers L and all attention heads (Pheads), we obtain cross-attention matrices (P,h) E To XTmme; entry Gyn,t token m to image token t.
CLIP VIT//14
Tokeniser
On the right desks what is to the left the laptop?
(f.h) is the weight from output
Figure 2: AttWarp-Distill architecture: CLIP vision tokens are FiL.-modulate by text and projected ID marginal predictors.
Attention map aggregation. We now average over out- put tokens, heads, and the chosen layers form single spatial map. Each image token index f. E {1 .1..... Ring)
corresponds to grid location (i,j) via
t 1) Wiea j. The aggregated score at (i.j)
Flat Thgadds
A
(E,h) dm
Thput
Theads
C
LEC
h=1
Here, amntt denotes the weight from output token to image token t. We upsample Å from hiaa X Wiett to the image resolution H W using Lanezos, smooth with le x k AvgPool, and optionally
(F.h)
apply scalar transform to control sharpness. The final attention score matrix is T (AAg). T {z, x², V }.. Here, E {1, 2...... H}. j E 21, W}. Sharper transforms like x² emphasize high-attention regions more than linear ones, which is useful for fine-grained queries. For LLaVA, we choose the 20th layer i.e. L = {20}. For Qwen, we use the 16th layer i.e. L = (16). The strategy for choosing layer(s) L and attention aggregation method explained quantitatively in App. B. For more details on implementation and design choices, refer App. G.I.
4 EXPERIMENTS
In this section, we detail our experimental framework and findings. We begin by describing the evaluation benchmarks and the baseline models used for comparison (Sec. 4.1). Next, we present our key quantitative results across multiple multimodal benchmarks (Sec. 4.2). Finally, we include ablations and analyses to share insights into AttWarp (Sec. 4.3)
5

---

Published as a conference paper at ICLR 2026
Original Imagee
A Atammtion Map
Parped Image
Original Imagee
Atteation Map
Warped Imaage
Query: What aaimal stauding betlnd the acimala that are staoding next the patay
Quey:: What type of suwaarch ás ousducted!
LLeVA: Does
X
AntWorg: Zebm
5) Query: is the aetwoek?
Quen: Isterual
X
ArtWorp: Unnsuooe
Pocus Groups
Pocia Geoups
Who the
Speaker X
AssWarpe Telectsion
Query: Belind what velieds is the buiding??
LLVAA: Bock
Autarp:: R.Capp
Query: there bobble in the image??
MaVA:
X
Van
X
AstWarpi
Figure AuWarp improves compositional spatial reasoning e.g. from GQA dataset (a) correctly identifying zebra behind the path, (b) television below the artwork; text understanding in documents e.g. from Doc (d) consumer focus groups, fine- fine-grained recognition of small/occluded objects e.g. from POPE (f) detecting bottle
4.1 BENCHMARKS AND BASELINES
Benchmarks. We evaluate Attwarp on nine diverse benchmarks designed to assess key multimodal capabilities, including general multimodal understanding, compositional reasoning, spatial relation- ships, visual hallucination, and fine-grained visual understanding (Fig. 3 shows qualitative results). GQA (visual reasoning): Reasoning about objects, attributes, and relations in real-world im- ages (Hudson & Manning, 2019).
TexvVQA (scene text understanding): Answering questions that require reading and grounding text
in natural images (Singh et al., 2019).
DocVQA (document image understanding): Extracting and reasoning over textual and structural information in scanned documents (Mathew et al., 2021).
POPE (robustness evaluation): Probing fine-grained hallucination and reliability in vision-language models (Li et al., 2023).
MMMU (general multimodal understanding): Broad multi multi-disciplinary evaluation across STEM, humanities, and social sciences (Yue al., 2024).
RealWorldQA (spatial reasoning in the wild): Spatial reasoning and relative localization in complex
real-world scenes captured from vehicles and other environments (xAI, 2024).
BLINK ffine-grained perception): Core visual perception skills via fine-grainee tasks (c.g., relative
depth estimation, object localization, counting) cast into a VQA-style interface (Fu et al., 2024).
MMVP fine-grainee perception): Evaluation on CLIP-blind image pairs that stress robustness to
systematic changes in visual patterns such as orientation and viewpoint (Tong al., 2024).
MIA-Bench (instruction following): Multimodal instruction following under layered constraints on
style, length, and content while remaining visually grounded (Qian al., 2024).
We report the result of AttWarp GQA, Tex(VQA, DocVQQ. POPE, and MMMU in Tab. 1 and results on MMVP, BLINK, RealWorldQA, and MIA in Tab. 2.
Baselines. For rigorous evaluation, beyond the base MLLMs, we also compare AttWarp to four representative baselines for test-time visual intervention. Baselines span strategics for editing input image guide MILM attention directly inference (see Fig. 4 for visual examples).
FGVP (Yang et al., 2023b) (Region Isolation via Masking): applies semantic masks on the target region and reversely blurs the background (or applies green mask) outside the target region. SoM (Yang et al., 2023a) (Visual Grounding with Explicit Markers): segments the input image semantically using off-the-sher model, and labels each segment with unique visual marker.

---

Published as a conference paper at ICLR 2026
Table 1: Main results Tex GQA, MMMU, POPE, and Voc datasets in accuracy (%). The A Accuracy row reports the absolute improvement of AttWarp- over the base MLLM.
# Methods
Key Technique
TextVQa
GQA
MMMU
POPE
DocVQA
LLaVA (Liu et al., 2024a)
(MLP vision-language connector open data)
1
Base MLLM
49.3
60.5
36.9
85.3
18.1
2
PGVP-mask (Yang et al., 2023b)
Green mask overlay
39.4
59.2
36.1
85.3
19.0
3 FGVP-blur (Yang et al., 2023b)
Blur background
33.9
59.5
35.0
83.1
18.6
4 SoM (Yang et al., 2023a)
Grounded segments
54.5
35.6
78.5
15.8
5 API (Yu et al., 2024)
Alpha channel fade
49.9
60.6
36.9
85.9
17.4
ViCrop (Zhang ct al., 2025a)
object crop
60.9
37.2
87.0
22.5
7
Attwarp (ours)
Rectilinear warping
58.1
63.7
40.4
87.5
25.5
8
AttWarp-Disttll (ours)
Hfficientt inference
57.2
62.7
38.8
87.4
22.4
AttWarp-Chaad Chhnrn (ours)
Adaptive Chains
88.2
27.6
10
Accuracy
+11.0
+3399
+4.7
+2.9
+9.5
Qwen (Yang et al., 2024a)
(Cross-attention VI. adapter & partially closed data)
11
Base MILM
81.0
62.4
47.3
86.1
77.3
12
FGVP-mask (Yang et al., 2023b)
Green mask overlay
77.3
55.8
46.0
84,4
56.6
13
FGVP-blur (Yang et al., 2023b)
Blur background
72.3
55.8
46.5
81.3
38.6
14
SoM (Yang al., 2023a)
Grounded segments
61.5
47.8
45.1
75.8
57.4
15
API (Yu et al., 2024)
Alpha channel fade
81.6
61.1
47.4
85.8
68.4
16
ViCrop (Zhang et al., 2025a)
Add object crop
83.8
60.6
47.1
86.7
82.5
17
AttWarp (ours)
Rectilinear warping
84.7
64.0
50.4
87.4
84.1
18
SttWarp-Disttll (ours)
Efficient inference
84.1
63.1
48.9
87.2
81.8
19
AttWarp-CCe in (owrs)
Adaptive Chains
85.9
64.8
51.0
88.0
85.3
20
A Accuracy
+44.9
+2244
+33.7
+1.9
+-8.0
APIPrompting (Yu et al., 2024) AAtention-Modulalie Image Representation): computes an attention heatmap using auxiliary VLM (LLaVA or CLIP), and overlays it onto the input image. ViCrop (Zhang al., 2025a) (Context Reduction through Attention- Guidedd Cropping): crops images around regions of high saliency, based on the model's attention map. Notably, ViCrop takes two image input iee. both the original and the copped image.
4.2 QUANTITATIVE RESULTS
In Tab. 1, we present results for three methods introduced this work, iee. AttWarp, AttWarp-Chail and AttWarp-Distill, evaluated on two MLLMs and five diverse benchmarks.
AttWarp outperforms all baselines on five benchmarks (Tab. 1). AttWarp achieve state-of-the-nn results on tasks including text recognition and understanding (TextVQA: LLaVA +8. 8%, Qwen +3,7%;
VoA:QA: LLaVA +7.4%, Qwen +6.8%), compositional and spatial reasoning (GQA: LLaVA +3.2%, Qwen +1.6%), general multimodal question answering (MMMU: LLaVA +3.5%, Qwen +3.1%), and fine-grained understanding and hallucination reduction (POPE: LLaVA +2.2%, Qwen +1.3%).
These consistent improvement across diverse tasks arises from AttWarp's 'ss capability highlight lask-relevant objects while preserving global context and spatial relationships, thus delivering strong
Question: On the right desk, what is to the left of the laptop?
Ours (Warping)
FGVP
SoM
API (Alpha Blending)
ViCrop (Cropping)
(Green Masking)
(Visual Grounding)
A: "Lamp"
A: "Plant"
A: "Chair"
A: "Books"
A: "Printer"
Figure 4: AttWarp and prior works of image manipulation the running example. While plausible, prior works are unable to answer the question correctly.
7

---

Published as a conference paper at ICLR 2026
performance global queries in GQA and fine-grained queries in POPE. Moreover, these results illustrate that AttWarp's 's is agnostic to image type effective across natural scenes (GQA, TextVQA, POPE), documents (Qoc and dense diagrams (MMMU). We provide per-category performance in App. C.I and study on the extent of warping in App. C.3. Overall, the superior performance of AttWarp underscores the significance of enhancing perception for question answering.
AttWarp is plug-and-play with standard MLLMs (Tab. 1). By default, we evaluate AttWarp using LLa consistent with prior works (Zhang et al., 2025a; Yu et al., 2024; Yang al., 2023a). We further demonstrate AttWarp versatility on a stronger MLLM, Qwen2.5-VL. Qwen2.5-VL. uses a distinct vision-language fusion strategy with a dedicated cross-modal attention module to integrate visual and textual features, contrasting VL'' direct projection of visual features into the language model's input space. The consistent performance gains of AttWarp across Qwen2.5-VI. and VLa v1.5-7b highlight AttWarp's 's robust, plug-and-play compatibility with diverse MILM architectures. Additionally, AttWarp achieves similar gains with two other architectures: a dynamic-resolltion and pixel-unshufling approach (Intern and an instruction-aware Q-Former for -ision-language alignment (InstructBLIP). See App. D.I for results.
Results visual-centric benchmarks: We fur- ther evaluate AttWarp on four visual-centric datasets (RealWorldQA, BLINK, MMVP, MIA-Bench). Visual-centric benchmarks are typically harder than text-centric benchmarks because the model must handle challenging pixel inputs (occlusion, small details, count- ing/localization) and align language with visi- ble evidence, rather than relying mainly on linguistic patterns or memorized text cues. Across all four, AttWarp consistently out- performs the base MILMM (Tab. 2). indicat- ing stronger fine-grained visual understand- ing, more accurate spatial reasoning, and improved visually grounded instruction follow- ing. AttWarp-Distill, retains most of these gains with lower computational cost, while Sitwarp-Chaine leverages warping at multiple layers for further improvements. As summarized in Tab. 2, all three variants, AttWarp. AttWarp-Distill across these four benchmarks, supporting rectilinear attention-guided warping as an effective and general mechanism for enhancing fine-grained visual grounding in MILMs.
Table 2: AttWarp performance (in %) on visual-centric benchmarks. We report individual accuracy for MMVP, as it aligns with the format of other evaluations. We use LLaVA as the base MLLM. For BLINK, we took the single- image subset.
LLavA
MMVP
BLINK
RealWoridQA MIA
Base MLLM
48.3
38.3
49.3
65.9
AnWarp-Distilled (ours)
49.3
39.7
51.1
66.4
AttWarp (ours) Attaarp-Chaiss (ours)
50.7
40.4
52.1
67.8
51.0
41.2
53.1
68.8
A Accuracy
+2.7
+2.9
+3.8
+2.9
and StWaa--Chhin improve performance
Chaining provides consistent performance gains over standard AttWarp (Tab. 1, rows 7 & 9, rows 17 & 19). The multi-step AttWarp-Chain further boosts per- formance by iteratively refining attention maps. Examin- ing LLaVA results (rows 7 & 9), we observe consistent improvements over AttWarp across all five benchmarks (TextVQA +2.2%. GQA +0.7%, MMMU + 1.2%% POPE +0.7 %, and VQA +2.1%) A similar trend is evident when evaluated on the stronger base model Qwen (rows 17 & 19). Qualitatively, Fig. demonstrates how AttWarp- Chairn--hhinnn adaptively adjusts the warping extent, resulting in enhanced visual grounding and improved task performance.
Origiaal
Quarry What is the frame roade up
LLaVa: Wood
X
AnsWarg Wood
X
AntWarp Chaba: Mesu
Figure 5: AttWarp-Chail improves on AttWarp
AttWarp-Distill balances accuracy and speed (Tab. 1, Tab. 3). AttWarp-Distin
Table 3: Comparison of computational over- head. Base MLLM used is L.L.VA. Metries are TFLOPs, peak VRAM (in GB), and number of MLLM passes. Values in brackets show rela- tive cost compared to ViCrop.
consistently out- performs the base MILM and matches or exceeds the performance of ViCrop (Tab. 3). Optimized specif- ically for inference efficiency, AttWarp-Diss -Dstill re- quires only single MLLM forward pass, making it ap- proximately 3x faster and 2.8 x more computationally efficient than ViCrop; aligning closely with the compu- tational cost of the base MILM (8.7 vs. 8.5 TFLOPs). In Tab. 3, we compare our method against ViCrop as it achieves comparable performance to AttWarp-Diste -isttil...
TFLOPs L
Pesk VRAM
MILM passes
ViCrop AttVarp-
24.2 8.7 (0.4k)
22
3
Base MLLM
8.5
Other baselines, such as FGVP, SoM, and
8

---

Published as a conference paper at ICLR 2026
0.16
Query: What title of the orange book?
Non Warp
Anraap
Probubility density
0.12
Toll
0.08
0.00
10
20
30
40
30
60
Mahalanobis stannce Gaussian fit training set
LLa Witches of cast end x AttMarp: memory death
Analyzing data shift.
(c) Attention alignment with Ground truth boxes.
(b) Distribution metrics (1 = lower is hetter).
(d) Attention GT alignment metries TextVQA.
Distribation ir under:
KID (Jrovis ())
FID (Junsie v)
Metric
No warp
With AttMarp
AttMarp
49.8
Non Rectilinear Warp
174.9
73.9
Pointing Game Accuracy
37.4%
42.4%
Proportion
0.117
0,155
Test Set
19.3
56.6
Figure 6: (a) Mahalanobis distance the train Gaussian. (b) Train AttWarp, Non-Rect. FID/KID summary. (c-d) Attention-reddistriutiio GT alignment. See Appx. for setup and additional plots.
APIPrompting, not only perform substantially worse (see Tab. 1), but also incur additional overhead due to multiple inference steps. For more details and cost analysis of AttWarp refer App. G.4.
generalizable (Tab. 1). We train We train AttWarp-Distill on the standard training splits of widely used open-source datasets (TextVQA, GQA, and Doc)QA), which form part of the training corpora of most base MLLMs. The substantial improvements on TextVQA (LLaVA +7.9 7.,%, Qwen +3.1 %), GQA (LLaVA +2.2%, Qwen +0. 7%), and Doc (LLaVA +4.3%, Qwen +4.5%) demonstrate its strong in-domain generalization capability, whereas its robust performance on POPE (LLaVV +2.1% 1, Qwen +1.1%), and MMMU (LLaVA VA +1.9%, Qwen +1.6%), highlights its effective out-of-domain generalization. Further details on the training procedure and marginal prediction in App. G.3.
AttWarp-Dist:ln
4.3 ABLATIONS AND ANALYSIS
Warping Improves MILMss Attention. To gauge how AttWarp reshapes the spatial faithfulness of the internal attention of the model, we adopt two iidely-adopted localization metrics that rely only on ground-truth boundary boxes (bbox) and the heat map itself: 1) Pointing Game Accuracy (top-1 attention in bbox): checks whether the single most salient pixel of the attention map falls within the GT bbox (Zhang al., 2016). 2) Proportion (fraction attention within box): the fraction of total attention mass that lands inside the bbox (Wang et al., 2020). As observed in Tab, 6d, the +5% relative jump in Pointing Game Accuracy and the +3.8% boost in Proportion TextVQA confirm that our rectilinear warp tightens the focus of the model: the attention mass and dominant peak shifts the task relevant region. These findings indicate that AttWarp improves the MLLM performance because of better attention distribution (see Fig. F and App.F). We further extend this study and empirically verify that AttWarp expands the correct image regions. We quantify this by checking how often AttWarp expands the bboxes of the relevant regions (refer App. E for more details).
AttWarp's rectilinear design preserves the image distribution. Pixel-level transforms risk shifting test images away from the manifold on which the model was trained. We probe this risk for LLaVA by first extracting ViT-L/14 CLIP features (Radford et al., 2021) for 12h randomly sampled GQA-train images and fitting full-covariance Gaussian to those embeddings. Figure 6a compares the resulting Mahalanobis distance histogram for three image sets: (i) Test images (blue); (ii) images warped by AttWarp (orange); and (iii) non-rectilinear warping baseline, inspired by (Recasens et al., 2018) (green). AttWarp almost exactly overlaps the unmodified test distribution (both peak at z5 29a). whereas distribution based Non- Rectilinear Warp shifts rightward (peak 37Ø) and exhibits a heavier tail, indicating a measurable distribution drift. We also test this aspect using standard two-sample metrics, FID (Heusel et al., 2017) and KID (Binkoowkki et al., 2018) between the training

---

Published as a conference paper at ICLR 2026
Bux MELM
bis uninal imaug?? Gourd Trath: Small
What clothing black?
Arrlay wollaa
Grume Trith: Blorse
Scewerith Loads
Ser
MLLM; Small
Aut Warps Large
MLLMM: Shirt
Av:Warge Parts
(a) Quantitative error plot.
(b) Examples of failure cases of AttWarp.
Figure 7: Error Characterization and Failure Cases.
distribution and each test variant. As evident from Table 6b, AttWarp achieves significantly better
alignment with the training distribution (31.5 vs. 174.9 KID and 49.8 vs. 73.9 FID), effectively reproducing the train-test baseline gap. Alongside this, can be seen in Figure 6a, approach closely matches the baseline metrics indicating that AttWarp preserves the underlying image manifold and introduces negligible distribution shift (see App. for graphs and details).
Error Analysis. We randomly sampled 150 VQA tasks from the GQA and TextVQA evaluation. A total of 61 were incorrect for the base LLaVA model, and 42 were incorrect for AttWarp. We bin these into the closest failure modes: GraiGGr Details, where the answer is very small in size; Hallucination when the answer includes details not present in the image; Misaligned Attention, when focus shifts to the wrong object; Size, for questions involving object scale; Semantically Correct, when the answer is correct but phrased differently; and Compositional Reasoning, involving multiple objects and relationships. Fig. 7a shows fewer errors in fine- aiee-graine and compositional cases. However, we note that warping can sometimes suppress peripheral context needed for global reasoning, and performance may degrade if the underlying attention is noisy. While absolute object sizes are changed, relative proportions are preserved, limiting errors in size-related tasks.
Reducing the Error Modes. The error analysis shows that AttWarp reduces errors across all categories compared to the base MLLM (e.g., hallucination: 12 - 9, fine-grained details: 12 - 3). However, AttWarp remains susceptible to errors related to object size and misaligned attention, motivating the development of methods that further enhance robustness. To explicitly target these failure cases, design a simple classifier that decides whether apply AttWarp. Concretely, reuse AttWarp-Distinn 's weights and network, and replace its last two layers with a binary classifier head. We create a training set by evaluating AttWarp on the training split of LtWarrp-Dsstll (App. G), and use these outcomes to train the classifier. We denote the resulting classifier-gater variant by AttWarp', i.c.. AttWarp' AutWarp' shown in Fig. 7a (green bars). AutWarp' further reduces errors in size and hallucination while keeping other categories essentially unchanged. Misaligned-attentin as both the base MILM and AttWarp fail on the same underlying attention misalignment. Designing mechanisms correct attention misalignment is, therefore, productive direction for future work.
Classifier + AttWarp. We then perform the same error analysis for
errors remain unchanged,
5 CONCLUSION
We introduced AttWarp, a plug-and-pll and-play, test-time self-correction mechanism that uses an MLLM's cross-modal attention to rectilinearly resample the input, expanding query-relevant regions while preserving global context. Without changing model weights or architecture, it consistently improves accuracy, spatial grounding, and hallucination rates across nine benchmarks and four MLLMs. By intervening before feature extraction, AttWarp complements internal attention refinements and shows that input-level, information eeformation-preserving transformations can help the same models see better.
6 ACKNOWLEDGEMEE
This research is based upon work supported by U.S. DARPA ECOLE Program No. HR001 12390060. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes, notwithstanding any copyright annotation therein. This work has also received support from the School of ICS UC Irvine and from an NSF grant CCF 2348624.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-23777 2022.
Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, and Chenliang Xu. Unveiling visual perception in language models: An attention head analysis approach. arXiv preprint arXiv:2412.18100. 2024.
Mikotaj Binkooski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR, 2018.
Marisa Carrasco. Visual attention: The past 25 years. Vision research, 51(13):1484-1555 2011.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer. 2024a.
Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-to-fine method for vision transformer. Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 7042-7052, 2023b.
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 17164-17174, 2023c.
Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu- Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi- modtl large language model for referring expression segmentation. In European Conference on Computer Vision, pp. 323-340. Springer, 2024b.
Wenliang Dai, Jumnan Li, D Li, AMH Tiong. J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general-purpose vision- languaee models with instruction tuning. arxiv 2023. arXiv preprint 2xXXx:2305506000, 2, 2023.
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.1445. 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision (ECCV), Cham, 2024. Springer Nature Switzerland.
Gregor Geigle, Radu Timofte, and Goran Glavaš. African or european swallow? benchmarking large vision-language models for fine-grained object classification. arXiv preprint arXiv:2406. 14496, 2024.
Sanjay Ghosh, Ruturaj G. Gavaskar, and Kunal N. Chaudhury. Saliency guided image detail enhancement. Proceedings of the National Conference on Communications (NCC), pp. 1-5. IEEE, 2019.
James J. Gibson. The Senses Considered as Perceptual Systems. Houghton Mifflin, Boston, 1966. ISBN 9780395044940.
Yanwen Guo, Feng Liu, Jian Shi, Kun Zhou, and Baining Guo. Image retargeting using mesh parametrization. IEEE Transactions Multimedia, 11(5):856-867, 2009.

---

Published as a conference paper at ICLR 2026
Hulingxiao He, Geng Li, Zijun Geng, Jinglin Xu, and Yuxin Peng. Analyzing and boosting the power of fine- grained visual recognition for multi-moda large language models. arXiv preprint arXiv:2501.15140, 2025.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1rXiv:1903.12220 122611 2019.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- -O. Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01662, 2022. URL https: //tp///aavvv.v//a///aa/eeaeeaaaa wrw. wwgggabs/2208... 01626.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, pp.
2017.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6700-6709, 2019. doi: 10.1109/CVR..2019.000
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. Your large vision-language model only needs a few attention heads for visual grounding. arXiv preprint arXiv:2503.06286 2025.
Zachi Karni, Daniel Freedman, and Craig Gotsman. Energy-based image deformation. Computer Graphics Forum, 28(5): 2009.
Peter Kaufmann, Olga Wang, Alexander Borkine-Hornune Aljoscha Smolic, and Markus Gross. Finite element image warping. Computer Graphics Forum, 32(2pt1):31-39, 2013.
Jeonghwan Kim and Heng Ji. Finer: Investigating and enhancing fine- fine-grained visual concept recognition in large vision language models. arXiv preprint arXiv:2402.16331 2024.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4026,4026, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9588 9779,9989,001 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluat- ing object hallucination in large vision-language models. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pp. 292-305, Singapore, 2023. Association for Computational Linguistics. doi: 10. 10.:8655/1//0223emnppp https://accanthhlloee org/2023 emmlp-main. 20/-
URL
Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 5334-5342, 2025.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13592-23601. 2023a.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892-3499 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
12

---

Published as a conference paper at ICLR 2026
Jinghui Lu,, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, et al. bounding box is worth one token: Interleaving layout and text in a large language model for document understanding. arXiv preprint arXiv:2407.01976, 2024.
Shunqi Mao, Chaoyi Zhang, and Weidong Cai. Through the magnifying glass: Adaptive perception magnification for hallucination-Fii free vlm decoding. arXiv preprint arXiv:2503.10183 2025.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2199 2199-2208, 2021. doi: 10. 109//ACC446300020.0000000
Sadegh Aliakbarian Miangoleh, Emre Yaksoy, Miika Aittala, Sylvain Paris, and Frédo Durand. Realistic saliency guided image enhancement. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 186-185, 2023.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thoughe prompting for large multimodal models. In Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14420-14431, 2024.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:230330007774 2023, doi: 10.48550//xXi.2300.. 08774. URL http://aaaxxxvvxaaaaeeeee
08774.
Georgios Pantazopooloss Alessandro Suglia, Oliver Lemon, and Arash Eshghi. Lost in space: Probing fine-grained spatial understanding in vision and language resamplers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2, Short Papers), 2024. The authors show that the compression performed by popular resampler modules leaves most spatial information "largely absent," evidencing lossy visual encoding stage.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11410-1442, 2022.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 14824, 2023.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. AAAI Press, 2018
Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfci Yang, and Zhe Gan. MIA- Bench: Towards better instruction following evaluation of multimodal LLMs. arXiv preprint arXiv:2407.01509, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Adria Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba. Learning to zoom: saliency-based sampling layer for neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 51-66, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-1665,, 2022.
Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir. comparative study of image retargeting. ACM Transactions on Graphics (TOG), 1-0,, 2010.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8317-8326, 2019.
13

---

Published as a conference paper at ICLR 2026
Yizheng Sun, Hao Li, Chang Xu, Chenghua Lin, Riza Batista- -avista-Navarro, and Jingyuan Sun. Silent hazards of token reduction in vision- lasoonlanguuae models: The hidden impact on consistency. arXiv e-prints, pp. arXiv-2503, 2025.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11888-1 1898, 2023.
Hossein Talebi and Peyman Milanfar. Learning to resize images for computer vision tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 497-506, 2021.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9568 99789 June 2024.
Haofan Wang, Zifan Wang. Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops (CVPR 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems,
2022.
Lihi Wolf, Michael Guttmann, and Daniel Cohen-Or.
Non-homogeneous content-driven video- retargeting. In IEEE International Conference Computer Vision (ICCV), pp. 1-6. IEEE, 2007.
XAI. Realworldqa, 2024. URL httpg:///wnggggggooooooootttttts org org/RealworlddA......
Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Erran Li. Vigor: Improving visual grounding of large vision language models with fine-gra fineggrained reward modeling. In European Conference on Computer Vision, pp. 37-53. Springer, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.11441, 2023a.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14177 2024b.
Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36:24993-25000 2023b.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.0770. 2023.
Hong-Tao Yu, Xiu-Shen Wei, Yuxin Peng. and Serge Belongie. Benchmarking large lasion- models on fine- iine-grained image tasks: comprehensive evaluation. arXiv preprint XiXX:25544414988 2025.
Runpeng Yu. Weihao Yu, and Xinchao Wang. Attention prompting on image for large vision-language models. In European Conference on Computer Vision, pp. 251 251-268. Springer, 2024.
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22202-28211, 2024.
14

---

Published as a conference paper at ICLR 2026
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning henchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), 2024.
Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. In Proceedings of the 14th European Conference on Computer Vision (ECCV), pp. 543-559, 2016.
Jiarui Zhang, Jinyi Hu, Mahyar Khayatkhoei, Filip Ilievski, and Maosong Sun. Exploring perceptual limitation of multimodal large language models. arXiv preprint arXiv:2402.07333 2024a.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to
look: Training-free free perception of small visual details with multimodal Ilms. arXiv preprini 1rXiv:2502.17422, 2025a.
Liang Zhang, Meng Zhang, Bo Wang, and Ruigang Wang. Optimized image resizing using seam carving and scaling. Proceedings of the 17th ACM International Conference on Multimedia, pp. 147-156. ACM, 2009.
Shan Zhang, Aotian Chen, Yanpeng Sun, Jindong Gu, Yi-Yu Zheng, Piotr Koniusz, Kai Zou, Anton van den Hengel, and Yuan Xue. Open eyes, then reason: Fine-grained visual mathematical understanding in milms. arXiv preprint arXiv:2501.066 064430,0003 2025b.
Shilong Zhang, Peize Sun, Shoufa Chen, Minn Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on rggionof-interess 2024. In URL http:://ppeervview net/forum.
Zhehao Zhang, Ryan Rossi, Tong Yu, Franck Dernoncourt, Ruiyi Zhang, Jiuxiang Gu. Sungchul enhancement via
Kim, Xiang Chen, Zichao Wang, and Nedim Lipka. Vipact: specialized vlm agent collaboration and tool-use. arXiv preprint arXiv:2410. 16400, 2024b.
Shen Zheng, Anurag Ghosh, and Srinivasa G. Narasimhan. Instance-warp: Saliency guided image warping for unsupervised domain adaptation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 8186-8195, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:25044 10479, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
This appendix provides additional material to support and extend the findings in the main paper. Each entry is clickable.
FAQ Clarifications on distributional integrity, layer usage, failure modes with poor attention, and multi-object focus.
Ablation Study on Attention Score Matrix Empirical analysis of where the best attention comes from and how we it.
B.1 Layer Selection Across MLLM Layers & Head Aggregation Beest layers to use and why averaging heads is robust.
B.2 Pixel-Space vs. Feature-Space Warping Why rectilinear pixel warping is stable and model-agnostic.
B.3 Effect of the Transform Hyperparameter sgrt//dennity/square/cuhss
(T)
Sensitivity across
B.4 Impact of Attention Bias, Corruptions, and Adversarial Perturbations Robust- ness under ImageNet-C noise and adversarial misdirection.
Task-Specific Analysis of AttWarp Detailed breakdown of where AttWarp helps and the few places it does not.
C.1 Task Categories, Benefits, and Limitations -Fine-grained perception, spatial reason- ing, and hallucination mitigation.
C.2 Distortion and Upper-Bound Concerns Geometry preservation and semantic cate- gory results.
C.3 Global Context and Warping Intensity - Adaptive strength via Jacobian-basee intensity.
C.4 Task-wise Comparison with ViCrop- Category-level head-to- ttad on GQA.
D Additional Experiments Broader evaluations showing generality and stability.
D.l Extended Generalization Results on InstructBLIP and Intern VL-3.
D.2 Fine-Grained and Category-Wise DocVQA Breakdown across document struc- tures.
D.3 Stability and Termination in AttWarp-Chaing - Fixed vs. adaptive depth.
D.4 External Attention Maps - Ussng Stable Diffusion and Qwen-VL attention to help LLaVA.
E Beyond Standard VQA: Extended Experiments Forward-looking pilots demonstrating versatility.
E.1 AttWarp Expands All Query-Relevant Regions Singlc/multi-ojject bbox expansion on TextVQA and gRef.
E.2 opec-Vocabylary Object Detection Integration with LISA-LLaVA.
E.3 Leveraging Smaller Models Improve Larger Models 7B attention boosting 34B.
Attention Redistribution: Reduction in Model Uncertainty Visual and quantitative evidence of sharper, less diffused attention.
G Implementation Details Reproducibility and configuration.
G.1 Quantitative Evaluation Setup - Moodes, layers, resolutions, thresholds, and base- lines.
G.2 Stable Diffusion Experimental Details Attention extraction and schedules.
G.3 Training Setup for AttWarp-Distln Teacher marginals and optimization.
G.4 Computational Cost Calculation and Analysis FLOPP//oken//VRAAA vs. ViCrop.
H Qualitative Examples Case studies of AttWarp and AttWarp-Chain.
H.1 Examples of AttWarp Corrections over baseline.
H.2 Examples of AttWarp-Chain Iterative refinement and convergence.
I FID and KID Analysis Train-tess alignment in CLIP feature space (ViT-B/32 and ViT- L/14).
J Alternative Interpretation: Warping the Visual Tokenization Grid
16

---

Published as a conference paper at ICLR 2026
A
FAQ
1. How is distributional integrity maintained during warping?
Ans.
Two primary reasons underlic this preservation. First, the rectilinear warping preserves spatial attributes, positional relationships, and structural integrity. At the tokenization stage, objects maintain their semantic identities. Second, the rectilinear warping directly aligns with the data augmentation strategy used during CLIP pre- pre-training. Specifically, CLIP pre-training uses RandomResized- Crop-mandomly sampling 8-100% of the image with aspect ratios 3/4-4/3, then resizing to 224 x244 ox--scaling rows and columns independently yet axis-aligned, thus exposing the model to varied scales and positions for robust, scale-invarian representations Radford et al. (2021). As the warp is defined as
(z'.y)) FF)
with monotone 1-D CDFs, its Jacobian is strictly diagonal. Consequently, (1) each ViT patch subjected only to wiss scaling (F% (z) along z, (y) along y), and (2) borders remain orthogonal. Hence, what the encoder encounters at the token level in AttWarp is identical in form to the resize-then-crop augmentation used during CLIP training, albeit with locally different scale factors. Thus, rectilinear warping keeps token statistics on the same manifold learned during pre- pre-training.
2. Did you use a single layer or multiple layers? The formulation you presented appears to be for multi-layer approach.
Ans. While the formulation demonstrates a general multi-layer capability, our quantitative and qualitative analyses indicated that single layer (specifically the 16th layer for the Qwen and the 20th layer for the LLaVA model) performed better than average across all layers in this instance.
3. If the attention is highly inaccurate, then what?
Ans. If the attention map highly inaccurate, both the base MLLM and the MLLM with Warp tend to fail and produce incorrect answers. However, in most practical scenarios, the attention maps are only moderately suboptimal. In these cases, AttWarp is particularly effective its warping mechanism enhances the attention distribution, leading to improved performance. Detailed experimentation focused on robustness are presented in Appendix B.5.
4. What happens when there are multiple objects of focus? Does AttWarp work in that case?
Ans. As demonstrated in Appendix 1. we conducted a study on cases with multiple objects of focus ranging from two up five distinct regions. Our results show that AttWarp consistently expands the target regions, with 89% of the annotated bounding boxes exhibiting increased area after transformation. In many instances, all objects of interest (up to five per example) are effectively expanded, highlighting the method's robustness in handling complex, multi-object referential expressions.
5. What if image warping largely distorts the image? Ans.
The extent of distortion introduced by image warping is governed by the choice of transformation function T. For the transformations we utilize-T e {sqrt, identity, -quaare--sgnifffaann significant distortion is not observed in practice. Extreme warping only occurs if the attention map is highly concentrated on a single, small region, which is rare and typically corresponds to extremely fine-grained queries. In such cases, the resulting distortion is often advantageous, as it further magnifies the most relevant region, thereby improving the model's 's ability to answer the query.
17

---

Published as a conference paper at ICLR 2026
B ABLATION STUDY ON ATTENTION SCORE MATRIX
B.1 LAYER SELECTION ACROSS MILM LAYERS
To select the attention map that best captures the visual semantic signal of fine-grained image details with respect to the query. compared two normalization schemes-absolute absolute and relative thrention-through every cross attention layer of LLaVA-I.5-7B. The absolute map is the raw cross- attention weight assigned to each image token when the model answers the question, whereas the relative map divides this weight by a caption-only baseline obtained from an auxiliary forward pass that asks the model to "describe the image briefly." The intuition is that relative normalization suppresses static scene priors (e.g. sky or grass that invariably attract some attention) and instead highlights query-specific regions. We benchmark both variants 2000 TextVQA validation images using Pointing-Gand Game Top-1 Hit (Zhang et al., 2016) and AM @all (Wang et al., 2020). Fig. presents layer-wise localization analysis for LLaVa 7b: scores increase through the mid-network for both schemes, peak with the absolute map from layer 20 (ToP-1 = 0.36, AM @ ALL R 0.22), and then plateau or decline; relative attention helps in earlier layers but never exceeds the absolute variant beyond layer 18 and likewise weakens past layer 20. These findings indicate that the deeper blocks progressively concentrate the model's "attention budget" on the object interest, but the final layers begin to divert attention to token generation. Consequently, all LLaVA-based warps in our work At AttWarp employ the absolute attention layer 20. An analogous sweep on Qwen-VZ VL reveals its optimum at layer 16. The qualitative evidence in Fig. 9 mirrors the quantitative trend: attention tightens around the "yamaha" logo on the bass drum from layers 10 20, underscoring the utility of this layer- wise study previously unexplored in prior works and guiding choice of layer 20 for LLa V--based models and layer 16 for Qwen- VL as the most reliable attention sources.
Practical Considerations against Adaptive Layer Selection Dynamically evaluating multiple layers per query would add significant runtime overhead, undermining the low-latency goals of test-time adaptation. Our focus this study was to validate the core AttWarp mechanism under the best static choices.
Top-1 Hit Rate
AM @all
0.4
Attention-mass fraction
absolute
absolute
0.20
Hit fraction
0.3
relative
relative
0.15
0.2
0.10
0.0
0.05
4
10 12
14 16 LLaVA A layer
18
20
24
4
10 12
14
16
18
20
24
LLaVA A layer
Figure 8: Layer-wise localisation quality of LLaVA-1.5-7B cross-attention maps on TexiVQA Zhang et al. (2025a) images. Curves report Top-1 Hit Rate (Zhang et al., 2016) (left) and AM@all (Wang et al., 2020) (right) for absolute (blue) and relative (orange) attention.
Question:
What is the brand of the bass drum? Answer: Yamaha
YAMAHA
Original Image
Layer
Layer
Layer
Figure 9: Below (left to right): (a) Original image, zoomed in at yamaha (answer), (b) to (d) attention maps captured from layers 10, 16, 20 respectively. As can be the attention localization improves drastically from layer 20, indicating improved query-specine spatial understanding.
18

---

Published as a conference paper at ICLR 2026
Method
TextQQA
GQA
MMMU
POPE
DocVQA
Base MILMM
49.3
60.5
36,9
85.3
18.1
AltWarp (Layer 20)
58.1
63.7
40.4
87.5
25.5
AntWarp (1 (layer 22)
58.4
62.8
39.1
87.1
24.9
Owen
Method
TextVQA
GQA
MMMU
POPE
DoeVQA
Base MLLM
$1.0
62.4
47.3
86.1
77.3
AntWarp (Layer 16)
84.7
50.4
87.4
84.1
Antarp (Layer
Table 4: Performance of AttW across different attention layers on LLaVA and Qwen backbones.
Robustness to Attention Layer Choice (Table 4): These results reveal simple and robust rule for selecting attention maps in MLLMM. As previously observed in CNNs and ViTs, MILMs too exhibit similar trait: deeper layers produce attention maps that become increasingly centered objects of interest. Specifically, we find that attention maps from layers with depth 2 15 in MLLMs consistently yield aas--appo,,,att,
maps for AttWarp. To validate robustness the specific choice of layer, we re-run AttWarp using new, much deeper layer (24) for both LLaVA and Qwen and evaluate on five benchmarks. Across all datasets, the layer- 24 variants closely match or slightly exceed the gains obtained with layers 16/20, demonstrating that AtrWarp is largely insensitive to the precise layer index as long as it is sufficiently deep. practice, we therefore recommend using earlier deep layer (e.g.. 16-20) to reduce computational overhead while preserving the full performance benefits.
gegion-o-interrst-aliinne
B.2 ATTENTION AGGREGATION STUDY
Robustness of Attention Head Aggregation Each attention head captures different aspect of the multimodal interaction (e.g., color, shape, texture, positional cues). Beyond selecting the optimal layer, we analyzed the strategy for aggregating attention across heads. Our analysis confirms that uniformly averaging attention from all heads provides the most robust and informative signal for warping. shown in Table 5, this approach significantly outperforms alternatives like max max-poo ppoling or using random subsets of heads, validating our data-driven design choice.
Table 5: Effect of attention head aggregation strategy TexIVQA accuracy (%). Our method of averaging all heads is the most effective and robust.
Aggregation Methodology
TextVQA Accuracy
Mean over all 32 heads (Ours)
58.1
Max-pooling across 32 heads (token-wise)
55.3
Mean over 8 randomly selected heads
54.6
Random single head (re-sampled per run)
51.9
B.3 PIXEL-SPACE VS. FEATURE-SPACE WARPING
An alternative to our proposed image warping is to directly manipulate the internal feature space of the multimodal LLM. We explored this option by injecting a bias into the hidden states after normalization in the first cross-attention block:
h' LayerNorm(h)
A-
where b constructed from attention weights and À controls the bias strength. Although conceptually attractive, this approach proved unstable in practice: performance gains were marginal (only +0.6 % on Tex.VQA). Moreover, the results of the feature space warping were highly sensitive to the choice of X. As highlighted in previous research Sun al. (2025), direct manipulations at the internal feature level inherently risk causing significant distribution shifts that interact unpredictably with architectural components such ore-or post-norman nost-oommaizzaionn layers.
In contrast, our rectilinear warping pixel-space avoids these pitfalls and offers several concrete benefits.
19

---

Published as a conference paper at ICLR 2026
Stability and robustness: The internal computation of the model remains unchanged, ensuring consistent behavior across datasets and queries.
Interpretability: The warp visually transparent: the expanded regions correspond directly to areas of high attention, allowing intuitive inspection and debugging.
Architecture agnostic: No architectural modifications are needed, making the approach compatible across diverse MLLMs and even with external attention sources.
Geometry preservation: The axis-wise CDF warp expands relevant areas while maintaining global structure and relative spatial layout, safeguarding spatial reasoning ability.
B.4 EFFECT OF THE TRANSFORM HYPERPARAMETER (T) IN ATTWARP
To assess the role of the transform function T, performed ablation across several functional forms. Across both Vex and GQA (Tables and 7), results are stable for simple choices such as square root, identity, square, and cubic. The identity and square transforms consistently achieve the best accuracy (58.1-58.3 on TextVQA; 63.3-63.5 on GQA), while alternatives yield only slightly lower scores. These findings show that AttWarp is robust the choice of T and that the identity transform serves as a strong default-deliverrig state-of-the-art accuracy without dataset-specific tuning and outperforming competitive baselines such as ViCrop.
Table 6: Effect of the transform function T TextVQA accuracy (%).
Table 7: Effect of the transform function T on GQA accuracy (%).
Transform
Accuracy (%)
Method
LLaVA
Qwen- VL
LLaVA Baseline
49.3
Baseline Model
60.5
62.4
T = sqrt
56.8
ViCrop
60.9
60.6
T Identity
58.1
AttWarp (Identity)
63.3
63.5
T = square
58.3
AttWarp (sqrt)
63.7
64.0
T cube
57.8
B.5 IMPACT OF ATTENTION BIAS AND ROBUSTNESS TO CORRUPTIONS AND ADVERSARIAL. PERTURBATIONS
natural concern that the performance Attwarp depends on the underlying pretrained model. the attention is biased- for example, by over-focusing on high-frequency features- this may limit the generalization our method. Similar to other test-time adaptation (TTA) approaches for multimodal LLMs, such as ViCrop, our method leverages attention maps to highlight query-relevant regions and is therefore subject to the same dependency. It is important to emphasize, however, that AttWarp is designed TTA mechanism enhance accuracy for visual question answering (VQA), rather than a debiasing approach to correct distribution shift in the model parameters.
To examine robustness under conditions where attention may be less reliable, we conducted additional experiments using synthetic corruptions following the standard IMAGENET-C protocol Hendrycks & Dietterich (2019). Specifically, we evaluated impulse noise, Gaussian noise, and shot noise depict aggressive high-frequency injections applied to the TextVQA dataset. All corrupted images were resized to the standard 512 x 512 resolution for evaluation, and no retraining or adaptation beyond test-time warping was performed.
Table 8: Accuracy (%) of LLaVA and LLaVA+AttWarp under different image corruptions, following the IMAGENET- C protocol.
Corruption
LLaVA
LLaVA + AttWarp
Impulse noise
36.8
40.4
Gaussian noise
37.6
41.0
Shot noise
36.0
39.8
These results demonstrate that while AttWarp inherits the biases of the attention maps it uses, it still provides consistent accuracy gains even under corruption. This analysis shows that AttWarp is not
20

---

Published as a conference paper at ICLR 2026
debiasing method but remains reliable and effective TTA mechanism improve VQA performance in both clean and noisy settings.
Robustness to Adversarial Perturbations. To further probe resilience, introduce targeted adversarial perturbations specifically engineered to misdirect the MLLM's initial attention distribution away from semantically pertinent regions (Fig. 10, top). This adversarial setting provides stringent test of AttWarp 's iterative refinement capability (Sec. 3.2). As illustrated in Fig. 10 (bottom), while the initial adversarial attack successfully perturbs attention and induces faulty warp, subsequent iterations progressively correct the transformation, re-aligning focus with the relevant visual content. This behavior highlights a critical self-correction mechanism: iterative AttWarp is able recover from compromised attention signals, ensuring more reliable grounding even under adversarial conditions.
Question: What brand liquor is on the right? Answer: Bowmore Islay
WMM
ISLAG
Batthhall
Original Image
Adversarial Perturbation
Perturbed Image
OWMOR R ISLAN
OOWOR ISLAY
OWMOR ISLAY SCOTCH
Botel Relawa
Batek Reliauk
Bated Rebeall
10
10
Depth1
Depth
Depth 3
Figure 10: (top row) the original image, the adversarial perturbation designed to corrupt attention, and the resulting perturbed image in which the task-relevant region is shrunk; (bottom row) outputs of AttWarp at Depth 1-3, illustrating how our method progressively overcomes the interference to refocus on the relevant region.
C TASK SPECIFIC ANALYSIS OF ATTWARP
C.I TASK CATEGORIES, BENEFITS, AND LIMITATIONS
AttWarp magnifies the query-relevant region while preserving relative object positions through its rectilinear design. This leads to clear gains in three categories: fine-grained perception, spatial reasoning, and hallucination mitigation. Table 9 reports the improvements across datasets. Across GQA's 15 semantic sub- sub-categories, AttWarp improves performance consistently. The only exception a marginal drop of 0.2% in the conpare-attrine sub-category involving direct size comparisons. This remains the sole degradation observed, underscoring the reliability of the approach.
21

---

Published as a conference paper at ICLR 2026
Table 9; Category-leve improvements of AttWarp. Accuracy in %. Best results in bold.
Task Category
Dataset
LLaVA
Att
A
Fine-grained perception
TextVQA (fine-grained)
49.9
59.6
+9.3
DocVQA (aable/is/ffrmm/anddiiiii
13.6
19.5
+5.9
Spatial reasoning
TextVQA (spatial)
54.5
64.9
+10.4
DocVQA (layout)
29.4
37.6
+8.2
MMMU (spatial)
38.2
44.8
+6.6
GQA (relation)
51.5
56.4
+4,9
Hallucination mitigation
POPE
85.3
87.5
+2.2
GQA (object)
86.1
89.4
+3.3
C.2 DISTORTION AND UPPER- BRUUD CONCERNS
concern is whether warping distorts shapes or positional reasoning. By design. AttWarp preserves grid alignment and relative geometry. This is reflected in GQA's semantic dimensions shown in Table 10, where accuracy improves across all categories.
Table 10: Performance on GQA semantic categories. Accuracy in %.
Category
LLaVA VA
AttWarp
Relation
51.5
56.4 (+4.9)
Attribute
67.8
69.3 (+1.5)
Category
51.7
55.1 (43.4)
Object
86.1
89.4 (43.3)
Global
62.5
65.5 (43.0)
Manual sub-sampling from other datasets confirms similar gains. AttWarp boosts positional reasoning and object shape sensitivity across TextVQA, DocVQA, and MMMU as shown Table 11.
Table 11: Cross-datasct improvements on positional reasoning and object shape. Accuracy in %.
Dataset
Category
LLaVA
AttWarp
TextVQA
Positional reasoning
54.5
64.9
+10.4
DoA
Layout (positional)
29.4
37.6
+8.2
MMMU
Object shape
36.6
40.7
+4.1
DocVQA
Diagram (object shape)
18.8
22.2
+3.4
These results show that AttWarp enhances rather than constrains reasoning involving spatial relations, positions, and shapes.
C.3 GLOBAL CONTEXT AND WARPING INTENSITY
For questions requiring global context, AttWarp adapts the strength of warping to the input. We quantify this with a warping intensity metric, defined as the mean log-change of grid-cell Jacobians. Fine-grained queries undergo stronger magnification, while global queries only mildly affected.
22

---

Published as a conference paper at ICLR 2026
Table 12: Warping intensity by question type. Values mean logche of Jacobian determinants.
Category
E| log (det J
Attribute
0.19
Category
0.18
Global
0.16
Object
0.19
Relation
0.22
Fine-grat grained
0.26
Fine tiee-graaned questions produce ~30% area change, amplifying small detail-rich regions. Global questions produce only a 17 % change, preserving scene layout while still enabling mild emphasis. This explains performance increase global tasks.
C.4 TASK-WISE COMPARISON WITH VICROP
ViCrop uses dual inputs (original and cropped images). natural concern is whether this dual dual-image design provides an advantage. Table 13 shows that AttWarp consistently outperforms ViCrop across semantic categories on GQA.
Table 13: Comparison with ViCrop on GQA semantic categories. Accuracy in %. Best results in bold.
Category
LLaVA
ViCrop
AutWarp
Relation
51.5
51.9
56.4
Attribute
67.8
68.2
69.3
Category
51.7
52.0
55.1
Object
86.1
86.3
89.4
Global
62.5
63.4
65.5
This analysis shows that AttWarp only avoids the pitfalls of dual-image inputs but also provides consistent improvements across all semantic dimensions.
D ADDITIONAL EXPERIMENTS
D.1 EXTENDED GENERALIZATION EXPERIMENTS
further examined the generalizability of AttWarp by applying to two additional multimodal LLMs: the recently released InternVI-2 8B (opensourced in April 2025) Zhu al. (2025) and InstructBLIP Dai et al. (2023). For both models, attention maps were extracted following the same protocol used for LLa and Qwen.
InstrucIBLIP On TEXTVQA and GQA, AttWarp improves over both the baseline InstructBLIP and the ViCrop baseline. Moreover, the chained variant (AttWarp-Chain) yields additional gains, confirming that iterative refinement can further sharpen iitention-driven warping.
Intern VL-3 8B On TEXTVQA and GQA, AttWarp consistently outperforms both the strong baseline Intern VL-3 and ViCrop. These results indicate that the benefits of our approach transfer effectively even cutting-edge architectures trained at scale.
23

---

Published as a conference paper at ICLR 2026
Table 14:
Generalization of AttWarp across InstructBLIP and InternVL-3.. Evaluation metrie: accuracy (%).
Best results bold.
Method
InstructBLIP TextVQA
Intern 8B
GQA
TextVQa
GQA
Baseline
35.2
49.4
80.2
61.4
ViCrop 46.6
49.7
82.7
63.9
AttWarp
47.8
51.3
84.6
65.9
Together, these extended evaluations reaffirm the robustness of AttWarp across diverse set of multimodal LLMs spanningnnnnngggnn both established architectures (InstructBLIP) and the latest state-of-the art models (nnterVV--3) The consistent improvements over strong baselines and ViCrop highlight AttWarp as broadly applicable, model-agnostic test-time adaptation mechanism.
D.2 FINE-GRAINED AND CATEGORY- WISE RESULTS ON DOCVQA
To further evaluate the capacity of AttWarp for both fine-grained recognition and global structural reasoning, we analyze its performance on the DocVQA dataset across diverse structural categories. These include Free Text, Table, Layout, Form, Handwritten, and Diagram, which together capture the spectrum of challenges in document question answering. All experiments are conducted on images resized 512x512. since original DocVQA images are much larger. This resizing ensures computational feasibility across methods. In particular, for dual-image methods such as ViCrop, processing full-resolution images creates prohibitively many tokens, making the resized setting especially relevant for fair comparison.
Table 15: Category-wise accuracy (%) Voc using 1wen2.5-VL--1 as the base MILM. All results are reported on images resized to 512x512 to ensure computational feasibility across methods. AttWarp consistently outperforms all baselines across every document structure category except Yes/No, where ViCrop performs best. Gains span both fine-grained (Form, Hand-writted Table) and global (Layout, Diagram) reasoning
categories.
Method
Overall
Text
Table
Layout
Form
Hand-written
Diagram
Others
Image
Qwen 77.8
78.1
76.3
85.6
75.8
63.2
79.6
80.0
71.4
82.1
API 68.4
68.9
60.8
79.5
68.9
60.9
72.1
80.0
66.1
64.3
ViCrop AttWarp
82.3
83.0
78.8
87.2
77.9
68.3
80.8
84.8
76.5
96.4
84.1
86.2
79.1
89.9
83.5
71.4
81.5
86.7
82.1
92.9
Table 15 shows that AttWarp delivers consistent and significant improvements over all baselines, only in overall accuracy but also within every category. The method achieves particularly strong gains in challenging fine-grained settings such as Form (+5.6) and Handwritten (+8.2), which require precise localization and interpretation. At the same time, improvements global categories such as Layout (+2.7) and Diagram (+0.7) highlight its ability to capture holistic structural cues. The only exception is the Yes/No category, where ViCrop slightly outperforms our approach. Importantly, these results were computed in the resized 512x512 setting, which avoids the prohibitive token explosion of large original images for dual-image methods like ViCrop, thereby ensuring fair and tractable comparisons. Overall, these findings demonstrate the robustness and versatility of AttWarp for document question answering tasks spanning both local detail and global structure.
D.3 STABILITY AND TERMINATION IN ATTWARP-CHAINS
This subsection provides additional results complementing the discussion in Section 3.2, where we introduced an adaptive stopping criterion for iterative warping. Since AttWarp-chains operates recursively, each warped input influences subsequent attention maps. While this iterative refinement initially sharpens focus and improves accuracy, excessive iterations can lead to over-expansion, noise, and performance degradation. This underscores the importance of principled termination rule.
24

---

Published as a conference paper at ICLR 2026
Figure 11 illustrates this phenomenon TEXTVQA (blue curve) and GQA (orange curve). Accuracy improves significantly the first few iterations rising from ~47% to ~60% on TEXTVQA, and from ~60% to ~44 GQA-bu but then plateaus and declines once the depth exceeds two to three iterations. This decline reflects the recursive instability of fixed-length warping. By contrast, the adaptive AttWarp- Chpain (dashed lines) consistently outperforms all fixed-depth settings by halting precisely when attention distributions stabilize.
65
Accuracy (%)
55
TextVQA
GQA
50
AuWarp-Chail
0
2 LLaVA AuW arp Depth
4
Figure 11: Impact of iterative warping depth accuracy for TEXTVQA and GQA. Fixed-length warping initially improves performance but degrades with excessive iterations recursive instability. The adaptive SttWarp-Chaiie (dashed lines), guided by the KL divergence stopping criterion, consistently achieves the best accuracy while avoiding over-warping.
The adaptive stopping rule monitors changes in successive attention maps and terminates the chain once the distributions converge. as formalized by the KL divergence threshold in Eq. 6. This ensures that refinement halts exactly when attention has sufficiently concentrated on query-relevant regions. In practice, AttWarp-Chain not only prevents instability but also achieves higher accuracy than any fixed-depth configuration while being computationally more efficient by avoiding unnecessary iterations.
D.4 EXTERNAL ATTENTION MAPS
AttWarp is compatible with attention from external models. All prior experiments applied AttWarp using internal cross-attention maps, i.e. extracted from the same MLLM used for down- stream inference. Here probe another axis of generalization: can model A (e.g.. LLaVA) benefit from an image warp constructed from the attention maps model B? Demonstrating such flexibil- ity would establish AttWarp as model-agnostic, general-purpost mechanism independent of the substrate attention provider.
To investigate this, source attention maps from two different classes of external models: (i) the text-to-image generative backbone Stable Diffusion 2.1 Rombach et al. (2022), and (ii) strong multimodal LLM, Qwen-VL. Yang al. (2024a). The warped images are then processed LLaVA. for downstream tasks. Results in Table 16 show that AttWarp consistently improves performance across both TEX and GQA, even when relying on external substrate attention. Notably, MLLMS serve as better sources than generative vision models: attention from Qwen- VL yields the strongest gains (+10% on TEXTVQA: QA. +3.4% on GQA), while Stable Diffusion also provides improvements though of smaller magnitude (+6.7%, +2.2%). Interestingly, even external MLLM attention outperforms the base I.LaVA, underscoring the benefit of stronger substrate models. Complementary experiments transferring attention in the opposite direction i.e. from a weaker LLaVA-7B to strengthen larger LLLVA-34B--a reported in Appendix E.3.
Table 16: Performance of AttWarp using attention from internal vs. external models. Evaluation metric:
accuracy (%). Best results bold.
Method (Source of Attention)
TextVQa
GQA
Base LLaVA
49.3
60.5
+ AttWarp (Internal: LLaVA)
58.1
63.7
+ AttWarp (Stable Diffusion Rombach (2022))
$6.0
62.7
+ AttWarp (Qwen-V VI. Yang et al. (2024a))
59.3
63.9
25

---

Published as a conference paper at ICLR 2026
These findings demonstrate that AttWarp is not restricted to internal attention but generalizes robustly across external sources as well, with stronger multimodal LLMs providing the most effective substrate attention maps.
E BEYOND STANDARD VQA: EXTENDED EXPERIMENTS
In this section, we present several forward-looking experiments to probe the broader applicability and potential of AttWarp. We demonstrate its effectiveness in expanding all query-relevant regions, generalizing to ppen-vocabulary object detection, and even improving the performance larger models using attention maps from smaller ones. Collectively, these results highlight the versatility and extensibility of our approach beyond standard VQA settings.
E. ATTWARP EXPANDS ALL QUERY-RELEE RELEVANT REGIONS
We evaluate whether AttWarp effectively enlarges query-relevant regions using ground-truth bound- ing boxes from prior datasets. Results show that method consistently expands the salient regions, both for single- and multi-object queries.
Table 17: Expansion of query-relevant bounding boxes under AttWarp.
Dataset
% Boxes Expanded
Mean Area Increase
TextVQA (single-region) gRef (multi-region)
94.0
+76%
88.6
+39 %
For TexIVQA (Zhang et al., 2025a), AttWarp expanded 94% of salient bounding boxes, with an average area increase of 76%, confirming its ability to magnify relevant image regions.
To test cases with multiple objects of focus, we used the gRef dataset (Liu et al., 2023a), which provides multiple ground-trute truth bounding boxes per query. Here, AttWarp expanded 88.6 % of target boxes with a mean area increase ratio of 1.39 (+39 % area).
These results demonstrate that AttWarp reliably enlarges query- relevvtel regions, including complex multi-object settings, while preserving their spatial grounding.
E.2 OPEN- VOCABULARY OBJECT DETECTION
We next evaluate the versatility of AttWarp on
Object Detection (OVOD), a setting that extends beyond VQA. In OVOD, the goal is to localize objects in images based free-form referring expressions rather than fixed label set. We use dataset of 10,000 examples to test whether approach improves localization under this challenging setup.
Our pipeline applies AttWarp with LLaVA-7B, guided by each referring expression. The warped image then processed by the LISA-LLaVA-7B framework Lai et al. (2024) to predict bounding boxes, which are mapped back to the original image space using the inverse warp. This enables direct IoU-based evaluation against ground-truth annotations.
Table 18: OVOD performance with LISA-LLaVA-7B. Accuracy in %.
Model
Original Images
Warped Images
A
LISA-LLaVA-7B
54.0
61.0
+7.0
Representative qualitative examples are shown in Fig. 12. These results confirm that AttWarp extends effectively to OVOD, improving localization accuracy by +7%.
26

---

Published as a conference paper at ICLR 2026
Referring expression: Man in black facing us
FREO
Original
Warped
Inverse Warped
Referring expression: Man in black facing us
Original
Warped
Inverse Warped
Referring expression: Girl with yellow shirt
FRED
Original
Warped
Inverse Warped
Figure 12: Qualitative OVOD results with LISA-LLa VA-7B, Each example shows the original, warped, and warped images. Predicted bounding boxes are in blue, ground-truth annotations in red.
E.3 LEVERAGING SMALLER MODELS TO IMPROVE LARGER MODELS
We test whether attention maps from a smaller model can enhance the performance of larger model. Specifically, extract attention maps from LLa VA-155--B to warp input images, which are then processed by the stronger LLaVA-L.6-3BB model.
Table 19: TextVQA accuracy (%) of LLaVA-1.6-34BB with and without image warping using attention maps from LLAVA-1.5-7B.
Model
Without Warping
With Warping (7B maps)
A
LLaVA-1.6-34B
72.6
74.1
+1.5
This result shows that weaker models' attention maps can be repurposed to improve the performance of stronger models, highlighting novel direction for leveraging model complementarities.
F ATTENTION REDISTRIBUTION: REDUCTION IN MODEL UNCERTAINTY
The qualitative examples below exemplify the attention redittiibred effect of AttWarp. Our approach consistently improves and redirects the focus of the model toward query-specific image
27

---

Published as a conference paper at ICLR 2026
regions. This be seen by the increased coverage of relevant bounding boxes and the sharper aligmment of attention peaks with target regions, in contrast to the diffused or off-target patterns often observed in the original attention maps. These visual trends support the quantitative gains reported in the main paper as discussed in Sec. 4.3 and Tab. 6d, where we showed improved localization on TEXTVQA.
Localization precision on GQA. To further validate query-specific focus on more datasets, we extend this study to the GQA dataset, which provides bounding-box annotations. Both Pointing Game accuracy and AM @ @all improve after applying AttWarp. Post-warp, the Pointing Game score rises from 0.412 to 0.419, and AM@all from 0.1 0.154. These gains align with the improvements observed on TEXTVQA and confirm that the rectilinear warp sharpens spatial localization across different benchmarks.
Taken together, the evidence from both TEXTVQA and GQA confirms that AttWarp not only pre- serves global distributional fidelity but also reduces model uncertainty by consistently redistributing attention mass toward query-relevant regions.
Question: What type of hard drive does the computer have? Answer: Macintosh
X
Os X YO S eii Ready Instan
OS te
Original
Warped Output
Question: What number follows the firstff at the top?
Answer: 785
ZEEE FX-790P MANUAL DEL PROPETAARO
FX 787 7XX7
COMETIC IE ROUL
VNNUAL D DEPROPIETADD ARD
Original
Warped Output
28

---

Published as a conference paper at ICLR 2026
Question: What is the alcohol content in this beer?
Answer: 9.5 %
OUD Uestomall
Wes Cuail
OUD
Dest snaali
Westmalle
Trinel
Original
Warped Output
G IMPLEMENTATION DETAILS
This section provides the implementation details for the experiments presented in Sec. 4.2 and Sec, 4.3. We first describe the setup for the quantitative results and experimental studies shown in Sec. 4.2 and Tab. 1. We then detail the configuration for the ablation studies for Sec. 4.3 (external models), We then include extended results of Tab. and additional ablation analysis.
G.1 QUANTITATIVE EVALUATION SETUP
As described in Sec. 3.4, we extract attention maps from the 20th layer for LLaVA-1.5-76 Liu et al. (2024b) and the 16th layer for Qwen2.5-VL-7b Yang et (2024a). The extracted attention maps are resized to the original image resolution for all downstream transformations and evaluation. For dataset-specific transforms in AttWarp. we apply the identity transform to TextVQA Singh al. (2019), DocVQAMt et al. (2021). POPELi et al. (2023), and MMMUYue et al. (2024), while for GQA Hudson & Manning (2019), the SQRT transform used.
All experiments are conducted using 8 NVIDIA A100. For LLaVA-1.5-7b, we use the official implementation! with an input image resolution of 336 x 336 and patch grid of 24 x 24. For Qwen2.5-VL-7b. all images are resized to 512 x 512 due to memory constraints and inference time constraints (e.g. ViCrop for DocVQA takes 90hours of inference time on one H100), which results in slightly reduced baseline accuracy compared to the numbers reported in the Qwen2.5-VL-7b official release.
We set the KL divergence threshold in AttWarp- Chain 0.2 (and compulsory stopping condition of iterations), based on empirical observations. For all basclines (APIYu al. (2024), FGVPYang al. (2023b), Yong al. (2023a), and ViCropZhang implementations. For SoM, each image is segmented with Semantic-SAl (Swin-L). For FGVP, the MLLM (LLaVA/Qwen) is first prompted output query-specific relevant objects; then SAM (ViT-H) and CLIP (ViT-L/14) masks those regions and produce two inputs: one with the background coussian-blurred and one with the masked region highlighted green. Hyperparameters of FGVP and SoM are selected by grid search over 100 randomly sampled images per dataset to select the optimal values. Hyperparameters for SoM: granularity 3, 0.4, text-size = 640; FGVP: blur 5, 0.5, IoU threshold 0.86, stability threshold 0.92, and minimum mask area 400. In the case of API and ViCrop, we exactly use the official implementation (AP?2 and ViCrop²) with the same hyperparameters.
al. (2025a)), use their official
Accuracy used as the evaluation metric for datasets: for each question-answar pair (g, a). the prediction is considered correct if the model output exactly matches with accuracy computed the average over all examples. TextVQA and VQA, we do not provide any ext-extracted tokens
httpp:///giaabb.cmmmaattaaaaaaaa
httpaab..ccmuuupp/ppppppppppptpp githhb..cm/accccaaaayttt////laa know
29

---

Published as a conference paper at ICLR 2026
to the MLLMs- onll the image and question are given, following the evaluation prompt format outlined in the respective papers. Finally, to ensure reproducibility and transparency, we will release all code, configurations, and analysis scripts required to reproduce experiments and results.
G.2 STABLE DIFFUSION EXPERIMENTAL DETAILS
In Appendix Sec. G.2 we use Stable Diffusion for external attention; here, we detail its experimental design. We adapt AttWarp to Stable Diffusson 2.1 (SD-2.1) by first inverting the input image into the model's latent space with a five-step truncated DDIM schedule [1000, 800, 600, 400, 200], conditioning on the question q at every step. Each inverted latent is then forwarded through ten denoising iterations while recording the cross cosssatttentonn tensors of the final two UNet layers previously identified by Hertz al. (2022) as exhibiting particularly sharp token-level groundings. Summing these tensors over heads and space yields token-wise energies from which we retain the top-k=20 tokens, average their channels into single low-resolution attention map, apply a square- root contrast stretch, and together with its inverse on the me--imensiional marginals-uss it to drive single sqrt CDF warp at 500x500 resolution. As reported in Table 16, this external attention improves the vanilla LLaVA baseline by +6.7% % on TexiVQA, yet remains below the gains from internal LLaVA attention (+8.8%) and from the stronger Qwen- WLven-VL<ne (+10.0%). We attribute this gap to task mis- aligmelignment diffusion models are trained for literal scene synthesis (e.g. "a giraffe in a misty forest"), whereas visual question answering centres on self-referential queries (e.g. "what animal is in the forest?"), so the resulting diffusion attention maps, although sharp, do not always highlight regions most informative for answering such questions.
G.3 TRAINING SETUP FOR ATTWARP-DISTIU
We use Qwen-2.5VL 2.BVL--BB as the teacher model to extract attention maps and derive corresponding marginals for training the student model. Specifically, attention maps from the teacher are first extracted, after which category-specific marginals are computed for each dataset. To adaptively scale attention distributions according to task granularity, we apply distinct transformations per dataset: a square-root transform for Doc a cube-root transform for GQA, and an identity transform for VeA and fine-grained datasets such as POPE. The student model is trained for a maximum of 20 epochs, employing early stopping based
validation-set performance. Optimization uses AdamW with a batch size of 16, learning rate of 3 x 10-1 weight decay of 1 x 10-1, and gradient clipping with maximum norm of 1.0. We show the predicted marginals in Fig. 13.
Prodeced Troun--Trush 10rrippot 12)
Producedd Groad- 1o0
XHaginaa
of
Figure 13: The marginal distributions predicted by the student networks closely align with the ground truth marginals, demonstrating robust knowledge transfer.
G.4 COMPUTATIONAL COST CALCULATION AND ANALYSIS
We present a detailed cost analysis to emphasize the efficiency and practical advantages of AttWarp compared to the competitive baseline (ViCrop (Zhang et al., 2025a)). Here, we specifically demon- strate results for -.55-- (similar trends hold true for other models, such as Qwen). Our approach significantly surpasses the FGVP (Yang et al., 2023b), SoM (Yang al., 2023a), and API (Yu et al., 2024) in accuracy. Therefore, we focus this analysis on ViCrop, the competitive baseline, which achieves comparable yet inferior performance to At Att across all datasets (see Sec. 4.2 and
30

---

Published as a conference paper at ICLR 2026
Tab. 1). Computational complexity for all MLLMs is calculated following the methodology described in (Lin et al., 2025; Chen et al., 2023b;;; 2024a).
processes single 336 336 image through VIT-L/14 encoder, producing 576 visual tokens and incurring a computational cost of 8.5 TFLOPs. In contrast, AttWarp requires two MILM passes per query: the first pass extracts attention maps up specific layer (e.g-, the 20th layer of LLaVA, where we insert hook), while the second pass generates the final output from the warped image based on these attention maps. The total cost for AttWarp -including both passes and the lightweight warping operation-corresppee The cost of the warping step itself is negligible.
Standard
to 1,152 vision tokens (2 x 576) and 13.8 TFLOPs.
In comparison, ViCrop adopts a more resource- intennsve pipeline, requiring three MLLM passes: two initial passes obtain relative attention maps (up the 14th layer), followed by final forward pass processing both the original and cropped image. This results in significantly larger number of vision tokens (2,304) and total computational load of 24.2 TFLOPs.
As summarized in Tab. 3, AttWarp delivers substantial computational efficiency. Specifically, it reduces the total number of vision tokens by factor of two relative to ViCrop (1,152 vs. 2,304) and requires 1.5 times fewer MILM passes (2 3), translating to a 1.8x reduction in compute cost (13.8 TFLOPs for AttWarp vv. 24.2 TFLOPs for ViCrop per query).
terms of memory, AttWarp achieves peak VRAM usage of 15.5 GB, identical to the base LLaVA-1.5y-77 model and within the capacity of standard GPUs (e.g., V100s). The peak VRAM required for ViCrop is 22 GB, significantly higher (by a factor of 1.46) compared AttWarp.
These results strongly motivate the use of AttWarp, demonstrating that it not only achieves superior accuracy (see Sec. 4.2 and Tab. 1), but also yields substantial savings in computational resources compared to approaches like ViCrop.
H QUALITATIVE EXAMPLES
Here, we present examples demonstrating the effectiveness both AttWarp and AttWarp-Chain. For each case, the predicted answers displayed beneath the corresponding images. For AttWarp-Chaal Chain, results are at varying depths (as expected), illustrating how our method adap- tively refines warping based on the specific query and image content. This depth-wise adaptation underscores the robustness and flexibility of our approach, enabling consistently strong performance across diverse queries and visual contexts.
H.1 ATTWARP
Example
Question:
What material is the chair made of?
Original
Attention Map
Warped Output
Qwen: Steel X
AttWarp: Plywood /
31

---

Published as a conference paper at ICLR 2026
Example 2
Question:
Is the vehicle behind donkeys?
Original
Attention Map
Warped Output
Qwen: Yes X
AttWarp: No /
H.2 ATTWARP-CHAIN
Example 3
Question:
How many apples are there in the image?
Original
Depth 1
Depth
LLaVA: 3
AttWarp: 3
AttWarp: 4 /
Example 4
Question:
What is a player's number?
Original
Depth 1
Depth
327
321
321
Qwen: 22 X
AttWarp: 22 X
AttWarp: 27 /
32

---

Published as a conference paper at ICLR 2026
Example 5
Question:
What year is this whiskey from?
Original
Depth
Depth
Depth
CCOPPERS CHO
LLaVA: 1990 X
AttWarp: 1996
AttWarp: 2006 X
AttWarp: 1995 V
I FID AND KID ANALYSIS
We quantify how closely each test variant remains on the training manifold using two complementary. CLIP-feature based distances: Fréchet Inception Distance (FID) and Kernel Inception Distance (KID). Lower values indicate closer alignment to the training distribution.
Why rectilinear warping preserves the distribution. Our warp is axis-aligned and monotone,
g = (z)( Fy())).
so its Jacobian is diagonal. At the token level this induces only per-axis rescaling of VIT patches while keeping grid orthogonality. This mirrors CLIP's resize/crop pre-training augmentation (e.g.. RandomResizedCropp, which likewise applies axis-wise scaling before tokenization. Consequently, the pooled CLIP image embeddings for rectilinearly warped images are expected to stay on (or near) the same feature manifold as training images.
Orig (test)
200
Orig (test)
40
KID x10 (1 closer to train)
AuWarp
KID x10 (4 closer to train)
ArtWarp
Non- Con-Rectilinear Warp
Non- non-Rectillinar Warp
150
30
100
20
TTst
%
0
10
15
20
40
60
80
FID (1 closer to train)
FID (1 closer to train)
(a) FID KID/X 10 distances between the training distribution and each test variant (Original, AttWarp. RottRectllinear Warp) in CLIP VIT-B/32 space. Points Non-Rectilinear Warp) in CLIP VIT-L/14 space. nearer the lower-left are closer to the training manifold.
(b) KID/ 10 distances between the training distribution and each test variant (Original, Atrrppp
Figure 14: Comparison of Train->(Test, AttWarp, Non-Rectilinear) FID-KID across two ViT backbones.
Setup. We reuse 12,000 GQA-TRAIN images to define the training reference used throughout the distributional analyses (also used for the Mahalanobis study in Section 4.3). Image embeddings are obtained with CLIP VIT-B/32 and VIT-I/144 backbones as follows:
33

---

Published as a conference paper at ICLR 2026
1. One embedding per image. We use CLIP's global pooled image embedding returned by encode_ l_ieagemmeeeeee (i.e., the projected [CLS] / pooled token). This is the representation CLIP trains to align with text, it is therefore the correct summary statistic for distributional tests. We do not mean-pool patch tokens.
2. Reference model. Fit a single full-covariance Gaussian W(u, Σ) on the training embeddings. A full covariance is required because CLIP feature dimensions are correlated; diagonal covariances overstate distances along correlated axes. We do not use a Gaussian mixture.
3. Mahalanobis diagnostic (used for histograms referenced in Fig. 6a). For each embedding x in split S {Orig (test), AttWarp, Non Rectilinear Sereeaaaaaar Warp). compute a single distance to the training Gaussian:
(as(z) V -1- - -z) p))
The histogram is built from (dv(x) x E 53. is a 12k-choose-2 pairwise computation. With population parameters, dm (z)² follows x²: with estimated parameters it approximates Hotelling's T
4. FID/KID. Using cleanfid Parmar et al. (2022) with default settings (64 batch. 8 workers, CLIP 224 preprocessing), compute Train distances for each backbone. The Gaussian in step 2 is fit once on train; all eval splits are compared to that same reference to reveal shift (we never refit on test/warped sets).
Results. Fig. 14 shows Train->Test distances for both backbones. In ViT-B/32 (Fig. 14a) and ViT-L/14 (Fig. 14b), AttWarp (orange circle) stays within the natural
gap, reducing FID (13.8 for B/32; 56.6->49.8 for L/14) with only mild KID changes (8.3~111.0: 19.3+ In contrast, the Non-Rectilinear warp (green triangle) produces a clear shift-FID rises to 15.5 and 73.9 and KID inflates markedly (37.6 and 174.9) connsst-nnsstteett with the Mahalanobis drift observed in Fig. 6a. These results quantitatively substantiate the design rationale above: rectilincar, exis-aligned warping preserves CLIP-fcatur distributional integrity, whereas free-form distortion does not.
J ALTERNATIVE INTERPRETATION, WARPING THE VISUAL TOKENIZATION GRID
AttWarp is described in the main text query- conditione image warp followed by the model's standard visual tokenization. Concretely, from an attention score matrix we compute 1D marginal profiles and their liverse-CDF warping functions (Sec. 3.1, Eqs. (1),3)), and generate the warped image by inverse-wary warp sampling (Eq. (4)). The vision encoder then converst the warped image into patches using its usual uniform grid, producing regular hieat x Wfeat lattice of visual tokens (Sec. 3.4).
Grid view The same computation admits an equivalent interpretation in which we keep the original image fixed and instead warp the patch grid. Let the model's standard tokenization partition the (warped) image domain into hreat x Wiett axis-aligned cells with uniform boundaries in the warped coordinate system. Mapping those cell boundaries back through the inverse warp implied by Eq. (3) yields non-uniform, query-dependent set of cut locations in the original image coordinates. Because the warp is rectilinear and monotone, these mapped boundaries remain ordered and define valid axis- aligned rectangular tiling of the original image. Thus, "warp-then teep--theettkkizze uniformly" is equivalent to "tokenize the original image with an attention-guided local resampling that is already implicit in Eq. (4)). Intuitively, AttWarp reallocates a fixed token budget: dense (small) cells are assigned to high-attention regions, while sparse (large) cells cover low-attention areas, preserving global context since no region is discarded.
non-uniform grid" (followed by the same
AttWarp-Distine (Sec. 3.3), the student trained to predict the horizontal and vertical marginals that parameterize the same inverse-CDP warp used by AttWarp. Under the grid interpretation above, predicting these marginals is equivalent to predicting the grid-cutting rule itself: the student directly outputs query-conditioned visual tokenization lattice (i.e., where the patch boundaries should fall, up to the model's 's fixed token budget). In this sense, AttWarp-Dist Distilll can be viewed training amortized, guery-conditioned visual tokenizer ppatch-grid generator) that approximates the teacher model's 's attention- induced token allocation, while leaving the underfying MILMM architecture and tokenizer unchanged.
Implication for AttWarp-Distil.
34


Reviews:
[{"review_id": "2ZquhK6xiH", "reviewer": "Reviewer_6v6J", "summary": "This paper introduces Constructive Distortion, a training strategy for large vision-language models designed to enhance robustness and fine-grained understanding through targeted visual perturbations. Instead of random noise or masking, the approach applies semantically constructive distortions—guided transformations (e.g., spatial deformation, contrast warping) that preserve semantics while challenging the model’s visual encoder. The method aims to improve generalization to distorted or out-of-distribution visual inputs without sacrificing in-distribution performance. Experiments on benchmarks such as MM-Vet, MME, and LLaVA-Bench demonstrate consistent improvements, particularly under degraded or perturbed visual conditions.", "questions": "Could you please provide more comprehensive evaluation (e.g., VQAv2) and (attention-based) token compression baseline?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Novel concept: Shifts from destructive to constructive data augmentation, promoting robustness while maintaining semantic fidelity.\n\n- Strong empirical results across diverse LVLMs and benchmarks, with detailed ablations on different distortion types and intensities.\n\n- Practical contribution: The approach is plug-and-play and compatible with existing LVLM training pipelines.\n\n- Clear motivation and presentation: The paper is well-written and the concept of “constructive” perturbation is intuitively appealing.", "weaknesses": "Lack of comparison with visual token compression methods. The paper does not contextualize its approach relative to recent efficient LVLM frameworks that also modify the visual representation process. Works such as [1] PVC: Progressive Visual Token Compression (Yang et al., 2024), [2] Efficient Large Multi-Modal Models via Visual Context Compression (Chen et al., NeurIPS 2024), and [3] An Image Is Worth 1/2 Tokens After Layer 2 (Chen et al., ECCV 2024) explore representation simplification and robustness trade-offs at the token level. Including comparisons or discussion would clarify whether constructive distortions yield complementary or competing benefits.\n\n[1] Yang C, Dong X, Zhu X, et al. PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision- \n\n[2] Chen J, Ye L, He J, et al. Efficient large multi-modal models via visual context compression[J]. Advances in Neural Information Processing Systems, 2024, 37: 73986-74007. \n\n[3] Chen L, Zhao H, Liu T, et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 19-35.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HMImKGoTqw", "reviewer": "Reviewer_BFqe", "summary": "This paper proposes AttWarp, a plug-and-play, attention-guided image warping method that improves fine-grained perception in multimodal large language models (MLLMs) without modifying their architecture or parameters. The method reallocates spatial resolution toward query-relevant regions using the model’s own cross-modal attention, yielding consistent gains across multiple benchmarks and MLLM backbones.", "questions": "please see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of leveraging model attention to reshape the input space rather than internal representations is conceptually elegant and complementary to existing attention-tuning methods.\n\n- The method is tested on five diverse benchmarks and four architectures, showing consistent improvements with detailed ablations.\n\n- The approach is simple, lightweight, and does not require retraining.\n\n- The paper includes ablations on attention quality, warping stability, and distributional integrity, which strengthen its credibility.", "weaknesses": "- The contribution of this paper feels more like a clever engineering refinement than a fundamentally new paradigm.\n\n- The method assumes reliable attention maps; performance may degrade under noisy or misaligned attention, but this limitation is only briefly mentioned.\n\n- The paper’s justification for why rectilinear warping improves reasoning remains empirical and lacks a more formal analysis of perceptual geometry or attention dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CRrsq4w6zu", "reviewer": "Reviewer_Tmyc", "summary": "The paper introduces **AttWarp**, a lightweight method for improving multimodal large language models' (MLLMs') fine-grained perceptual grounding in cluttered scenes. AttWarp leverages an MLLM's cross-modal attention to perform rectilinear warping on input images during testing, reallocating resolution to query-relevant areas while preserving global context and image information.  Without modifying model weights or architecture, this attention-guided warping enhances the readability of small objects and subtle relationships. Experiments show consistent accuracy improvements across five benchmarks (TextVQA, GQA, DocVQA, POPE, MMMU) and four MLLMs (LLaVA, Qwen-VL, InternVL, and InstructBLIP), outperforming competitive baselines. These results highlight AttWarp's ability to optimize spatial resolution for query-relevant content while preserving global structure, boosting MLLM performance with warped inputs.", "questions": "Do you think applying reinforcement learning (RL) to AttWarp could further enhance its capabilities, using MLLMs to validate the effectiveness of perturbed images in solving queries and providing reward feedback?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clarity and Ease of Use**: The paper is well-written, easy to understand, and straightforward to follow. The proposed method, AttWarp, is plug-and-play, delivering significant performance improvements without requiring additional training.  \n\n2. **Intuitive and Novel Approach**: Using attention feedback to enhance the resolution of focus areas is an intuitive yet innovative idea. The method avoids retraining models while achieving substantial gains, making it particularly practical and impactful.  \n\n3. **Comprehensive Validation**: The exploration of AttWarp through two variations—AttWarp-Chain and AttWarp-Distill—effectively demonstrates the feasibility and upper bounds of the method's generalization capabilities. Its success across multiple multimodal language models and benchmarks further validates the robustness and versatility of the approach, making it an interesting and promising contribution.", "weaknesses": "While AttWarp has demonstrated significant improvements across a range of text-centric multimodal tasks (e.g., TextVQA, GQA, DocVQA, POPE, MMMU), it lacks evaluation on visual-centric benchmarks that focus more on fine-grained visual perception. Tasks such as **MMVP**, **BLINK**, **RealWorldQA**, and **MIA**, which emphasize nuanced visual grounding and object-level understanding, are particularly relevant for showcasing the strengths of AttWarp's attention-guided resolution reallocation.  \n\nIncluding these visual-centric evaluations could illustrate its potential for enhancing visual perception capabilities further, as the method is well-suited to improve the detection of subtle details and spatial relationships that are critical for such tasks. Without this, the generalizability and impact of AttWarp on visually demanding applications remain underexplored. Evaluating the method on these benchmarks would provide a more comprehensive picture of its capabilities and further highlight its benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fTn4Xh2Xqb", "reviewer": "Reviewer_J5yH", "summary": "This paper introduces AttWarp, a lightweight, plug-and-play image warping technique designed to improve the fine-grained perceptual grounding of MLLMs. The method begins by extracting cross-modal attention maps from the MLLM's internal layers based on a given image and query. These attention maps are then used to guide a rectilinear warping of the input image, which reallocates spatial resolution to magnify query-relevant regions while compressing less informative areas. The authors demonstrate consistent performance gains across five diverse benchmarks and four different MLLM architectures, showcasing the method's effectiveness and generalizability.", "questions": "1. Is there a pattern in the indices of optimal attention layers across different MLLMs? Can we approximately determine which layer is most suitable for AttWarp?\n\n2. Are there methods to mitigate the limitations of AttWarp in cases such as size and hallucination?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. As a model-agnostic enhancement, AttWarp is a practical plug-and-play solution that can be readily applied to improve existing models without retraining.\n\n2. The paper introduces a complete framework, including an iterative version (AttWarp-Chain) for hard cases and a distilled version (AttWarp-Distill) for applications.\n\n3. The work provides compelling ablations and analyses that validate its design choices. A key finding is that the rectilinear nature of the warp preserves the underlying feature distribution of the images, thus avoiding the out-of-distribution issues that can plague other image manipulation techniques.\n\n4. The method shows consistent performance gains across a wide range of benchmarks and MLLM architectures.", "weaknesses": "1. The claim of being \"plug-and-play\" is slightly weakened by the need to identify the optimal attention layer for each new MLLM architecture (e.g., layer 20 for LLaVA, layer 16 for Qwen-VL). This requires an empirical, model-by-model search, which adds a setup cost for new models.\n\n2. In Error Analysis, the authors state that AttWarp is prone to errors in cases such as  size, hallucination, and misaligned attention. Have the authors attempted any framework modifications to specifically address these failure cases? For example, have they considered introducing a classifier to determine when to apply AttWarp?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)