Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
Authors: Yuheng Shi
Xiaohuan Pei
Minjing Dong
Chang Xu
Venue: ICLR 2026 Poster (2026)
Keywords: MLLM
Self-Distillation
Fine-Grained Perception
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose an efficient method to improve MLLM's fine-grained perception by training a module to predict Regions-of-Interest using clean pseudo-labels distilled from the model's own noisy attention maps.

Abstract: Multimodal Large Language Models (MLLMs) require high-resolution visual information to perform fine-grained perception, yet processing entire high-resolution images is computationally prohibitive. 
While recent methods leverage a Region-of-Interest (RoI) mechanism to focus on salient areas, they typically present a difficult trade-off: training-based approaches depend on large-scale annotated datasets, while training-free methods that utilize the model's internal attention are computationally inefficient and less accurate, requiring either multi-pass prefill stages or reliance on the slow auto-regressive decoding process.
In this paper, we propose an efficient, annotation-free Self-Distilled Region Proposal Network (SD-RPN) that resolves this trade-off. The SD-RPN is built around a pipeline that transforms the noisy attention maps from the MLLM's middle layers into high-quality pseudo-RoI labels by explicitly denoising the signal and resolving ambiguity. We use these labels to train a lightweight Region Proposal Network (RPN) that learns a more precise localization. This RPN is also highly efficient, predicting the RoI in a single forward pass using features from the MLLM's middle layers, decoupling RoI identification from the auto-regressive generation and avoiding costly multi-pass operations.
To validate our approach, we integrate the framework into the LLaVA-1.5 architecture. Despite being trained on only a few (e.g. 10K) question-answer pairs, our method demonstrates exceptional data efficiency and generalization, achieving over a 10\% absolute accuracy improvement on unseen benchmarks, including TextVQA, DocVQA, and V-Star. Our work presents a practical and scalable solution for enhancing the fine-grained perception of MLLMs without requiring costly supervision or full model fine-tuning. Code is available at https://github.com/YuHengsss/SD-RPN .

Published as a conference paper at ICLR 2026
CATCHING THE DETAILS: SELF-DISTILLED Rol PRE- DICTORS FOR FINE-GRAINED MLLM PERCEPTION
Yuheng Shi
Xiaohuan Pei' Minjing Dong²
Chang Xu'
University of Sydney Ciity University of Hong Kong
yshi00878uni. syyo:/ sydney. edu. au, xiaohuan. pei @ syddnyy edu. au, jinn edu.hk, c.xuesydney edu. au
ABSTRACT
Multimodal Large Language Models (MLLMs) require high-resolution visual information to perform fine-grained perception, yet processing entire high- resolution images is computationally prohibítive. While recent methods leverage Regiio-of-Innerest (Rol) mechanism to focus salient areas, they typically present a difficult trade-off: training- based approaches depend on large-scale an- notated datasets, while training-free methods that utilize the model's internal at- tention are computationally inefficient and less accurate, requiring either multi- pass prefill stages or reliance on the slow auto-regressive decoding process. In this paper, we propose an efficient, annotation-fire -free Soff-Distilled Region Proposal Network (SD-RPN) that resolves this trade-off. The SD-RPN is built around a pipeline that transforms the noisy attention maps from the MLLM's ' middle layers into high-quality pseudo-Ro Rol labels by explicitly denoising the signal and resolv- ing ambiguity. We use these labels to train Region Proposal Network (RPN) that learns more precise localization. This RPN is also highly efficient, pre- dicting the Rol in single forward pass using features from the MLLM's mid- dle layers, decoupling Rol identification from the auto-regressive generation and avoiding costly musti-pass operations. To validate approach, we integrate the framework into the LLaVA-1.5 architecture. Despite being trained on only few (e.g. 10K) guestion-answer pairs, method demonstrates exceptional data effi- ciency and generalization, achieving over 10% absolute accuracy improvement unseen benchmarks, including Tex(VQA, DocVQA, and V-Star. Our work presents practical and scalable solution for enhancing the fine fine-graing geeeggrinnd pereep- tion of MILMs without requiring costly supervision full model fine-tuni ane-tuning. Code is available at htttt::////e /ttp:///ggtuuu...//// thub. com www.nnnss//SS--n-n...
INTRODUCTION
Recent years have witnessed significant advancements in Multimodal Large Language Models
(MLLMss, which have evolved from foundational architectures like LLaVV (Liu et al. 2023b) to
more sophisticated systems (Wang al. 2024a Bai et al.) 2023 Chen al., 2024a such as
Qwen2.5-VL (Bai et al. 2025) and Intern VL-3.0 (Zuu et al. 2025). The performance of MLLMs
tied the quality of their visual perception. In typical MLLM architecture, a vision encoder,
such as CLIP (Radford et al. 2021 processes visual signals and projects them into the embedding space of Large Language Model (LLM) (Touvron et al. [203) Zheng al. (023) for subsequent reasoning. Consequently, the richness of these visual features is crucial for the model to achieve comprehensive and -raiined understanding of the input.
To enhance this fine-grained perception, scaling the resolution of visual inputs has emerged direct and effective strategy. The approaches to achieve this have evolved considerably. Initial meth- ods (Liu et al. (203bb Dai et al. 2023 Bai et al. (202)) relied on fixed-resolution input. Subsequent research introduced more flexible techniques, such as S (Si et al. 2024). Any- iny-Resolution (Liu 2023a; Chen al. 2024b), and Naive Dynamic Resolution (Wang al. 2024a; Bai et al.
et al.
2025 to handle higher-resolutee imagery. However, processing entire high-resolutin images uni-
formly is computationally intensive. More recently, alternative paradigm (Yu al. 2025
Shi

---

Published as a conference paper at ICLR 2026
Training Stage
Inference Stage
Pscudo-Label Generation
1.2
Response: Blue star.
Stenslankkk
Layer B+R
Init.
0.5
Mobetive thrrggher
Layer B+1
LM Head
7ML7
Layer &
Layer B
Layer B
:
Layer 1
Weights
Share
Layer
Layer
Necmug
27.5
No
D00
Q: Which gallery will they be found Blue stur. (Training only)
Rel image. (Inference only)
Ikeeite Vlc mimppin
Ioolin 5 Minnporr
(a) Figure 1: (a) The Pipeline SD-RPN The RPN is trained with pseudo-labels effectively predict Rols. These Rols are then used to crop fine-grained sub-images for the final inference stage. (b) Performance Comparison. Performance evaluation with S (Shi et al., (024) and ViCrop (Zhang ct al. (225a)) on the LLaVA-1,5-7B baseline. Accuracy is averaged over five Document and OCR
benchmarks.
et a 2025a) has gained traction: identifying a Region-of- ofgion--o-Innerest Interest (Rol) within a low-resolution in- put and then selectively scaling up only that specific region. This Rol-centric approach has proven to be more efficient and effective means of improving visual detail perception.
While the Rol paradigm marks a significant step toward efficient high-resolution processing, cur- rent methodologies for identifying these regions still present notable limitations. Approaches such as VILA-HD (Shi et al. 2025a), for instance, rely on large-scale pre-training with detailed annota- tions, a process that is both data-intensive and computationally demanding. Furthermore, they often require a complete prefilling stage for the initial low-resolution image, which can impede both train- ing and inference efficiency. An alternative direction leverages the intrinsic localization capabilities of MLLMs (Kang et al.] 2025b; Zhang et al.] 2025a) without training, specifically by computing cross-attention scores between image tokens and the corresponding textual description. However, leveraging this internal attention for Rol identification is often computationally inefficient in infer- ence, as current methods typically require either complex, multi-pase pastipppssssss operations during the prefill stage (Zhang et al. (025a) or rely on the inherently sequential and slow auto-regressive decoding stage (Wang et al., 2024b). Consequently, an effective and efficient method for Rol identification that avoids both rehancc on extensive annotated data and the high latency of auto-regressive decod- ing or multiple forward passes remains a critical, yet underexplored, challenge.
In this work, we introduce SD-RPN, novel self-distillation framework (as shown in Fig. Ia de- signed to overcome these limitations by efficiently harnessing the MILM's intrinsic localization capabilities. Our approach is motivated by the insight that while an MI.LM's internal attention provides a strong Rol signal, it is too noisy (e.g., attention sinks, incomplete activation) for direct supervision. Previous studies reveal that using such noisy signals for dense supervision yields sub- optimal results (Wang et al., 2022 Tian et al. 2021). To address this, we propose a pseudo-labeling pipeline that transforms the noisy attention map into sparse and reliable supervisory signal. This pipeline first denoises the map by removing sink tokens based on feature norms. Subsequently, it employs a selective classification strategy that assigns discrete labels based on confidence thresholds and a minimal bounding box around high-attention tokens, which resolves the ambiguity inherent in the original noisy map. We use these pseudo labels train RPN. By learning from this dis- tilled knowledge, the RPN develops a more precise localization function than methods relying on raw, ambiguous attention. Beyond its accuracy, the RPN's architectural design confers significant efficiency gains. Composed of a few transformer blocks built upon the frozen MILM backbone, the RPN operates on features from the model's middle layers. This strategic placement allows it to pre- dict the Rol by executing only partial forward pass up to the MLLM's middle layers, completely decoupling localization from the slow, auto-regressive generation process. The entire framework is trained end-to-end, distilling the localization knowledge from the model's tention into the efficient RPN. This enables a dynamic two-stage inference process where the model first predicts salient regions and then analyzes high-resolution crops to generate its final response.
cisponse-guider at-
2

---

Published as a conference paper at ICLR 2026
Our SD-RPN achieves significant enhancement (Fig. lbp in fine-grained perception without the cost of full-model fine-tuning or large-scale annotated datasets. Moreover, our framework demon- strates exceptional training efficiency and generalization. Using only 10K random samples from GQA (Hudson & Manning 2019) and OCR-V -VA (Mishra et al. 2019) within the LLaVA-1.5 (Liu et al.
2023a) framework, our approach yields substantial gains unseen data, boosting accuracy by over 10% on TexIVQA (Singh et al., 2019), Doc (Mathew et al., 2020), and V-Star (Wu & Xie, 2023). We further confirm the robustness of our method by applying to the more recent BeepSeek-VL. (Lu et al., 2024) and Qwen2.5-V VL (Bai et al. (025) architectures. Our contributions are twofold. First, we introduce a robust pipeline to denoise the internal attention maps of an MLLM, generating high-quality pseudo-labels for supervision. Second, we propose novel, annotation- self-distillation framework that trains a RPN predict Rols by leveraging the MILM's s intrinsic localization knowledge.
2
RELATED WORKS
2.1 PERCEPTION IN MLLMS.
Recent studies have established that MILMs often struggle with fine-grained perception, limitation rooted in the challenge of efficiently processing high-resolution visual inputs (Tong et al.) 2024b
Kang et al. 2025b; Shh al., 2024 Liu et al. 2024a] Chen al.]
2024a). One line of research
has focused on enhancing the model's s global visual understanding.
This includes developing more
sophisticated vision encoders (Wang et al. 2024a Bai al. 2025 Zhang al.
2024b Ge et al,,
2024 Luo et al. 2024), supplementing the LLM with full high-resolution images [Lu et al. 2024a
Zhu et al. 2025 Huang et al. 2024 Tong et al.] 2024a Cha et al., 2024 Li et al. 2025] (022bb)
and incorporating external tools (Zhao et al. 2024). More recent works (Shi et al. (2025a); Yu jet al. (2025); Zheng et al. (205); Shao et al. (2024) have demonstrated that identifying the Rol first in relatively low-resolution visual input and then scaling the resolution specifically for the Rol is more efficient and effective. However, they come at the expense of extensive training, requiring massive supervision and costly annotations. While recent training-free methods (Zhang et al. 2025a Wang et al., 2024b. [Shi et al. (2025b) attempt to identify Rols by utilizing the internal perceptual capabilities of VLMs, they are often hampered by noisy activations or require slow auto-regresssive decoding stage and multiple forward passes, which hinders both performance and efficiency.
2.2 SELF-DISTILLATION IN MILMS.
Knowledge distillation is an effective paradigm for transferring knowledge from a teacher to a stu- dent network (Hinton et al. 2015). Among its variants, self-distillation (Zhang et al. 2019) emerges as unique form which the teacher and student share the same architecture, and has been widely adopted in vision-languaa
tasks. In multimodal pre-training, several works (Oquab et al. 2023 Cai
jet al. 2024; Dong et al., 2023 Kim et al. 2025) leverage self-distillation to improve cross-modal alignment and representation learning, where it is commonly employed to enhance feature extrac- tion. In multimodal downstream tasks, recent efforts (Kong et al.] 2024 Peng et al. 2025 Hou et al.) 2024) take advantage of self-distillation to refine vision-text region alignment and thereby improve grounding and reasoning performance. As self-distillation eliminates the need for a larger pre-trained teacher, it naturally offers strong potential for extracting high-resolution and fine-grained perceptual cues directly from MLLMs themselves.
METHOD
3.1 PRELIMINARIES
The standard architecture of a Multimodal Large Language Model (MLLM) comprises three core components: a vision encoder, E,, a vision-language projector, P, and a Large Language Model (LLM), L. Given an image-text pair, (x,, the MLLM processes the inputs to generate tex- tual response. The vision encoder ε, extracts feature vectors from the input image Ee. These visual features are then transformed by the projector P into sequence of visual embeddings, H = (x,), which are aligned with the LLM's input space. The input text x, is pro- cessed by a tokenizer and an embedding layer to produce sequence of text embeddings, which
3

---

Published as a conference paper at ICLR 2026
image attention
Pseudo-Label Generation
Response
Vision Encoder
Rol
Remove sink
MgØl
ussignment
Rol
tokens
Label
Layer I
Layer!
Question: Are there any red umbrellas or motoreycles?
MLLM:
Sink token
PG bounding box
FG token
BG token
Ignorod token
Figure 2: An overview of our pseudo-label generation pipeline. FG and BG denote the foreground and background respectively. Layer index is omitted for simplicity.
typically includes system prompt and user query, yielding H ays and H encoding H H H user in parallel (called the prefilling stage), the MLLM generates responses auto-regressively (called the decoding stage). Due the lack of parallelization, the decoding stage typically much slower than the prefilling stage when processing the same number of tokens.
uueey respectively. After
The attention scores are key to identifying image regions relevant to the text (Kang et al. 2025b) From the cross-modal attention in layer represents the averaged importance of each visual token across textual tokens of user query or response. For single attention head, this map is computed as M softmax (K())) /Va), where Q/ E IR xi and K ((Fxw)) wd are the query and key matri- ces from the N₂ response tokens and H W visual tokens, respectively,
could derive Rol map, M Rot E RR which
Σ A//N,, A
3.2 PSEUDO-LABEL GENERATION FOR Rol
Allentinn Scoee Douat Soake)
Ground
0.8
While the Rol map MRØI provides valuable signal for local- izing text-relevant image regions, it is often fraught with noise. As illustrated Fig. 2 these raw maps can exhibit erroneously high attention in background areas and incomplete activation across the true foreground object. To overcome this, we pro- pose a pseudo-label generation pipeline, which is depicted in Fig. 2 to transform the noisy Rol map into a sparse and reli- able supervisory signal for training our RPN.
0.8
100
Count
10e
0.
Relative Attenzion Score
The first source of noise we address is the sink tokens. These visual tokens that attract substantial attention despite being semantically irrelevant to the grounded object. As identified in recent studies Daarett et al. 2023 Kang et al. (2025a), sink tokens can be identified by the high 122norm of their corresponding feature vectors H,, To mitigate
Figure 3: Attention magnitude VS. Localization accuracy.
them, first denoise the initial Rol map MR suppressing sink tokens identified via predefined norm threshold Tnorm yielding a cleaner version, M
[ (MKa))) ((MRa)) otherwise
if (H Tnorm
Label Assignment. Following the removal of sink tokens, the denoised Rol map M more clearly highlights the foreground area. However, it not yet an ideal supervisory signal, it still suffers from an obscure foreground-backgroun
margin and incomplete object activation. To empir- ically investigate the margin issue, we analyze the denoised attention maps M on the TextVQA subset (Zhang et al.] 2025a). Fig. plots the proportion of tokens falling inside the ground-truth (GT) bounding box function their sample-wise relative attention score, calculated as a /mmaxx for each attention value a E M The histogram reveals that the proportion of tokens inside the GT box is approximately 40% for high attention scores (e.g.. > 0.2) and 10% for low scores (e.g., < 0.1). However, the ambiguous middle-range of attention not only exhibit distinct localization pattern from the more decisive high and low scores, but they also vastly outnumber the high-attention tokens. Using such signal for dense supervision would inevitably force the model to learn from these numerous ambiguous regions. To address this, we avoid regressing on M directly and instead implement selective binary classification that assigns foreground background only to

---

Published as a conference paper at ICLR 2026
KB+R
M
Vision Encoder
Layer
Jayer
Layer B+1
Jakee Fi-2 3it
Q Rol
Share Weights
Weights
Question: Are there red
Init.
K
M
umbrellas or motorcycles?
MILM: Yes.
What is common to the dresser and the table? MLLM: Color.
Question:
H
Layer
Layer B+1
Layer B+R
H
Pseudo-L Generation
H
Question: What the name of furninure color the table?
u000
H r(n)
Matrix Multiplication
MIL.M: Dresser.
Figure 4: Overview of our SD-RPN framework. Our RPN (top) is initialized from and built upon a frozen MI.LM backbone to efficiently predict a dense Rol map. It is trained via self-distillation (bottom), where pseudo-labels are generated by denoising the full MLLM's internal response-to- image attention maps. Superscripts denote layer indices; subscripts denote token sources. We omit the system prompt tokens for brevity.
high-confidence tokens, leaving ambiguous tokens to be ignored. This approach is also designed to alleviate the incomplete activation problem. We define a minimal bounding box, B5 which encloses identified foreground tokens. Tokens inside this box that is not classified foreground is explicitly ignored, which prevents the RPN from receiving erroneous background signals from inactivated parts of the true object. Guided by these criteria, the final pseudo-label map, MR is constructed by:
if token j E &Sgg if token j E Sbg. otherwise (ignored).
(MRol))
(2)
where the foreground S/g
0
Tfg 0max) and the background set is Soy {
j & Big and aj Tbg Gmax): }. In Appendix we also provide theoretical view of why learning to predict Rol labels is superior than using the raw attention maps directly.
3.3 Rol PREDICTION VIA SELF-DISTILLATION
As validated by recent studies (Kang et al. 2025b Shao et al. 2025), the middle layers of MLLMs exhibit significant potential for Rol prediction. Given these observations, can we leverage the local- ization capabilities inherent in the middle layers of MILMs build more efficient Rol predictor in both inference and data? To answer this question, we propose a tunable Self- Distilled Region Proposal Network (SD-RPN) that consists of R transformer blocks built upon the first B frozen MILMM layers, which serve as the backbone.
Predicting the Rol Map. The architecture of our SD-RPN is illustrated Fig. We initialize the RPN's weights using those from layers B to B + R of the pretrained MLLM, strategy that enables the efficient transfer of learned representations (Shao al.) 2025). Instead of predicting sparse bounding boxes, our well-initialized RPN trained predict dense Rol map, MR The prediction process is triggered by specific query tokens derived from the conversational context. Concretely, for a given image and a conversation with n turns, we first collect the sequence of hidden states, H, from the second last layer of the RPN:
Hyys H., H H. H..)): H,yy
(3)
where the subscript and r denote user and response tokens. From this sequence, we specifically isolate the hidden state corresponding the final token of each user question, H these
5

---

Published as a conference paper at ICLR 2026
tokens immediately precede the model's answers and serve the most direct prompt for generating a grounded response. These n query vectors are collected into a single tensor, HRol. which serves as the input for our Rol prediction head:
Hool concat H (((n)-11))
(m)l-1)))
where HRoi E R These query vectors and token hidden states Hg, are then projected into the query and key space using the linear layers (LPa and LPx) from the RPN's final attention block:
QRal = (Norm (Noo)(HRo))) K = (Norm(HH)
(5)
where Norm denotes the layer normalization (Zhang & Sennrich 2019 Ba et al. 2016). The pre- dicted dense Rol map, MRot is then computed via a simple matrix multiplication of these query and key matrices: MRØt = QR..K For brevity, the head dimension is omitted here; in practice, at- tention scores are computed per head and then averaged. Notably, this design is more efficient than exsiting frameworks (Shi et al. 2025a Zhang et al. 2025a) as it only need single forward through partial LLM layers.
Training via Seff-Distillation. The RPN is trained entirely through self-distillation to predict the pseudo-label map MR which generated via the pipeline in Section 3.2 by minimizing a bi- nary cross-entropy (BCE) loss, defined as CBCE Gace(MooI+ MRa)). While the text responses used for pseudo-label generation could theoretically originate from stronger teacher model (e.g.. GPT-4) or human annotations, our empirical results reveal a crucial insight: using the MI.LM's self-predicted responses yields superior performance (detailed in Tab, 4b). We attribute this connter-intuitive re- sult to the principle of representational consistency. Although an external teacher's ' response may be more accurate, the attention maps it induces can be -uist--distributinn for the student RPN. In con- trast, the attention maps from self-generated responses, even if imperfect, are inherently aligned with its own internal visual grounding mechanisms. This creates a more consistent and attainable distilla- tion target, especially in our data-efficient setting. This self-sufficient framework thus removes any dependency on external models or annotated data.
3.4 TWO-STAGE INFERENCE WITH Rol
The predicted Rol map, MRal, enables a dynamic, two-stage inference process that significantly enhances the model's fine- aineggrined perception capabilities. The first stage involves predicting and post-processing MRØ to produce a clean binary foreground mask, B. To consolidate activated re- gions and ensure robustness against noise, the dense map MR is reshaped into a 2D map (7), smoothed with a Gaussian filter (9), and then binarized using a fixed threshold (T):
1, if ((
B(x, 1)
T,
(6)
0, otherwise,
where (z, y) represents the spatial coordinates. With this binary mask, we proceed to the second stage to extract fine-gen mne-grained visual features. We explore two different upscaling strategies for the predicted Rol. The first, which we term Box Upscaling, processes each salient region independently. We first identify all distinct connected-component regions, (R H= within the mask B. A minimal, axis-aligned bounding box, b,, is then computed for each region. These bounding boxes are used to crop sub-images, {2,, J. from the original source image, which are then encoded to produce new, high-resolution visual embeddings H P
b; bbox bbo))
H (P(E.
where bbox(.) the operator that returns the coordinates of the minimal bounding box enclosing the input region. Alternatively, our second strategy, which we term Masked Upscaling, takes unified approach. This method first computes a single, all-encompassing bounding box, ball. that encloses the union of all connected foreground regions, U R,, This unified bounding box is used to crop single sub-image, Teg which is then passed through the vision encoder and projector to produce a set of high-resolution embeddings HD E
bau = bbox((x RR), i=1
H Pank
= PPB o E, (x
(8)

---

Published as a conference paper at ICLR 2026
Table 1: Performance on Document & OCR benchmarks. Dataset subseripts denote the evaluation split. Performance subscripts show the absolute improvement (1) over the baseline. Throughput is relative to the baseline, measured on a single NVIDIA A6000 GPU.
Methods
Throughput DocVQA..
ChartQA
OCRBenchast InfoVQA... TextVQA.g.
Ave.
LLaVA-1.--B
1,0x
21.5
18.1
31.4
20.4
46.1
27.5
+S²
0.70x
27.1
18.9
32.6
22.5
52.6
30.7
+ViCrop
0.42x
27.0
20.0
33.2
21.4
57.2
31.8
+SD-RPN(OUS)
0.62x
34.2
20.6
37.3
22.3-19
58.8
34.6,21
LLaVA-1.5-13B
1.0x
23.5
18.1
33.7
23.4
48.7
29.5
+S²
0.71
30.7
20.3
36.4
24.7
54.5
33.3
+ViCrop
0.39x
30.2
20.1
36.1
25.9
60.3
34.5
+SD-RPN(OUSS)
0.51x
39.4
21.2
39.6,5.9
24.8-1.4
63.4
37.7
DeepSeck-VL-1.388
1.0x
37.0
48.6
38.9
21.7
55.8
40.4
+SD-RPN(Ours)
0.47x
54.4
51.7
40.8
24.1
65.9
47.4
DeepSeck-VL-7B
1.0x
50.7
61.4
42.4
30.4
63.0
49.6
+SD-RPN(OurS)
0.40x
67.2
62.2
47.9
36.8
71.5
17.5
Qwen2.5-VL-3B +SD-RPN(OurS)
1.x
87.1
82.6
73.8
62.7
76.9
76.6
0.56x
89.5
83.8
76.3
67.0
79.7
79.3
Qwen2.5-VL-77 +SD (P-RPS(
1.0x
92.0
82.6
81.5
70.2
81.1
81.5
0.57x
93.6,16
85,5
82.9
76.9
83.5
84.5
Table 2: Performance on Vision-Centric and High-Resolution benchmarks. ported the original publication.
denotes results
V Bench
POPE
HR-Bench 4K
HR- HR-Bench 8K
Methods
Attr
Spatial
Overall
F1
Acc.
FSP
FCP
Overall
FSP
FCP
Overall
LLaVA-L.5-BB
48.7
52.6
50.3
85,9
87.4
39.5
35.5
37.5
32.5
33.8
33.8
+5²
53.0
59.1
55.5
87.1
87.9
49.8
38.3
44.0
40.5
36.5
38.5
+ViCrop
53.9
50.0
52.4
88.0
88.6
60.8
34.8
47.8
38.5
33,8
36.1
-SD-RPN(OUES)
70.4
71.1
70.7
87.2
87.9
59.0
35.5
47.3
48.8
34.5
41.6
LLAVA-15-11BB
47.0
56.6
50.8
85.9
87.1
41.5
44.5
43.0
36.6
38.5
36.6
+S²
43.5
59.2
49.7
87.3
88.1
51.8
46.5
49.1
41.3
44.5
42.9
+ViCrop
47.8
57.9
$1.8
88.0
88.7
66.3
40.3
53.3
44.5
37.0
40.6
+SD-RPN(Ours)
60.9
65.8
62.8 12.0
87.9
88.6
58.8
47.0
52.9
51.8
39.8
45.8
MeepSeck-VL-L.BB
34.8
55.3
42.9
86.1
87.1
40.5
33.0
36.8
30.1
29.3
30.1
+SD-RPN(Ours)
55.7
59.2
57.1
87.5
88.3
56.8
34.0
45.4
44.8
32.5
38.66as
DeepSeck-Vi VL-7B
37.4
50.0
42.4
86.0
87.1
47.5
41.5
44.5
37.3
41.3
39.3
+SD-RPN(Ours)
52.2
55.3
53.4
88.2
89.0
58.8
39.8
49.3
52.3
46.1
Qwen2.5-VL-3B
81.7
60.5
73.3
87.8
88.6
80.5
52.0
66.3
70.0
47.8
58.9
+SD-RPN((OUs)
65.8
81.2
88.9
89.3
88.0
58.3
73.1
79.5
65.3
Qwen2.5.
80.0
75.0
78.0
86.7
87.9
81.8
62.8
72.5
72.8
63.6
+Thyme"
83,5
80.3
82.2
91.0
63.0
77.0
86.5
57.5
72.0
+DeepEyes"
91.3
88.2
90.1
87.7
91.3
59.0
75.1
86.6
58.5
72.6
+SD-RPN(Ours)
96.5
79.0
89.5
88.4.
88.9
1.0
92.5
64.5
78.5
16.0
86.5
60.5
73.5
where represents masking operation that uses B select the foreground features from the en- coder's 's output. Box Upscaling can achieve a higher effective resolution for small, individual regions and Masked Upscaling better preserves the global spatial and positional relationships between all foreground elements, which is crucial for structured data. In the second stage, the high- high-resolution visual tokens are inserted into the sequence immediately following the original visual tokens. The LLM then performs auto-regressive decoding this augmented context to generate the final answer. We present a empirical comparison of these two upscaling strategies our ablation study.
EXPERIMENTS
4.1 EXPERIMENT CONFGURATIONN..
Benchmark Settings. To ensure comprehensive evaluation, test SD-RPN framework
across a range benchmarks, following the protocols established works (Shi et al. 2025a Zhang
et al. 2025a Zheng al. 2025). Our results are presented in parts clearly delineate perfor-

---

Published as a conference paper at ICLR 2026
mance across different domains. The first part focuses the Document & OCR category, which assesses performance on text-rich images, and includes five benchmarks: Doc (Mathew et al. 2020), ChartQA (Masry et al.] (022). OCRBench (Liu et al.] 2024b), Info (Mathew et al. 2022), and TextVQA (Singh et al. 2019). 'he second part focuses on Vision-Centric and High- Resolution tasks, including V-Star Bench (V*) (Wu & Xie (023), POPE (Li al.] (023), and HR-Bench (Wang et al.) 2024b). Notably, for TexiVQA, we do not provide the model with external OCR annotations, a strategy adopted from ViCrop (Zhang et al. 2025a) to ensure a fair evaluation of the MLLM's s intrinsic perceptual capabilities.
Baselines and Comparison Methods. To evaluate SD-RPN, we integrate it into three prominent MLLM families. Our primary experiments are conducted on the widely-used LLaVA-1.5 (Liu et al., 2023a) architecture, across both 7B and 13B scales. To demonstrate the generalizability of our approach, we also apply it to the more recent DeepSeek-Vi VI. (Iuu et al. 2024) and Qwen2.5-VI. (Bai al. 2025) models. We compare our method against several representative techniques for high- resolution visual processing. For the LLaVA-1.5 models, we benchmark against S (Shi et al.
2024). a full-tuning method, and ViCrop (Zhang et al, 2025a), a training-free cropping baseline. For the Qwen2.5-VL model, we additionally compare against DeepEyes (Zheng et al. 2025 and
Thyme (Zhang et al. (0255)), which are recent reinforcement learning-basd baaning--beddddd method. To ensure a rigorous and fair comparison for full reproducibility, have re-implemented all competing meth- ods within the unified Imms-eval evaluation library (Zhang et al.) 2024a), unless noted otherwise.
Implementation Details. To train our proposed SD-RPN, we generate pseudo-labels from com- bined dataset of GQA (Hudson & Manning 2019) and OCR-VQA (Mishra et al. 2019), which provides supervision for natural scenes and text-rich images, respectively. These pseudo-labels derived from the models' internal sisponse-to-image attention maps. Inspired by recent analyses of MLLM attention mechanisms (Kang al. 2025b), we select attention maps from the middle layers for this process. During the laeudo-label generation pipeline, the relative attention thresholds for defining the foreground (S) and background (Sng) sets are empirically set to 0.2 and 0. respec- tively. For experiments involving Qwen2.5-VI, which natively supports dynamic resolutions, we standardize the input for consistency. On most benchmarks, we set the number of visual tokens to 576 to align with the LLaVA and DeepSeek-V VL baselines. For high-resolution benchmarks, we increase this to a maximum of 4096 visual tokens to effectively process the detailed inputs. Further details on the layer selection rationale and
can be found in Appendix
4.2
MAIN RESULTS
We present our main results in Tab. [ text-rich benchmarks and Tab. Tab.2ra for vision-centric bench- marks. Our evaluation assesses both performance and efficiency.
Integrating our SD-RPN framework yields substantial and consistent performance gains across all evaluated MLLM families. On the text-rich benchmarks (Tab. [T], our method boosts the average score by approximately 7% over the LLaVA-1.5 and DeepSeek- VL baselines and 3% over the more powerful Qwen2.5-VL baseline. The benefits are also evident on vision-centriand high-resolution tasks (Tab. 2). where our approach achieves average improvements of over 10% on V* Bench and over 6% on HR-Bench. Notably, on high-resolution benchmarks, SD-RPN achieves performance comparable or superior DeepEyes which relies visual Chain-of-Thought (CoT) that incurs substantially greater computational cost. To ensure a fair comparison on V-Star Bench, we note that the image cropping strategy employed by ViCrop was disabled during our evaluation. Beyond these quantitative results, Fig. [ offers a qualitative comparison with the LLaVA-1.5 and Qwen2.5-VL baselines. Additional visual examples are provided in Appendix
Performance.
Efficiency. The two-stage inference process used by our method and ViCrop inherently involves a trade-off, reducing throughput exchange for higher accuracy. This additional latency stems from three primary sources: the initial Rol prediction stage, the feature extraction for the high-resolution Rol crops, and the increased number of visual tokens processed by the LLM in the final generation stage. However, our SD-RPN is architected for superior efficiency in the critical Rol prediction step. By leveraging only a subset the MLLM's layers in a single forward pass, our method is significantly faster than competing two-stage approaches. For example, when integrated into the LLaVA-LL5-7B model, our Rol prediction stage is 1.5x faster than that ViCrop, demonstrating a more favorable balance between performance and computational cost.
8

---

Published as a conference paper at ICLR 2026
Input Image
Rol Heatmap
Mixed Image
Input Image
Rol Heatmap
Mixed Image
What is written the top of the yellow fridge?
No ping
X
What of motareyele helmer?
+SD-RPN: Warning
A. White B. Blue C. Red Black
Pirtsburgh
X
What the color the brom?
X
What city written the
+SD-RPP: Philadelphia
v
A. Black B. Yellow Gray Red
SSDRRP.
Figure 5: Qualitative comparisons Tex and V-Star benchmarks. These examples show challenging cases where the baseline model fails while our SD-RPN succeeds. For each question, model generates dense Rol Heatmap (center) to localize the relevant area. The Mixed Image (right) shows the final high-confidence region identified by SD-RPN, which is used for fine-grained perception to find the correct answer.
4.3
ABLATION STUDY
Table 3: Ablation study on the key components of our method.
(4) Setting
Throughput
OCRBench
TextVQa
POPE
V*
Ave.
(0)
LLaVA-1.5-7B (Baseline)
1.0x
31.4
46.1
85.9
50.3
53.4
(1) Response-to-Imanl Image Attention
0.42x
32.8
53.0p69
84.3
57.6
56.9
(2) Attention Prediction
0.58x
32.2
56.8
85.7
61.3
59.0+s3
+Label Assignment
0.55x
33.9
57.3
85.6
68. 68.6,18.5
61.4m79
+Remove Sink Tokens
0.55x
36.2
57.9
86.8
68.6,18.
62.4r9.
(5) +Masked Upscaling
0.62x
37.0
58.7
87.1
67.5
62.6r92
In this section, we first present ablations for
design choices, including the pseudo-label generation process, backbone layer depth, and data efficiency. These experiments were conducted using default thresholds (Tia 0.2, Thg 0.1). We then provide a detailed ablation study on these
Tfg and Thg thresholds determine their optimal values
On Pseudo-Labe Gerneration & SD-RPN. Tab, 3ahh.33presee presents a detailed ablation study dissecting the core components of proposed framework. We begin by evaluating baseline strategies for leveraging attention. Box upscaling is adopted as the upscaling setting except additional note. In Response-to-Imant
(#1), use the RPN is trained to regress these attention scores via a Mean Squared Error (MSE) loss. While both methods improve upon the baseline, their average gains of 3.8% and 5.3% are limited, confirming our hypothesis that using noisy, unprocessed attention maps for direct supervision is suboptimal. The introduction Label Assignment strategy (#3), which creates high-confidence labels and ignores ambiguous regions, yields a significant performance jump to 7.9% average improvement. This is further enhanced by Remove Sink Tokens step (#4), which denoises the attention map to achieve 9.0% average gain. These results validate that proposed denoising pipeline is crucial for generating high-quality supervision and enabling precise Rol prediction. Aligning with previous analysis in Sec. masked upscaling achieves superior performance OCRBench and TextVQA. Given its better throughput (0.62x vs. 0.55x), we adopt it as the default setting.
Attention map directly for Rol identification. In (#2), our
On Backbone Layers. In Tab. 4a we conduct ablation study the number of frozen back- bone layers (B) that the RPN is built upon, keeping the number of trainable RPN layers fixed. We observe clear trend performance: as the backbone depth increases, the average performance steadily improves, reaching its peak with the B15R3 configuration, which achieves 9.2% gain over We thank the reviewers for motivating this deeper analysis of hyperparameter settings.

---

Published as a conference paper at ICLR 2026
Table 4: (a) Varying the number of frozen backbone layers (B) while keeping the RPN layers fixed. (b) Varying the number of training data samples. The i symbol indicates that pseudo-labels were generated using ground-trutt trunh responses from the LLaVA supervised fine-tuning dataset. All scores accuracy (%) except for POPE, which reports the Fl-score.
Setting
Through- put
OCR- Bench
Text- VQA
POPE
Vo
Ave.
Setting
OCR- Bench
Text- VQA
POPE
V*
Ave.
Baseline
1.xx
31.4
46.1
85.9
50.3
53.4
Baseline
31.4
46.1
85.9
53.4
B3R3
0.71x
51.0
87.0
53.9
16.5
10K
35.2
58.3
87.6
61.3
60.6
B6R3
34,4,30
53.5
87.7
17.
58.2
25K
35.8
58.54124
87.1
59.7-94
60.3r69
B9R3
0.66x
35.8744
56.34102
87.1
59.7
59.7-63
50K
37,5
58.31m0
86.9
67.0p16
62.4
B12R3
0.60x
34.5
57,8
87,4ms
67.0r16.7
61.9ra5
100K
36.5151
58.34122
86,8
66.0+157
61,9ra5
B15R3
x.62
35.0
58.7
87.1
67.5
62.6-42
152K
37.0p56
58.77126
87.
67.5
62.6r92
B18R3
0.52x
p5
57.8
87.3
66.0p157
61.7
152K!
35.2
56.9
1108
86.6
63.9r136
60.7
(a)
(b)
the baseline. Beyond this point, performance begins decline. The relationship between back- bone depth and efficiency is more nuanced. While the inference cost of the Rol prediction stage is positively correlated with the number of backbone layers, the overall throughput is not monotonic. This is because more precise Rol prediction, often produced by deeper backbone, can reduce the number of irrelevant or noisy visual tokens that are passed to the second, more costly generation stage. This interplay between the increasing cost of the first stage and the potentially decreasing cost of the second stage explains the fluctuating throughput values.
On Data Efficency. We conduct ablation studies the size of the training set, with results presented in Tab. 4bb Performance generally improves with an increased num- ber of training samples. Remarkably, even when trained on only 10K samples, the framework achieves substan- tial gains over the baseline (e.g., over 10% on improve- ment on both Tex and V-Star). Further increases in data size continue to yield performance gains. The peak
Table 5: Ablation study on pseudo-label thresholds T/g and Tbg.
OCR- Bench
Text- VQA
Tyo
The
POPE
V
Ave.
Baseline
31.4
46.1
85.9
50.3
53.4
0.10
0.10
35.1
57.4
87.4
65.4
61.3
0.15
0.10
36.3
87.2
62.2
performance is ultimately achieved when training on the full combined dataset of 152K samples. The final row of Tab. 44b] presents a crucial comparison where we gener- ate pseudo-labels using ground-truth responses from the
0.20
0.10
37.0
58.7
87.1
67.5
62.6
0.25
0.10
37.0
86.7
68.1
0.30 0.10
36.6
58.1
86.9
64.9
61.6
0.20 0.075
36.7
58.4
86.9
70.7
63.2
0.20 0.05
37.3
58.8
87.2
70.7
63.5
0.20 0.03
37.8
58.7
87.6
68.6
63.2
LLaVA supervised fine-tuning (SFT) dataset. Counter- intuitively, this approach yields notably lower average performance of 60.7% compared to the 62.6% achieved by our standard self-distillation method. This suggests that the attention maps pro- duced by the model's supervision signal for distilling localization knowledge.
generated responses provide more effective and internally consistent
On Foreground and Background Threshold. Tab. J.objaaab the pseudo-label thresholds. We find that setting Tfg = Thy = 0.10 (a simple binary classification) performs poorly (61.3% avg.). as suffers from ambiguous tokens. By optimizing the thresholds Tfg 0.20 and Thg 0.05, establish new, stronger default. This setting achieves 63.5% average performance, surpassing our original default (62.6%) and the baseline (53,4%) by 0.9% and 10.1% ,, respectively.
5 CONCLUSION
We proposed SD-RPN, a self-distileed region proposal framework that efficiently exploits the in- trinsic localization signals of MILLMs to identify Regions of Interest without external annotations or auto-regressive decoding. By attaching a RPN to frozen backbones and training it with denoised pseudo-labels, our method achieves consistent improvements fine-grained perception while main- laining strong efficiency and generalization. Extensive experiments confirm its advantage over both full-image scaling and training free heuristics, and ablations highlight its robustness and data ef- ficiency. Our work establishes a principled direction for scalable high-resolution perception in MLLMs, and opens avenues toward adaptive token allocation and broader multimodal applications such video and document understanding.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENTS
This work was supported in part by the Start-up Grant (No. 9610680) of the City University of Hong Kong, Young Scientist Fund (No. 62406265) of NSFC, and the Australian Research Council under Projects DP240101 1484 and FT230100549.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06600 2016.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. local- ization, text reading, and beyond. arXiv preprint arXiv:2308./220 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923,, 2025.
Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Zhucun Xue, Yong Liu, and Xiang Bai. Llava-kd: A framework of distilling multimodal large language models. arXiv preprint wfXv:2410.162366 2024.
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal 1lm. In CVPR, 2024.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. NeurIPS, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng. Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked self-distillation advances con- trastive language-image pretraining. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 10995-11005, 2023.
Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, and Bo Zheng. Convllava: Hierarchical backbones as visual encoder for large multimodal models. arXiv preprini 245. 2024.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Xiaojun Hou, Jiazheng Xing, Yijie Qian, Yaowei Guo, Shuo Xin, Junhao Chen, Kai Tang, Meng- meng Wang. Zhengkai Jiang, Liang Liu, et al. Sdstrack: learning for modal visual object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26551-26561, 2024.
symmetric adapter

---

Published as a conference paper at ICLR 2026
Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alle- viating the semantic sawtooth effect for lightweight milms via complementary image pyramid. arXiv preprint arXiv:2408.02000 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In ICLR, 2025a.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. Your large vision-language model only needs a few attention heads for visual grounding. In CVPR, 2025b.
Sanghwan Kim, Rui Xiao, Mariana-Iuliana Georgescu, Stephan Alaniz, and Zeynep Akata. Cosmos: Cross Cross-modality self-distillation for vision language pre-training. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 16690014700, 2025.
Jun Kong, Jin Wang, Liang-Chin Chin Yu, and Xuejie Zhang. Multimodality self-distillation for fast inference of vision and language pretrained models. IEEE Transactions Multimedia, 26:8928- 8940, 2024.
Wentong Li, Yuqian Yuan, Jian Liu, Donggi Tang. Song Wang. Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal Ilm. International Journal of Computer Vision, 2025.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llavanext: Improved reasoning, ocr, and world knowledge, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- lasion-language understanding. arXiv preprint 2rXiv:4403.05525 2024.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: offtuure-o--resluttio adaptation for multimodal large language models. arXiv preprint 1:Xiv:2403.0300: 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. Doevqa: A dataset for vqa on document images. corr abs/2007.00390 (2020). arXiv preprint arXiv:2007.003900 2020.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievqa. In WACV, 2022.
12

---

Published as a conference paper at ICLR 2026
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR). IEEE, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Yi Peng, Peiyu Wang, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, et al. Skywork rlv: Pioneering multimodal reasoning with chain-of- thought. arXiv preprint arXiv:2504. 055599 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprchensive dataset and benchmark for thain-of-thougeee reasoning. NeurIPS, 2024.
Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang. Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing twig accelerate large vision-language models. In ICCV, 2025.
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do not need larger vision models? In ECCV, 2024.
Baifeng Shi, Boyi Li, Han Cai, Yao Lu, Sifei Liu, Marco Pavone, Jan Kautz, Song Han, Trevor Damell, Pavlo Molchanov, et al. Scaling vision pre-training to 4k resolution. In CVPR, 2025a.
Yuheng Shi, Minjing Dong, and Chang Xu. Harnessing vision foundation models for high- performance, training-free open vocabulary segmentation. In ICCV, 2025b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: nigh-perffrmance instance segmentation with box annotations. In CVPR, 2021.
Peter Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. NeurIPS, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.00 092888, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- model's perception the world at any resolution. arxiv preprint arXiv:2409.12199 2024a.
Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: training-free free framework for high-resolution image perception in multimodal large language models. arXiv preprint, 2024b.
Xinlong Wang. Zhiding Yu. Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning segment objects without annotations. In CVPR, 2022.
13

---

Published as a conference paper at ICLR 2026
Penghao Wu and Saining Xie. V*: Guided visual search mechanism in multimodal Ilms. arXiv preprint arXiv:2312.14135 2023.
Runpeng Yu, Xinyin Ma, and Xinchao Wang, Introducing visual perception token into multimodal large language model. In ICCV, 2025.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. NewrIPS, 2019.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to look: Training-free free perception of small visual details with multimodal Ilms. In ICLR, 2025a.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a. BRL.https:////
org/ abs/2407.
12772
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3713-3722, 2019.
Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang. Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508. 11630, 2025b.
Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, et al. Llava-uhd v2: an milm integrating high-resolution feature pyramid via hierarchical window transformer. arXiv preprint arXiv:2412.13877 2024b.
Xiangyu Zhao, Xiangtai Li, Haodong Duan, Haian Huang, Yining Li, Kai Chen, and Hua Yang. Mg-llava: Towards multi-granularity visual instruction tuning, arXiv preprint arXiv:2406. 17770, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023.
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505. 14362, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Yc, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfci Cui, Xuchui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang. Haomin Wang. Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang. Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanced training and test-time recipes for apen-source multimodal models, 2025.
14

---

Published as a conference paper at ICLR 2026
A MORE IMPLEMENTATION DETAILS.
Training Setting. The hyperparameters used to train our RPN were adapted from TLM (Shao al. 2025) and are detailed in Table 6 All models were trained server using four NVIDIA A6000 GPUs Our training dataset consists of 152K samples, with 72K sourced from the GQA dataset (primarily natural images) and 80K from the OCR-VQA dataset (text-rich images). As point of reference for computational cost, training the RPN for the ALaVA-1.5-7B model on our full 152K -samle dataset completes in around two hours.
Table 6: Training hyperparameters for the RPN.
Config
Setting
optimizer
AdamW
weight decay
0.
optimizer momentum batch size
$1. 82 = 0.9, 0.98 128
learning rate schedule
cosine decay
peak learning rate
5e-5
warm-up strategy
linear
warm-up ratio training epochs
0.03
Table 7: Model-specific layer configurations. "Selected Attention Layers" refers to the layers from which attention maps were extracted for pseudo-label generation. "Frozen Backbone Layers (B)"
refers to the number of initial layers kept frozen, upon which the trainable RPN built.
Model
Selected Attention Layers
Frozen Backbone Layers (B)
LLaVA-1.5-7B
14
15
LLaVA-1.5-13B
13-16
15
DeepSeek- CoopSeck--L-1.33 VL-1.3B
7-14
9
DeepSeck- PoepSeck-VV-BBB VL-7B
9-17
12
Qwen2.5-VI.cd VL-3B
20-22
21
VL--7B
16-23
16
Layer Selection Rationale. The specific layers used for pseudo-label generation and for defining the RPN backbone were empirically chosen for each model architecture, with the final configurations detailed in Table 7 For generating pseudo-labels,
selection of attention maps is inspired by recent analyses (Kang et al. 2025b) which demonstrate that the middle layers an MLLM contain the most potent visual grounding signals. These intermediate layers represent "sweet spot" where visual and textual representations are sufficiently fused for localization, yet before the model's final layers become overly specialized for abstract reasoning and text generation, potentially weakening the direct visual signal. When a range of layers specified, the attention maps were averaged to create a more stable and robust signal. The number of frozen backbone layers, B, determines the depth and richness of the features that the trainable RPN operates on. This choice represents a critical trade-off. A deeper backbone (larger B) provides more semantically rich features but increases the computational cost of the Rol prediction stage and can limit the learning capacity of small RPN. Conversely, a shallower backbone is faster but may not provide features that are sufficiently aligned for accurate Rol prediction. The values in Table 7 were determined through ablation studies (see Table 4a) find the optimal balance between performance and efficiency for each model.
Algorithm Details. To provide concrete and reproducible specification of our training procedure, we detail the complete end-to-end process in Algorithm The algorithm consists of a main training loop and the core helper routines. The main training loop iterates through the dataset, performing two key stages for each sample. First, the "teacher" stage (lines 3-10) uses the full, frozen MLLM to generate pseudo-label. This involves regressvvyy generating response, extracting the raw response-to- tomigge attention map from a pre-selected middle layer, and then refining this map using RemoveSinkTokenn and Assignl abels helper functions. This produces the final denoised
15

---

Published as a conference paper at ICLR 2026
Algorithm Self-Distilled RPN: training and helper routines
Require: Full MILM with L layers; frozen depth B3 RPN depth Rt; pseudo-label layer dataset
D; thresholds (Trarmy ff.. Tog]; optimizer for trainable RPN parameters
Ensure: Trained RPN on top of the frozen backbone
1: Initialization: Initialize RPN layers B+1:B+ R from L: freeze layers 1:B; initialize optimizer.
2: for each (I,, V4. E DD do
// Teacher: generate pseudo-labels from the full MLLM
3: Ypod + Decode(L, Zv. x1)
>> Auto-regressive response
4: aulll + Concat(x:: Xprsa)
5:
H u. Q K, + Features( 2,, full
6:
A + Softmax (K,)
Softmax over visual tokens
7:
MR Mean eads And A)
8:
MR - RemoveSinkT ( H Tnomn
9:
MROl + Labels Tfg. Tbg)
Values in (-1.0, 13
10:
mask < (MRol # 11)
// Student: forward through frozen 1:B and trainable B+1:B+R
Hy: HRol Features CRPN, Tus It, B+RR-1
12:
QRal + LPg L (RRo)), K. f LP&(N (Norm(H.)))
13:
MRØI t Q
Logits for dense Rol map
14: loss +- BCBWBCCWWi Logiiss [Ra[mmakk], [am[mask)
15:
Update(optimizer, loss)
Helper routines
1: function TEMOVESSIKKTOKENSSA H., Tnorm)
2: return M with entries 0 where Horjlle Tnorm
3: function ASSIGNLABELSCN Tfg: Tbg)
4: amax + max, M,
5:
S/9 - {j: M; 2 Tg9 amax)
6:
Big Min.Enclosinggarr
7:
Sbg <- [i & Big- Mj S Thg Omax
8:
Build M with M, for jES My for jESag. and -1 otherwise
9:
return M
Table 8: Ablation on the region proposal layers.
Setting
OCRBench
TextVQA
POPE
V*
Ave.
LLLVA-1.5-7B
31.4
46.1
85.9
50.3
53.4
B15RI
35.8
54.7
87.1
56.0
58.4,5.0
B15R2
35.9
57.8
87.2
63.4
61.1 17.7
B15R3
37.0
58.7 112.6
87.1 11.2
67.5 117.2
62.6
B15R4
37.1
58.4
87.1
67.5
62.5
pseudo-label, MR Second, the "student" stage (lines 11-15) trains the RPN. The RPN performs forward pass on the original input to predict a dense Rol map MRol- masked BCEWithLogits loss is then computed between the predicted map and the pseudo-label target, ensuring that ambiguous regions (where the mask false) do not contribute to the gradient. Finally, the RPN's parameters updated via backpropagation.
B
MORE ABLATIONS
These experiments blow were conducted using default thresholds (Tfg = 0.2, Thg = 0.1), as they were implemented before threshold ablation.
16

---

Published as a conference paper at ICLR 2026
Table 9: Ablation on the setting of Pre-smoothing and Post-smoothing.
Setting
OCRBench
TextVQA
POPE
V*
Ave.
LLaVA-1.5-7B
31.4
46.1
85.9
50.3
53.4
Pre-smoothing
34,8
58 -8.4
87.9
60.7,10.4
60.5
Post-smoothing
37.0 156
58.7 112.6
87.1 112
67.5 11.22
62.6
Qwen2.5-V1-38
Qwen2.8.VI.7B
Baseliie Hhbrrs
am
202
1024
Bascine
Hhrs
75-
STC
Porformance
70-
3172
65-
60-
30-
50-
Throughput (tokens/S)
Throughput (tokens/))
Figure 6: Performance- Throughhpt Trade-off on the V-Star Benchmark. Each point on the plot
corresponds to different maximum number of visual tokens. Our approach achieves a superior trade-off. The x-axis is on logarithmic scale for clarity.
On Region Proposal Layers. In Table g we present an ablation study on the depth of the trainable Region Proposal Network, varying the number of tunable layers (R). We observe that increasing the RPN's depth from one to three layers yields a significant performance improvement (from 58.4% to 62.6% average score). Beyond this point, we found that the performance gains diminished, failing to justify the additional computational cost incurred during both training and inference. Therefore, to achieve the best balance between model capacity and efficiency, we selected R=3 as the default setting for our framework.
On Pseudo-label Smoothing. Our standard pipeline applies a Gaussian filter to the RPN's final pre- dicted map, a step we term post-smoothing. We investigated an alternative approach, pre-smoothing, where the filter is instead applied directly to the pseudo-labels during their generation. As shown in Table 9 the post-smoothing approach consistently outperforms pre-smoothing across most bench- marks, achieving a +2.1% higher average score. We hypothesize that this performance gap stems from distributional shift induced by pre-smoothing, Applying filter to the sparse pseudo-labels makes them denser, effectively enlarging the target foreground area. While this might appear ben- eficial, it creates significant discrepancy between the dense, artificially softened labels and the sharper, more sparse predictions natural to the self-initilized RPN. This distribution gap compli- cates the training process, particularly in our data-efficient setting. This difficulty is reflected in the final convergence loss, where the pre-smoothing strategy results in a 2.5x higher value than post- smoothing (0.05 vs. 0.02). We conclude that foreing the network to learn these artificially blurred targets increases ambiguity, ultimately degrading the precision of the Rol predictions.
C MORE COMPARISON AND ANALYSIS
C.I ANALYSIS OF THE PERFORMANCE- THROUGHPUTT TRADE-OFF
More recent MLLMs. such as Qwen2.5- -VL, adopt dynamic resolution techniques that handle images of various sizes by producing varying number of visual tokens. A key concern is whether our Rol- selection approach is more effective than simply increasing the native input resolution of the baseline model. To address this, we analyze the trade-off between performance and efficiency by scaling the input resolution for both the Qwen2.5-VL baseline and our proposed SD-RPN. The results of this analysis on the V-Star benchmark are presented in Fig. 6 The figure plots performance against throughput (tokens/s), where the ideal model would be in the top-right corner. It is evident that the trade-off curve for method is consistently superior to that of the baseline. For example, on Qwen2.5-VL. V-L,,B our method achieves a performance of 77.5 at 5.1 tokens/s using a 576-resolution input, level the baseline only reaches at 4096 resolution with much lower throughput of 1.0
17

---

Published as a conference paper at ICLR 2026
tokens/s. This demonstrates that SD-RPN provides significantly better balance between accuracy and computational cost, confirming the effectiveness of our approach over naive resolution scaling.
C.2 COMPARISON TO EXTERNAL GROUNDING MODELS
A practical concern regarding our SD-RPN is its computational cost, as it utilizes a portion of the MLLM. may seem counter-intuitive, but our SD-RPN is, in fact, significantly more computation- ally efficient than using an external grounding model like Grounding-DINe reason is that our RPN re-uses computations that the MLLM must perform anyway. An external model (e.g., Grounding-DINe
or YOLO-World. The
would require redundant computation in the form of a separate, full image encoding pass, which is computationally expensive. contrast, our approach uses feature re-use: our RPN attaches to the MLLM's middle layers and operates on features that are already computed during the MILM's initial forward pass. The only extra cost is passing these features through the RPN layers (e.g., 3 layers in our B15R3 configuration), which is a minimal addition.
To provide concrete analysis, we benchmarked our method (on LLaVA-1.5-7B) against Grounding-DINN DINO-Base model on a NVIDIA A6000 GPU. The results, presented in Table [0, are clear. Our RPN is over 20x faster in its specific prediction step (10.3ms vs 231.2ms) because it avoids a redundant, full-image encoding. This also leads to a much faster overall evaluation time on the V benchmark. Our method is also significantly more accurate (62.6% 57.6% avg.). This is because our RPN is an integrated part of the MLLM and has access to its contextual understanding of the query. External models are excellent at finding literal objects (e.g.. "the bottom man") but fail at abstract, reasoning-base based queries that don't name a specific object, such as the example in the last row of Figure 7. Our RPN, trained on the MLLM's internal attention, excels this.
-DNNO. Table 10: Efficiency and Performance: Ours (SD-RPN) vs. External Grounding- DINO. The "Grounding-DINoX. (IBox)" method crops one sub-image using the single most confident bound- ing box, while "(2Box)" crops two sub-images using the top boxes.
Method
RPN Time (ms)
V Eval Time (s)
OCRBench
TextVQA
POPE
V
Ave.
Ours (SD-RPN)
10.3
73
37.0
58.7
87.1
67.5
62.6
Grounding-DINC DINO-B (2Box)
231.2
248
32.5
54.5
82.4
62.3
$7.6
Grounding- DINO-B (1Box)
231.2
113
32.5
52.6
86.9
56.5
57.1
C.3 DEEPER ANALYSIS ON SOURCES OF ATTENTION NOISE
In our work, we identify and actively mitigate two primary sources of noise from the raw attention maps used to generate pseudo-labels. First, there are Type I: Sink Tokens. We build on recent studies (Darcet al. 2023 Kang et al.) (025a) which identify "sink tokens"-
tokens that attract high attention despite being semantically irrelevant to the grounded object. Second, there are Type 2: Ambiguous Foreground/Backgroung Signals. This is more common, semantic, and lask-dependent form of noise. As illustrated in Fig. and described in Section 3.2, these are "noisy maps" that exhibit "erroneously high attention in background areas and incomplete activation across the true foreground object."
As the first type of sink token belongs to a well-defined source of noise, we now investigate the second type. Our empirical study finds that questions or images in the following three categories are prone producing noisy attention maps, as illustrated in Fig. The first source of noise is cluttered scenes or dense text (c.g., the first sample in Fig. 7 in these cases, the model's 's attention can be fraught with noise as it diffuses across multiple irrelevant- -uteaaan--ut-lluuibbee regions. The second type of noise is caused by abstract or reasoning-heavy queries. In practice, we find that a simple query (e.g.. "find the cat"") tends to produce clean attention. However, many VQA queries are abstract, such as the second sample in Fig. 7 For such queries, the attention map reflects the model's complex reasoning process, which can appear noisy as it attends to multiple pieces of related (but not all relevant) information before settling on answer. The last type of noise is related to incomplete activation, where the attention map only weakly or partially covers the true region of interest, as shown in the last sample of Fig.
18

---

Published as a conference paper at ICLR 2026
Input Image
M
Input Image
M Rof
Input Image
Rof
Hew many Items can you get for $5
Who at the center all this?
Who was the photographer?
org
Philippe moliter
Figure 7: Examples of noisy and sparse attention maps for abstract queries.
C.4 COMPARISON WITH TOKEN COMPRESSION METHODS
We provide detailed comparison of our SD-RPN with other SoTA methods designed for high- resolution visual understanding in this section. To facilitate fair and transparent comparison, Ta- ble [1 summarizes the performance of method against several competitive models key doc- ument and OCR benchmarks. The table also highlights critical architectural and data differences, such the base Large Language Model (LLM), the size of the training dataset, and the maximum input resolution, all of which are significant factors that influence final performance.
Table 11: Comparison with SoTA nigh-resolution methods on document and OCR benchmarks. Denotes results obtained using OCR tokens. "Denotes results reported the LLL V...VA-Tooenaackeee HD paper (Li et al.) 2025). The "+ "+152K" in the #Data column indicates the number of additional samples used for fine-tuning our SD-RPN on the corresponding bascline models.
Method
Res.
TextVQA..
OCRBench
DocVQA...
ChartQA...
LLaVA-1.5
Vicuna-7B
1.2M
336x336
58.2"
31.4
22.2
18.1
LLaVA-15 SD-RPN
Vicuna-78
+152K
336x:36
64.2"
37.0
Monkey et 2024b
Owen-7B
1.4M
896x896
67.6
66.5
65.1
LLaVA-NeXT
1.3M
672x672
Mini Gemini HD et 20245
Vicuna-7B
2.7M
536x1536
68.4"
456"
65.0"
Hi
Vicuna-7B
2.7M
DeepSeck-VL.
Vicuna-7B
1024x 1024
63.0
42.4
49.3
61.4
DeepSeck VL SD-RPN
Vicuna-7B
+152K
1024x1024
71.5
47.9
64,5
63.6
QuVn2.5-VL
Qwen2.5-78
672x672
81.1
81.5
92.6
Qwea2,5-VL 5-VL SD-RPN
Qwen2.5-7B
+152K
672x672
83.5
82.9
94.4
85.5
The results Table II lead to two primary conclusions. First, the overall performance our method, like others, strongly correlated with the capabilities of the underlying base MILMM, When applied to an earlier baseline such as LLa our method's absolute scores on benchmarks like DocVQA understandably lower than SoTA methods built on more advanced foundations, which often leverage richer training sets and higher input resolutions. This performance difference is pri- marily reflection of the base model's capacity rather than limitation of enhancement mod- ule. Second, our SD-RPN module demonstrates consistent efficacy across base models of varying strengths. When integrated with a more competitive baseline like DeepSeek VL-7B, our method achieves performance on par with or slightly exceeding other leading methods. To further validate this generalizability, we applied SD-RPN to the powerful Qwen2.5-VI. model. Even on this SoTA baseline, our method delivers significant and consistent gains across all benchmarks, utilizing an identical and minimal fine-tuning recipe.
This highlights a key advantage of our approach: SD-RPN provides data-efficient and scalable pathway for enhancing the ine-grained perceptual capabilities of MLLMs. Unlike methods that ne- cessitate extensive supervised fine-tuning a resource- -eoource-fffciientt solution for improving high-resolution understanding that can readily adapt to the continuous advancements in base MI..Ms..
massive datasets, deff-distillation technique offers
19

---

Published as a conference paper at ICLR 2026
D PROMPT USAGE
Below are the prompts used for pseudo-label generation with LLaVA-1.5 and DeepSeek- VL. Each template follows the respective model's standard input format.
<image> USER: (question) Answer the question using a single word or phrase. ASSISTANT:
DeepSeck-VI. VL
<image_ planeeeppachhoder> USER: (question) Answer the question using single word or phrase. Assistant:
E VISUALIZATION
To provide deeper, qualitative understanding of our model's advantages, Figure presents a se- ries of challenging visual question-answering examples. These samples, drawn from the V-Star, TextVQA, and DocVQA benchmarks, were specifically chosen because they require recognizing and reasoning about fine-grained details within cluttered scenes. The baseline model, which relies on processing the entire image at global level, often fails to perceive these critical details. con- trast, our SD-RPN first employs its region proposal mechanism identify the most salient Rol for a given question-visualizae
by the green bounding boxes for subsequent fine-grained perception. This two-stage approach allows model accurately perform tasks like detailed text extraction, object attribute identification, and spatial reasoning, achieving correct answers where the baseline consistently fails.
In addition to the successful cases, we also provide failure case analysis in Fig. 9 The failure cases be broadly attributed three main reasons:
Incomplete Activation. The top-left "cup" case is representative example. The RPN correctly identifies the measuring cup but only activates on a small portion of it. This partial Rol provides incomplete visual information to the MLLM, leading an erroncous prediction.
Localization Error.
The top-right "book" example illustrates this type. The RPN is sup- posed highlight the book's 's cover read the name but erroneously highlights an unrelated region. Given this misleading visual cue, the MILM fails to produce the correct result.
Spatial Relationship Error. The bottom row shows errors of this type. Here, the RPN highlights all relevant objects correctly (e.g., "bowl" and "faucet"). However, the MLLM still produces the wrong result, suggesting that it struggles to preserve and reason about the precise spatial relationships between objects when they are processed as fine-grained crops.
F WHY PREDICTING Rol SCORES OUTPERFORMS USING RAW ATTENTION
Let X denote token-level features available to the region-of-interes. (Rol) predictor, Y E (0,1) is the latent foreground (FG) indicator, and A E [0, 1] is the model's response-to-image attention used as noisy proxy for Y. Define the posterior 17(x) = Pr(Y
Regression view (continuous Rol scoring). We train a predictor by minimizing the population squared loss
R(h) [((((X) 1)3
20

---

Published as a conference paper at ICLR 2026
What is the color of the shovel?
the baby stroller on the left right side of person orange?
yellow red C. blue D. black
SDDRPN:
A. Left B. right
SD-RPN: B
X
What writen in english the bottom right, under the reetangular imagas?
2nd from the lett?
+SD3-RPN:
+SD-RPN: Grappa
Register domain
X
x
What letter the taxi, letter inside black circle?
What the first namo of
penson's name the ded case?l
+SD-RPN:
+SD-RPN:
v
Budger Estimate for Phannacuutical Compendia Surveillance?
$600
x
many patients for Calcium placebu/Trace minerals placebo
SD $000000
V
SSD-PPX: 42
Figure 8: Qualitative comparison of our SD-RPN against the LLaVA-1.5-77 baseline on challenging samples from the V-Star, TextVQA, and DocVQA benchmarks. For each example, the top image displays the original input with the Region-of- Interett (Rol) predieted by our method, while the im- age below it visualizes the dense Rol map. These examples highlight SD-RPN's ability to precisely localize and analyze fine-grained visual information (e.g., text, small objects), leading to correct answers where the baseline, which processes the full image, consistently fails.
21

---

Published as a conference paper at ICLR 2026
Input Image
Rol Heatmap
Mixed Image
Input Image
Rol Heatmap
Mixed Image
Steve
How many cups the mecsuring cup?
What male namo is written on the white book?
SD-RPN:
+SD-RPN: Doug
the boad the left right side the faucet? Right B. Left
Is left right side the red vehicle? Right B. Lefl
+SD-PPN:
+SD-RPN:
Figure 9: Failure case analysis of SD-RPN on the LLaVA-1.5-7B baseline. We present examples where the baseline model succeeds, but our method fails. These failures can be categorized into three main types: incomplete activation (top-left), localization error (top-right), and spatial rela- tionship error (bottom row).
By pointwise conditional minimization, the optimal predictor is the conditional expectation
h" (x) E[A X = z].
(9)
Under standard noise models, h" an affine function 17(x:):
B[A X = (1 p1 po) n)(x) po,
class-conditional noise (CCN),
140 + (F 140) n(x), additive activation model.
In both cases, E[A X = x] is strictly increasing in nıx)) when Po + < or thresholding h* recovers the Bayes decision boundary up a constant shift.
> Pa. so
Noise reduction via conditional averaging. The raw attention signal A can be decomposed into its "signal" component, defined hX( E[A X]. and its "noise" component, X]. By construction, the noise is zero-mean conditional on the features (Ele | X] = 0). We can measure the quality of any estimator by its Mean Squared Error (MSH) with respect to this true underlying signal, h (X).
A E[A
The MSE of the optimal predictor ("X) with respect to the true signal is, trivially, zero:
E((h(() ht (X)) =
In contrast, the MSE of the raw attention signal A with respect to the true signal is: E[Var(A X)].
E [(A - ht (x)) = E[(A E[A x)> [ E[A XJ) x]]
This term, the expected conditional variance, represents the irreducible error inherent in the attention signal. As long as the attention is not a perfectly deterministic function the features (i.e., Var(A X) 0), we have:
E ( (X) h*)))) E {[AA>>(x))*
This inequality formally proves that the optimal predictor h² hX) is a strictly better, denoised esti- mate of the underlying signal than the raw attention A. Any learned predictor h that successfully approximates ht will therefore also be stable and accurate predictor.
22

---

Published as a conference paper at ICLR 2026
Classification view (binary Rol selection). For symmetric class-conditional noise (symmetric CCN) with flip rate p < 2'
Pr(A=1 X 2) (1- m()) + p.
(10)
strictly increasing transform of 7)))). Minimizing classifcatiion-caitaaan surrogate on A yields a scorer whose optimal threshold (shifted by p) implements the clean Bayes rule. In contrast, select- ing Rols directly via A suffers from the conditional variance Var(A X), leading to higher false positives/negatives, especially in low-margin regions.
Implication for Rol prediction. A Rol head trained to regress learns EJA X], which (i) order-preserving with 7/()) by the affine forms above, and (ii) enjoys reduced variance. Thresholding learned predictor h(X) therefore yields more accurate and stable Rol proposals than thresholding raw attention A, matching our empirical gains. Our proposed Label Assignment strategy can be seen further refinement of this principle, where we structure the learning problem as a classification task on high-confidence tokens to make the training process even more robust to the noise in A.
G THE USE OF LARGE LANGUAGE MODELS
Throughout the preparation of this manuscript, we utilized Large Language Model (LLM) as general-purpose writing assistant to enhance the quality and clarity our text. The LLM's role was strictly limited to that of linguistic refinement tool. We used it to help polish our initial drafts by:
Rephrasing sentences and paragraphs for improved readability, conciseness, and a more formal academic tone.
Correcting grammatical errors, spelling, and punctuation.
Improving the logical flow and transitions between sentences.
Assisting with the generation LaTeX code for tables based on our provided experimental data.
It is important to state that all core research ideas, the conceptualization of the proposed framework, the experimental design, the analysis of results, and the final scientific conclusions presented this paper were conceived and articulated entirely by the human authors. The LLM did not contribute to the scientific ideation or the results this work. The authors have reviewed, edited, and take full responsibility for all content in this manuscript.
23


Reviews:
[{"review_id": "4kf99zdk6T", "reviewer": "Reviewer_2wEb", "summary": "This paper proposes a Self-Distilled Region Proposal Network (SD-RPN). The pipeline first converts the response-to-image attention from the MLLM into high-quality pseudo RoI labels through two key steps: removing sink tokens and assigning labels. Then, the RPN is initialized using the corresponding layers of the MLLM. It collects hidden-state sequences from the second-to-last layer and uses the corresponding token sequences to predict an RoI map. Finally, the RPN is trained with self-distillation, constrained by these pseudo RoI labels. Trained on only a few question-answer pairs, the method achieves great accuracy and efficiency improvement on unseen benchmarks.", "questions": "1. SD-RPN was tested on LLaVA and DeepSeek-VL. How does it transfer to other architectures such as Qwen2.5-VL?\n2. What is the performance of SD-RPN if pseudo-labels were replaced by ground-truth bounding boxes? Is it the upper bound of SD-RPN?\n3. What is the performance of SD-RPN compared to thinking with image methods?\n4. Why was BCE chosen over other robust alternatives (e.g., focal loss or IoU-based loss) given the heavy class imbalance between foreground and background tokens?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Annotation-free RoI distillation mechanism that elegantly leverages internal attention for supervision. \n2. Improvement on accuracy and efficiency with comprehensive experimental validation across diverse benchmarks. \n3. Clear motivation and insight, especially the analysis of noisy attention and the role of sink tokens.", "weaknesses": "1. The pseudo-label generation method relies on fixed thresholds (e.g., $τ_{fg}$=0.2) to distinguish foreground, background, and ignored regions. These thresholds directly affect the quality of the pseudo labels, but the paper lacks experimental validation with different parameter settings across multiple datasets.\n2. Lack of experiments on very recent MLLMs (e.g., Qwen2.5-VL).\n3. Efficiency improvement can be further quantified in wall-clock latency and GPU cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BFLUk2a0L", "reviewer": "Reviewer_ScXn", "summary": "This paper proposes a method to enhance the visual comprehension capability of a multimodal large language model (MLLM) by training a region-proposal network, which is initialized from a subset of the MLLM's parameters. The training data for the region proposals are derived from the attention maps of intermediate layers in the MLLM. To obtain more precise and less noisy region-of-interest (ROI) targets, the authors introduce several techniques, such as removing sink tokens and dynamically assigning labels to instances.\n\nWith the trained region-proposal network, the MLLM can perform two-stage inference: in the first stage, the model predicts ROIs using the proposal network; in the second stage, it processes a high-resolution version of the selected ROI to enable more detailed visual information extraction. The authors conduct extensive experiments to validate the effectiveness of the proposed method, and provide ablation studies to assess the contribution of each design component.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The overall presentation of the paper is strong. The manuscript is well-written, logically structured, and easy to follow.\n\n* Extensive experiments are conducted to demonstrate the superiority of the proposed method across multiple MLLM benchmarks.\n\n* The method achieves notable performance improvements, particularly on OCR-related tasks, highlighting its strengths in fine-grained visual understanding.\n\n* The ablation studies and visualizations provide clear evidence of the effectiveness of the key components in the proposed approach.", "weaknesses": "* I question the necessity of extracting ROIs from attention maps, as they can be noisy and spatially imprecise. Given the availability of large-scale object detection datasets (e.g., OpenImages) and powerful MLLMs capable of generating captions, it may be feasible to construct a dataset with high-quality region-text annotations. It would be helpful for the authors to discuss whether their approach offers any advantages—such as cost, scalability, or task-specific alignment—over using such manually or model-annotated datasets.\n\n* The authors propose several strategies for utilizing the predicted bounding boxes. However, a concern arises: if only the cropped ROI is used for answering the current question, could this lead to a loss of global visual context? While the global scene may appear irrelevant to the immediate query, it might provide valuable information for subsequent questions in a multi-turn dialogue. In such cases, does the proposed two-stage inference risk discarding useful contextual cues, potentially harming performance over extended conversations?\n\n* The RPN is designed to predict ROIs based on the provided textual context. However, using a portion of an MLLM with billions of parameters for this purpose seems computationally heavy. Since the goal resembles that of open-set grounding models, it would be insightful to compare against lightweight alternatives such as fine-tuned Grounding DINO or YOLO-World. Could such models achieve comparable ROI localization performance with significantly lower computational overhead?\n\n* The standalone performance of the proposed RPN remains unclear. There are no dedicated benchmarks or quantitative metrics reported to evaluate its accuracy in ROI prediction. Could the authors clarify how they validated the quality of the generated proposals during development? For instance, were human evaluations or proxy metrics used?\n\n* I noticed that some visualization results are provided in the supplementary material. Including a few representative examples in the main paper—such as attention maps, predicted ROIs, and corresponding model outputs—would greatly enhance the reader’s understanding of the method’s working mechanism and effectiveness.\n\n* I am also interested in potential failure cases. Are there instances where the baseline MLLM answers correctly but the proposed method fails? Analyzing such cases could shed light on the limitations of the current approach and guide future improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CsywXoQSgA", "reviewer": "Reviewer_KUum", "summary": "This paper introduces a novel annotation-free framework termed the Self-Distilled Region Proposal Network (SD-RPN), designed to address the challenge that MLLMs face in comprehending fine-grained text or objects within high-resolution images. It circumvents the limitations of existing approaches: training-based methods that rely on large-scale annotated datasets, and training-free techniques that suffer from computational inefficiency and lower accuracy. The proposed approach transforms the inherently noisy attention maps within MLLMs into high-quality pseudo-RoI labels, which are subsequently used to train a lightweight RPN. Furthermore, the weights from middle layers of the MLLM are used to initialize the RPN, enhancing its performance. Empirical results demonstrate the high effectiveness of the method, achieving substantial absolute accuracy gains (over 10%) on benchmarks such as TextVQA and DocVQA when integrated into existing MLLMs.", "questions": "1. How were the specific threshold values (e.g., 0.2 and 0.1 for foreground/background definition) determined?\n2. Which types of images or questions are more prone to introducing noise in the attention maps?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a self-distillation framework for training a lightweight RPN, resolving the difficult trade-off between training-based methods, which require costly human annotations, and training-free methods, which are often inefficient and inaccurate.\n2. The pseudo-label generation pipeline proposed is clear and feasible, as it involves (i) identifying and removing \"sink tokens\" and (ii) adopting a selective classification strategy that labels only high-confidence foreground/background regions.\n3. The paper validates the effectiveness of the proposed approach through a wide range of benchmarks.", "weaknesses": "1. This paper introduces thresholds in stages such as pseudo-label generation and RoI prediction. However, the authors do not justify the rationale behind these specific threshold values, nor do they explore how variations in these thresholds might affect model performance.\n2. The paper seems to lack an in-depth analysis regarding the sources of \"noise\" during pseudo-label generation. For instance, it does not adequately address the types of noise involved, how such noise is generated, or what kinds of images or tasks are prone to producing noisy labels.\n3. The qualitative examples presented in Figure 5 effectively showcase the successes of SD-RPN. However, a comprehensive evaluation should also include an analysis of its failure modes. The paper would be strengthened by showing examples where SD-RPN fails and diagnosing the cause.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XmtzXor6V2", "reviewer": "Reviewer_jaiz", "summary": "**Catching the Details: Self-Distilled ROI Predictors for Fine-Grained MLLM Perception** proposes a Self-Distilled ROI Predictor (SD-RPN) to address the high-res image perception problems. Unlike previous sft-based or training-free ROI-based methods, this paper pioneered using the filtered middle layer attention map as supervision signal, and trained several middle layers to serve as the SD-RPN. The overall idea is clear, elegant, and generalizable to other encoder+llm type of VLMs. The effectiveness of the proposed method is demonstrated via experiment, with solid and comprehensive growth across benchmarks.", "questions": "1. Have authors considered ablating shallow layers? If not, is there a more convincing explanation?\n2. Why are DocVQA and ChartQA significantly lower than the token compression methods? \n\nI have a positive attitude towards this paper's contributions. I would maintain the high score if authors can address my concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The idea of Self-Distilled ROI Predictor is valuable, as it fully leverages the contextualization power of a VLM model, without relying on human-annotated data.\n2. The training is not computation-efficient, and demonstrates consistent growth when data quantity is increased (Tab. 4b). This shows the potential of scaling up of this method.\n3. The writing and illustration of the paper are excellent. The paper has expressive and clear figures, solid algorithms and detailed explanation for methods.", "weaknesses": "1. The experiments are limited to ROI methods and do not compare against other SoTA methods for high resolution. If authors could add some work such as Monkey, Token-Packer, Mini-Gemini, Honey-bee, LLaVA-NeXt, which have competitive numbers on the selected benchmarks such as DocVQA, the paper would be more solid.\n2. The design of taking middle layers is not ablated. I have read Appendix A. While intuitively the middle layers should be a balance of visual details and global semantics, it should be proven, since using the middle layers can be much more expensive in computation compared with shallow layers.\n3. (minor) The 1.6x slowdown does not support the claim of \"light weight\". Maybe try rephrasing the claim to be clearer.\n4. (minor) The hyperparameters, like B and Sbg, Sfg thresholds, are not ablated.\n\nReferences:\n[1] Li, Z.; Yang, B.; Liu, Q.; Ma, Z.; Zhang, S.; Yang, J.; Sun, Y.; Liu, Y.; and Bai, X. 2023d. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607\n\n[2] Li, W.; Yuan, Y.; Liu, J.; Tang, D.; Wang, S.; Zhu, J.; and Zhang, L. 2024a. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392\n\n[3] Li, Y.; Zhang, Y.; Wang, C.; Zhong, Z.; Chen, Y.; Chu, R.; Liu, S.; and Jia, J. 2024b. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814\n\n[4] Cha, J.; Kang, W.; Mun, J.; and Roh, B. 2024. Honeybee: Locality-enhanced projector for multimodal llm. In IEEE CVPR, 13817–13827\n\n[5] Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024a. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)