Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG
Authors: Wenbin Wang
Yongcheng Jing
Liang Ding
Yingjie Wang
Li Shen
Yong Luo
Bo Du
Dacheng Tao
Venue: ICML 2025 oral (2025)
Keywords: Multimodal Large Language Models
High-resolution Image Perception
Primary area: deep_learning->large_language_models
TL;DR: We propose Retrieval-Augmented Perception (RAP), a training-free framework that retrieves and fuses relevant image crops while preserving spatial context, with RE-Search dynamically selecting the optimal number of crops.

Abstract: High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs).  To drive progress beyond the limits of heuristic methods, this paper advances HR perception capabilities of MLLMs by harnessing cutting-edge long-context techniques such as retrieval-augmented generation (RAG). Towards this end, this paper presents the first study exploring the use of RAG to address HR perception challenges. Specifically, we propose Retrieval-Augmented Perception (RAP), a training-free framework that retrieves and fuses relevant image crops while preserving spatial context using the proposed Spatial-Awareness Layout. To accommodate different tasks, the proposed Retrieved-Exploration Search (RE-Search) dynamically selects the optimal number of crops based on model confidence and retrieval scores. Experimental results on HR benchmarks demonstrate the significant effectiveness of RAP, with LLaVA-v1.5-13B achieving a 43\% improvement on $V^*$ Bench and 19\% on HR-Bench. Code is available at https://github.com/DreamMr/RAP.

Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Wenbin Wang' Yongcheng Jing? Liang Ding² Yingjie Wang?
Li Shen Yong Luo
Bo Du 1 Dacheng Tao
Abstract
Query: Where is the small stone coirn located relative to the waterfall?
High-resolution (HR) image perception remains a key challenge in multimodal large language mod- els (MLLMs). To drive progress beyond the lim- its of heuristic methods, this paper advances HR perception capabilities of MLLMs by harness- ing cutting-edge long-context techniques such as carmeval-augmenned generation (RAG). Towards this end, this paper presents the first study ex- ploring the use of RAG address HR perception challenges. Specifically,
High-Resolution image
At the battom right of the waterfall
K=2
Layout
MLLM
The simple schematic of our Retrieval-Augmented Perception (RAP)
Baseline RAP
70
w/
Accuracy (%)
60
v1. 138 RAP
IntamV--11.5-000
GF74o
propose Retrieval- Augmented Perception (RAP), training-fie free framework that retrieves and fuses relevant image crops while preserving spatial context using the proposed Apatial-Awarene modate different tasks, the proposed Retrieved- Exploration Search (RE-Search) dynamically se- lects the optimal number of crops based model confidence and retrieval scores. Experimental results on HR benchmarks demonstrate the sig- nificant effectiveness of RAP, with VA-v1.5 13B achieving a 43% improvement on V* Bench and 19% on HR- HR-Bench. Code is available attps://gitae wwwpDreawwt///eeecctew.. /RAP.
8.
LLoVh-ovd..50
LLaVA-1.8-72
LL0-6--1B-4-77
Cogy
Lo
W-VL-308
Liat5k-2.55728
30
0.58
7B
133
348
Layout. To accom-
(b) Our RAP consistently improves performance across all MLLMs
Figure I. (a) Overview of the proposed Retrieval-Augmente Per- ception (RAP) framework, which divides the HR images into image crops for retrieval, followed by layout reconstruction to retain spatial information; (b) Performance comparison MLLMs across various model sizes. demonstrating consistent im- provements with our RAP on HR-Bench.
MLLMs (Liu et al., 2024a; Bai et al., 2023; Liu et al., 2024b; Wang et al., 2023; Abdin et al., 2024) typically process im- ages at a fixed resolution (e.g., 448 x 448). While this design streamlines the computational pipeline, introduces signifi- challenges, such as shape distortion and blurring when handling high-resolution (HR) images. These distortions notably impair the performance of MILMs, especially in tasks that involve analysing real-world images with varying resolutions, such as visual grounding and optical character recognition that demand fine-grain ane-grained visual details (Zhang et al., 2024a; Tian et al., 2022: 2023; Wang et al., 2024).
1. Introduction
Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language understanding, rea- soning, and interaction, leveraging visual signals to process and interpret visual information (Yin et al., 2023). Current
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Mul- timedia and Network Communication Engincering, Wuhan Univer- sity, Wuhan 430072, China "Nanyang Technological University, Singapore 639798 'The University of Sydney. Australia "Shenzhen Campus of Sun Yat-sen University. China. Correspondence to: Liang Ding <liangding liam gmail.com>>, Yong Luo <luoy- @ng@v wwg@@wh..ee www.www.uuu.uoo edu.edd.cn.. Dacheng Tao <dacheng.tad @ comi.
In response to this dilemma, emerging research on enhanc- ing the HR image perceptual capabilities of MLLMs has gained increasing attention. Existing approaches can be broadly categorised into three groups: (1) cropping- cropping-based methods (Chen et al., 2024c: Liu et al., 2024b; Li et al., 2024c), (2) HR visual encoder methods (Luo et al., 2024; Ge al., 2024; Lu et al., 2024), and (3) search-based meth-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Retricval-Augmented Perception: High-Resollutinn Image Perception Meets Visual RAG
ods (Wu& & Xie, 2024; Wang et al., 2025; Shen et al., 2024). Despite notable progress, both cropping- based and HR vi- sual encoder methods still require downsampling HR images to mitigate excessively long visual token sequences, result- ing in substantial loss of fine-grained details. Although search-based methods avoid downsampling, they face sev- eral limitations. These methods follow a top-down search from high to low resolution; however, at the initial stage, models struggle to accurately perceive small objects, often resulting in erroneous search paths. Furthermore, search- based approaches rely on hierarchical, layer-by-layer re- trieval, preventing parallel processing and rendering them inefficient and cumbersome for deployment.
their original order, 2) ordering them in descending order based on retrieval scores (Jin et al., 2024), and 3) preserv- ing the relative positional relationships among the retrieved crops. Our empirical results suggest that maintaining the relative positional relationships of the image crops signifi- cantly enhances HR perception, particularly for tasks that depend on spatial relationships.
In response to the 2nd question. this paper investigates the impact of the number of retrieved image crops. Our findings reveal that the optimal number of retrieved crops depends on the task type. For single-instance perception tasks, a small number of crops suffices for significant performance im- provements, whereas too many crops degrade performance due to the high image resolution. In contrast, for cross- instance perception tasks, fewer crops result in information loss and reduced performance, while more crops help pre- serve essential details and minimise performance degrada- tion. However, an excessive number of crops still harms performance due to challenges from overly high resolution.
These limitations prompt our rethinking of the fundamen- tal challenge in HR perception. Ideally, effective HR per- ception requires an MLLM with robust long-context capa- bilities- for instance, processing an 8K HR image with ViT-L/114 (Dosovitskiy et al., 2021) generates approximately ~300K visual tokens. This raises the question of whether the key to HR perception lies in enhancing the long-context capacity of MILMMs, rather than relying on existing heuris- tic approaches, particularly in light of recent encouraging advancements in long-context techniques for general LLMs. In particular, retrieval-augnen augmmetedd generation (RAG) has proven highly effective in recent long-context LLMs, by retrieving crucial fragments and reducing the impact of ir- relevant information (Jin et al., 2024). Motivated by this, this paper poses a largely overlooked question: Is it possible to directly enhance the long-context capability of MLLMs using RAG, as in general L.LMs, to overcome the limitations of existing HR perception methods?
In tackling the 3rd question, we integrate the insights gained from the previous investigations to design a new framework, which we term Retrieval-Augmentee Perception (RAP). As illustrated in Figure (a)), RAP processes high-resolution im- ages by retrieving image crops relevant to the query through Vi (Yu et al., 2024). We propose a simple yet efficient layout method, termed as Spatial- preserves the original relative spatial relationships among the image crops. To determine the optimal number of re- trieved image crops, we introduce a novel scheme termed as RE-Search (Retrieved-Exploraadeee Search), which adap- tively adjusts the number of crops based on the model's confidence in the sufficiency of the retrieved information.
Layout, which
However, exploring this research question presents signifi- cant challenges, as images, unlike text, are two-dimensional (excluding the channel dimension) and are characterised by width and height. As a pilot study, we begin by focusing on two key aspects: the layout of retrieved image crops and the impact of the number of retrieved crops on perfor- mance. This leads to the following specific challenges: 1) How should the retrieved image crops be organised? Fur- thermore, the number of retrieved key fragments plays a critical role in RAG performance (Jin et al., 2024), prompt- ing our second research question: 2) How does the number of retrieved image crops influence the final performance? Building on insights from these two questions, we further pose a third research question: 3) How can RAG systems he designed to enhance MI.LM perception of HR images?
In particular, VisRAG is first used to compute the similarity scores between each image crop and the query. We then retain the top K crops with the highest similarity scores, ensuring their relative spatial relationships are preserved through the Spatial-Awarenes Layout. To determine the optimal K, we construct RE-Tree, where each node rep- resents a new image synthesized by retaining different pro- portions of the image crops. The search process within this tree is guided by both the retrieved similarity scores and the model's confidence whether the image offers sufficient information to answer the query.
Our contribution is thereby the first investigation into using visual RAG to enhance HR image perception in ML.Ms. This is accomplished by a novel RAP, a training-free frame- work that comprises Spatial-Awarenee Layour to preserve the positions of image crops and RE-Search to adaptively select the optimal number of retained crops. Experiments demonstrate that RAP consistently delivers significant im- provements, with an average accuracy increase of 24% on HR image benchmarks and even general MLLM tasks.
To address the 1st challenge, we conduct series of experi- ments using the HR-Bench (Wang et al., 2025) to investi- gate the effects of different layout strategies. We evaluate state-of-the-ar (SOTA) MLLMS (Liu al., 2024a;b) across various layout configurations. Specifically, we compare three strategies: 1) arranging the retrieved image crops in
2

---

Retrieval-Augmented Perception: High-Resolltion Image Perception Meets Visual RAG
2. Related Work
3. Pilot Study
MLLMs consist of a Visual Encoder (Dosovitskiy et al., 2021; Radford et al., 2021) for extracting visual features and LLM (Touvron et al., 2023a:b) for decoding text, both initialized from pretrained models. A multimodal Connec- tor (e.g,, MLP) links the vision and language modalities. To align the resolution used during visual encoder pretraining (e.g, 336 x 336 in LLaVA), images are typically resized, which can distort and blur HR images. To address this, existing approaches fall into three categories: 1) cropping- based methods, HR visual encoder methods, and 3) search-based methods.
In this section, we conduct a systematic investigation into the challenges associated with employing RAG to enhance the perceptual capabilities of MLLMs, motivating the design of the proposed RAP framework in Sect. 4.
3.1. Preliminary
In this section, we introduce the pipeline for applying RAG to MILMs for the perception of HR images. Given an HR image, we divide it into image crop set, denoted as V = [VI....,uy.l where n is the number of image crops. Inspired by Yu et al. (2024), the query and image crops are independently encoded as text and images within the VLM,, yielding a sequence of hidden states. Subsequently, the similarity scores between the query embedding and the image crop embeddings are computed. The similarity score s(g, V) is calculated by the cosine similarity of the query and image crop embeddings:
Cropping-based methods. Representative cropping- basedd methods for HR MLLMs (Chen et al., 2024a; Zhang et al., 2024b; Liu et al., 2024c), such as LLaVA-v1.6 .6 (Liu et al., 2024b) and LLaVA-ov (Li et al., 2024a), segment images into multiple image crops. Each image crop is indepen- dently encoded using ViT (Dosovitskiy et al., 2021) and subsequently concatenated for LLM processing.
g VT
HR Visual Encoder. High-resolution image understand- ing can be enhanced by incorporating HR visual encoders without substantially increasing the number of visual tokens. For instance, Vary (Wei et al., 2023) and Deepseek-Ve VL (Lu et al., 2024) adopt the SAM (Kirillov al., 2023) to improve the performance of MI.LMs on HR images. MiniGemini- HD (Li et al., 2024b), LLaVA-HR HR (Luo et al., 2024), and ConvLLaVA (Ge et al., 2024) utilize ConvNeXt (Liu et al., 2022), employing techniques such as cross-attention or adapter to extract visual features.
s(q, V) = (1 |lall |[ ) 2
(1)
Finally, the top K image crops are selected based on the s(g, V) to facilitate the MLLM's perception of HR images.
In the following sections, we systematically analyze the impact of retrieved image crop layouts and quantities on HR-Bench, which consists of HR-Bench 8K and HR- Bench 4K. HR-Bench 8K, with 8K-resolution images from DIVSK (Gu et al., 2019) and the Internet, includes Fine-grained Single-instanee Perception (FSP) and Fine- grained Cross-instance Perception (FCP) tasks. Cropping 8K images around relevant objects produces HR-Bench 4K.
Search- based methods organize images into tree structure to extract query-relevant regions through top-down approach. 2C (Wang et al., 2025) leverages visual memory store objects and coordinates, re- trieving crops to generate text and reduce detail loss. Zoom Eye (Shen et al., 2024) employs a tree search algorithm to directly identify and extract relevant crops from HR images. Wu & Xie (2024) propose SEAL,, a architercee that actively reasons and retrieves essential visual information.
Search-based Methods.
3.2. Impact of the Layout of Retrieved Image Crops
This subsection investigates the relationship between the layout of retrieved image crops and the performance of MLLMs in the RAG system.
Experimental setting. We compare three layout strategies: 1) Sort according the retrieval scores in descending order; 2) After selecting the top K image crops, arrange them in the order in which the image crops appear, 3) Maintain the relative positional relationships of the image crops. We conduct experiments on HR-Bench using LLaVA-v1.6-7B.
Multimodality RAG. Multimodal RAG tasks include matching images to text and retrieving text-image pairs to answer questions (Chang et al., 2022; Han et al., 2017: Xia et al., 2024a;b). Yu et al. (2024) propose Vision-based Retrieval-augmented Generation to effectively utilize and retain data in multimodal documents.
Observations. As shown in Table 1, retrieving key image crops through RAG significantly improves performance on the FSP task but results in noticeable performance drop on the FCP task. Furthermore, maintaining the relative positions between each image crop achieves better perfor- mance balance between the FSP and FCP tasks.
Existing methods enhance MILMs' ability to perceive HR images, but processing extremely HR images (e.g., 8K) remains challenging. Inspired by RAG's success in handling long contexts for LLMs, this paper for the first time explores its use to improve MILMMs HR image perception.
Insights Maintaining the relative positional relationships between retrieved image crops is essential, particularly for
3

---

Retricval-Augmented Perception: High-Resolltion Image Perception Meets Visual RAG
FSP Task
FCP Task
Table 1. The effect of different layout strategies. While all three
80
strategies improve fine-grained pereeption, only strategy 3) excels
40
FCP tasks by preserving positions, achieving superior perfor- mance compared to other strategies.
70
35
Accuracy (%a
HR-Bench 4K
IR-BBench 8K
50
30-
FSP
FCP
Avg.
FSP
FCP
Avg.
40
Baseline
49.0
46.8
47.9
37.2
44.2
40.8
25
+ +1)
74.8
38.3
56.5
56.8
26.8
41.8
30
+
72.3
38.5
55.4
61.3
25.5
43.4
+3)
74.0
41.5
57.8
59.5
30.0
44.8
4
16 32
20
16 32
Number of Retrieved Image Crop
Number of Retrieved Image Crop
LL5VA.11.57B
LLaVA-VI.5-1B
LLaVA-116-8
L1-VA-11.--188
tasks requiring spatial awareness.
Figure 2. The effect of the number of retrieved image crops on model performance. FSP and FCP represent the fine-grained single-instance perception tasks and fine-grained cross-instance perception tasks, respectively.
3.3. Impact of the Number of Retrieved Image Crops
This subsection investigates the relationship between the number of retrieved image crops and the performance of MLLMs in HR image perception.
ships between the image crops. To adaptively select K, we propose Retrieved- Exteovved--xlorattion Search (RE-Search), which determines K based on the model's 's confidence in V and its similarity to the query. The Spatial- Layout and RE-Search are presented in the subsequent sections.
Experimental setting. We analyze the relationship between performance (i.e., accuracy) and the number of the retrieved image crops, using the LLaVA-v1.5 and LLaVA-v1.6.
Observations. As shown in Figure 2, we visualize the rela- tionship between the number of retrieved image crops (i.e., K) and performance. As K' increases, more image crops are introduced, providing additional visual information that enhances performance on FCP tasks. However, this also raises the image resolution, increasing the likelihood of the model generating incorrect answers. Conversely, smaller K retains only essential visual information, improving perfor- mance on FSP tasks but sacrificing significant visual details, which causes a notable performance decline on FCP tasks.
4.2. Spatial- Layout
In Sect. 3.2, we find that maintaing the positional relation- ship between image crops is essential. Thus, we propose simple and efficient method, termed Spatial-Awareness Layout. We denote M E {0, R| as binary matrix of size R x C, where R and C represent the number of rows and columns of image crops V, respectively. The M.j = 1 indicates an image crop to be preserved and Mr, = 0 indi- cates the image crops to be removed. We seek to construct a compressed matrix M' by removing any row or column of M that is entirely zero Formally, we define two index sets:
Insights. Different types of tasks require different num- bers of retrieved image crops K. For FSP tasks, smaller K improves results, but larger K reduces performance by increasing resolution. Conversely, for FCP tasks, larger K preserves visual information and outperforms smaller K.
R' = [il M.M.M. 1), C' = -J. = 1}. (2)
The compressed matrix M' E 10, with N,- = [R'| and N. = |C",, is then constructed according to: Mi; = Mr,, where R'] and j = -. This guaran- tees that M' retains all rows and columns of M containing at least one entry equal to 1, effectively discarding rows and columns composed entirely of zeros. Moreover, an mapping function c: (0, ..., N, 1] x {0, ..., No + {0, R- 1} x {O.... C 1} is defined as ((i.)) = (R' CD))), thereby enabling each coordinate (i,j) in the compressed matrix M' to be mapped back to its original position (i, in M. Finally, we initializes an blank image V' and iter- ates over the mapping 4 where each pair (i.3) is mapped to (i.j). For each mapping, Vi[l is assigned to the cor- responding V[] We use image V' to replace the HR image V for the MILMM to answer the query. The implement
4. Proposed Retrieval-Augmentte Perception
4.1. Method Overview
Driven by the aforementioned insights in Sect 3, we propose a novel framework - Retrieval-Augmennte (RAP). The design principle of RAP is to retrieve key image crops to replace the original HR image, preserving essen- tial visual information while reducing resolution to improve MILM perception of HR images. To achieve this, we di- vide the image into various crops, calculate similarity scores (Eq. 1) with the query, and select the top K image crops to synthesize& a new image V". We design a Spatial-Awarend Layout algorithm to maintain the relative positional relation-
Perception
4

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Image:
Query:
Where is the small stone cairn located relative to the waterfall?
RetrievalAuggmente Perception
Step
Step 2
Step
Awareness Layout
depth:
5: 0.3
s:
20.1s 0.1
depth:
0=2 0.3
#=01
0.2
depth: 3
K=2
Retrieval
Synthesis
A 0.1
@=0.3
0-03 A-2.2
At the bottom right.
1
1
0
0
0
Compress
L0
1_
0
0
RE-Search
M'
Can answer query?
image
h()
k(-)
hO-
Selected : Calculate Cost
Unselected
Layout
User
MLLM
MLLM
MLLM
Calculate heuristic function
Figure 3. Detailed illustration of our proposed RAP with running example. We firstly divide HR image into multiple image crops and compute the similarity score a between the query and image corps to retrieve the key image crops. We design simple and efficient method called Spatial-Awarnness Layout to maintain the relative positional relationships of the image crops. Since the number of image crops is highly sensitive to the task type, we propose RE-Search, which identifies the optimal K based on the model's 's confidence scores and retrieval scores.
of Spatial-Awareness Layout is shown in Algorithm 1.
Building upon the strengths of A", we introduce Retrieved- Exploration Search (RE-Search). In the following parts, we will elucidate the RE-Tree, novel structure that elegantly represents the search states within RE-Search, and the RE- ward function, which serves the guiding heuristic for this innovative approach.
4.3. Retrieved-Exploration Search
In Sect. 3.3, we find that different types of tasks significantly influence the choice of K. Here, we utilize a search algo- rithm to obtain the optimal K. For search algorithm, we consider two primary factors: Efficiency, ensuring high effi- ciency for optimal user experience, and Robustness, guar- anteeing consistent results across multiple runs in image perception tasks. Existing tree-search methods (Wang et al., 2025) require visiting all nodes, leading to low efficiency. With the development of O1, many recent works (Yao et al., 2024; Zhao et al.) employ Monte Carlo Tree Search (MCTS) to find the optimal reasoning path. However, MCTS relies on random sampling, resulting in a lower robustness. A" search algorithm uses heuristic function to intelligently guide its exploration. This heuristic allows A' to priori- tize promising paths, significantly accelerating the search process. Furthermore, A" explores the nodes in the same order and find the same optimal path, ensures high robust- ness. However, effectively defining the state representation and designing an appropriate heuristic function for A" is non-trivial challenge.
RE-Te Tree Representation. Inspired by Wang et al. (2025); Shen et al. (2024), we model the HR image as a tree. Unlike existing search-based methods, we represent distinct nodes at the same layer by preserving different K image crops. This enables the model perceive lower-resolution images from the begining, mitigating the risk of the MILM converg- ing to suboptimal solutions. We denote P = {pi,..., Pn as the retention ratio. For instance, for the first child node n1, we retain the top N' x P1 image crops. The N' repre- sents the number of image crops for the current image. To obtain a complete image for calculating the REward func- tion, we employ Spatial-Awarent Layout to assemble the individual image crops into complete image V
REward Function. A search is best-first search algo- rithm that prioritizes nodes with the lowest combined cost, calculated as the sum of the actual cost g(t.) from the start node to to t, and the estimated cost h(tz) to the goal. In our RE- Rearch, the path from ig to f, is represented as the
5

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Algorithm Spaaiall-wareness Layout
answer the query. Therefore, we assign lower weight to h((t,) at the beginning and gradually increase its weight as
function SpatialLayout(V. M)
R' < {i j s.t.Ma, =}
the tree depth grows. Mathematically, the cost function f(t)
can be written as:
C' [j s.s. =1}
Nr | No + ICI
f(ts) = = (1-
(1-w) w) g(ts) + w h(tt);
(5)
Construct a binary matrixM' for i 1 N, 1 do for - Nc- -1- do
1 à +b, b,
w = (1 6) b) (1
(6)
i + [' j C
where is a bias value, set here 0.2 and d denotes the depth of the image tree.
M1 6 M
end for end for Initialize a blank image V for 7= == N, do
4.4. Algorithmic Workflow
In this section, we introduce how to use our RAP per- ceive HR image. Given HR image I, we first divide the HR image into various image crops V, with the size of each image crop not exceeding the predefined resolution of the retriever's 's image encoder. Subsequently, we utilize VisRAG (Yu et al., 2024) to compute the cosine similarity between the query and image crops. We use RE-Search to search the optimal K image crops and using the Spatial- Awareness Layout to synthesize the image Vy, which re- places the original HR image V as input to the MLLM. We denote as the answering confidence which is calculated by Eq. 4. When e exceeds a predefined threshold т, the search terminates. We set T = 0.6 throughout the paper. The implementation of RAP is shown Appendix A.
for j= N. do
i + -- j- C'ūi]
if M then
f.j
V[i.j] [i
end if end for end for return V' end function
progression from the original HR image to the currently retained top-K image crops. We use the similarity score between these K image crops and the query as g(t,):
5. Experiments
a
(ts))
Σ s(1, vz),
In this section, we evaluate our RAP HR benchmarks and general MLLM benchmark. Further experimental results, including the influence of inference computation scale and the effect of the hyperparameter, are provided in Appendix B. Case studies are illustrated in Figures 8~~9 in the Appendix C.
where n represents the number of image crops, and o; rep- resents the i-th image crops for current image V. Inspired by Shen et al. (2024), we use the model's confidence in whether the current image V can answer the given query the cost from l,, to the goal:
5.1. Results on HR Benchmark
h(t,,) = - Po("Yes")pp (g). V).
(4)
Benchmarks. We evaluate our RAP on two HR bench- marks: V Bench and HR-Bench. V Bench, derived from SA-1B (Kirillov al., 2023), averages resolution of 2246 x 1582. More details about HR-Bench can be found in Sect. 3.1.
where Po represents the MILM and Ph(-) represents the prompt (e.g.. "Question: {q}- Could you answer the ques- tion based on the available visual information? Answer Yes or No.") used to query the MLLM for calculating the confi- dence that the answer is "Yes". We utilize the model's confi- dence to estimate the cost from the current to the target node, analogous to the heuristic function in the A algorithm. A lower h(-) indicates a higher likelihood of containing essen- tial information, warranting prioritized exploration.
Main Results. As shown in Table 2, compared to the base- line MLLM, the performance of nearly all models signifi- cantly improved with our RAP. demonstrating the model- agnostic trait of RAP. We find that our RAP can bring signif- icant improvements in both FSP and FCP tasks. Our RAP brings maximum of 21.0% and 21.7% accuracy improve- ment on HR-Bench 4K and HR-Bench 8K respectively. Additionally, for tasks requiring spatial reasoning capabili- ties, RAP demonstrates significant improvements compared to the baseline (e.g., +39.5% accuracy on V Bench using
Since MILM cannot accurately perceive the HR image at the beginning, the h(tz) provided at shallow depths of the tree is unreliable. As the tree depth increases and the image resolution gradually decreases, the model becomes more confident in determining whether the current image can
6

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Table Comparison of RAP (upon several advanced models) with existing works on high-resolution benchmarks.
The hest
performance in each task is in-bold.
V Bench
HR-Bench 4K
HR-Bench 8K
Method
Attrihute
Spatial
Overall
FSP
FCP
Overall
FSP
FCP
Overall
Open-source MLLMs
LLaVA-v1.6-7B (Liu al., 2024b)
60.9
63.2
61.8
49.0
46.8
47.9
37.3
44.3
40.8
LLaVA-v1.6-13B (Liu et al., 2024b)
60.0
64.5
61.8
49.8
41.3
45.5
38.0
38.3
38.1
LLaVA-v1.6-34B (Liu et al., 2024b)
55.3
50.5
52.9
44.5
50.3
47.4
LLaVA-HR-X-I33 (Luo al., 2024)
61.3
46.0
53.6
49.5
44.3
46.9
LLaVA-HR-X-7B (Luo et al., 2024)
51.3
64.5
56.5
57.8
46.3
52.0
42.0
41.3
41.6
IntemV1-1.5-26B (Chen et al., 2024c)
69.5
51.8
60.6
69.3
48.5
57.9
Yi-VL-34B (Young et al., 2024)
46.0
42.8
44.4
39.5
38.5
39.0
Closed-source MIL.Ms
GPT 4o (Hurst et al., 2024)
66.0
70.0
48.0
59.0
62.0
49.0
55.5
QWen-VL-men (Bai ct al., 2023)
65.0
52.0
58.5
54.0
51.0
52.5
Baseline and RAP
LLaVA-v1.5-7B (Liu al., 2024a)
43.5
56.6
48.7
38.5
33.8
36.1
33.0
31.3
32.1
-w/ RAP
90.4
96.1
91.1
73.8
40.5
57.1
72.3
35.3
53.8
LLaVA-vI.5-13B (Liu et al., 2024a)
41.7
35.3
47.1
45.2
41.3
43.3
37.5
38.0
37.8
ww/ RAP
89.6
90.8
89.8
74.3
46.0
60.1
76.5
42.0
59.3
LLaVĀ-ov-0.5B (Li al., 2024a)
63.5
64.5
63.9
63.5
39.5
51.5
47.3"
38.3
42.8
-w/ RAP
80.0
84.2
83.6
80.3
42.3
61.3
81.8
45.3
63.5
LLaVA-v1.5-7B). The results show that our method has clear advantage with HR images.
the baseline. However, we find a significant improvement in the FSP task, but there is a noticeable performance drop in the FCP task. By incorporating the Spatial-Awarende Layout, the relative positional relationships between image crops are preserved, leading an improvement in accuracy the FCP task compared + VisRAG. Finally, we utilize RE-Search to determine the optimal K for different samples, resulting in significant improvements in both the FSP and FCP tasks, with an average improvement of 21.7% accuracy compared to the baseline.
5.2. Results on General Multimodal Benchmark
Benchmark. We conduct additional evaluations of RAP us- ing the MME-RealWorld (Zhang et al., 2024c), manually curated benchmark designed for partical, real-world scenar- ios. This benchmark encompasses five primary categories and 43 sub-class tasks. Due space constraints, we present results for 9 sub-tasks that exhibit notable performance vari- ations with RAP.
5.4. Performance and Efficiency
Main Results. As shown in Table 3, RAP improves the performance of LLa VA-.1.1---13B on most sub-tasks, espe- cially on MO/Orientation (+7.3%), AD/Intention (+6.0%), and OCR/license (+10.)%) However, we observe that tasks involving Diagram and Table types do not exhibit significant improvements and, in some cases, even performance degra- dation. We find that this due to the reliance of such data on the model's spatial awareness and reasoning capabilities, which are inherent limitations of current MLLMs.
Efficiency concerns regarding RAP may arise among re- searchers. To address this, Table 5 presents a comparative analysis throughput and accuracy against SOTA search- based methods (e.g., DC² and Zoom Eye). RAP achieves superior efficiency and performance by directly computing the relevance between image crops and the query, elimi- nating the need for hierarchical image partitioning, thereby significantly accelerating the search process. More com- parison results with search-based methods can be found in Appendix B.5.
5.3. A Ablation Study
To better understand the role of each module in our RAP, we conduct ablation study on Bench 8K using LLaVA-v1.5. 7B. As shown in Table 4, we first use VisRAG to retrieve key image crops, replacing the original HR images, resulting an average improvement of 4.5% accuracy compared to
5.5. Alternative to Model Logit Confidence
In RE-Search, we use the logit as a measure of the model's confidence that the current image V can answer the given query. However, for some closed-source models, is not possible access the model's output logits. To tackle

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Table Comparison of the RAP against the haseline MI..M the MME-Rea/World benchmark. MO: Monitoring; AD: Autonomous
Driving. The "A(t)" represents the performance gains of our RAP against the baselines.
MO
AD
OCR
Method
Property
Orientation
Color
Intention
Attention
Visual
license
Text
Address
LLaVA-vI.5-13B
31.0
14.7
21.9
16.6
27.2
36.3
46.6
46.0
39.7
-w/ RAP
38.0
22.1
27.8
22.6
31.3
42.3
56.9
52.5
45.1
A(t)
+7.0
+7.3
45.9
+6.0
+4.2
+6.0
+10.3
+6.5
+5.4
Table Ablation study of different module in RAP. "SL" denotes Spareel-Awareness Layout. We first incorporate VisRAG to retrieve K key image crops, where K = 8. Then, we add Spatial- Awareness Layour preserve the relative positional information of the image crops. Finally, we incorporate RE-Search to search the optimal K.
Table 6. Comparison of the performance of different confidence score calculation methods. "G" represents the generation-bas confidence score and "L" represents the logit-based confidence ScoTe.
Method
HR-Bench 4K
HR-Bench 8K
LLaVA-ov-0.5B
51.5
42.8
RAP (G)
57.0
52.1
HR-Bench 8K
A(1)
w/ RAP (L)
61.3
63.5
FSP
FCP
Overall
LLaVA-v1.5-7B
33.0
31.3
32.1
Casine Similarity Score:
VisRAG
52.3
25.0
38.6
+6.5
40
Logit-based
SL
50.0
27.5
38.8
+6.8
Gemeration-based
RE-Search
72.3
35.3
53.8
+21.7
30
Table Evaluation of performance and inference efficiency. We analyze the correlation between throughput (samples per minute) and accuracy of LLaVA-v1.5. 13BB-B3B enhanced with our RAP, com-
Pruportion (%)
20-
paring it agains search-based methods on HR-Bench 4K.
Method
Throughputt
Accuracyt
DC² (Wang al., 2025)
2.1
51.5
10
Zoom Eye (Shen al., 2024)
3.3
58.0
RAP
4.2
60.1
0.2
0.4
0.6
0.8
1.0
this issue, explore simple alternative approach using generation-basee confidence scores. Specifically, we design a scoring prompt that asks the model to evaluate whether the given image contains sufficient information to answer the question. The constructed scoring prompt can be found in Appendix A.4.
Confidence
Figure 4. Analyzing the distribution of logit-based and generation- based confidence scores.
We conduct experiments HR-Bench using LLaVA-ov- 0.5B. The experimental results are shown in Table 6. Al- though the generation-basee confidence score performs worse than the logit-based confidence score, still shows a clear improvement over the baseline (achieved an average improvement of 7.4%), demonstrating that generation -bsed confidence scores through the model can also lead signif- icant performance gains.
two types confidence scores, indicating a remarkable de- gree of alignment between their distributions. Interestingly, generation-based confidence scores tend to be consistently higher than their logit-based counterparts. Nonetheless, RAP utilizing generation-basee confidence scores continues to deliver substantial improvements.
5.6. Effect of Crop Size
To further investigate the extent to which the generation- based confidence score can replace the logit- confi- dence score, we calculate the cosine similarity between the two scores for the same images. As shown in the Fig- ure 4, a high cosine similarity score of 0.97 between the
To investigate the impact of crop size, we perform experi- ments on HR-Bench 8K using -LaVA--v-0.5B. As shown in Table 7, we find that while variations in crop size result in relatively minor differences, all configurations of our RAP yield substantial performance gains over the baseline.
8

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Table Relationship between erop size and RAP performance.
Table 9, Effect of retrieval on HR-Bench 8K. We compare two methods: randomly retaining K image crops (Random) and re- trieving K image crops. The "half" refers to retaining half of the imgge crops (Retrieval).
HR-Bench 8K
Method
FSP
FCP
Avg.
LLaVA-ov-0.5B
47.3
38.3
42.8
RAP (224 x 224)
82.0
40.8
61.4
HR-Bench 8K
Method
RAP (448 448)
81.8
45.3
63.5
FSP
FCP
Avg.
w/ RAP (896 896)
76.8
43.3
60.0
Random (K = 4)
25.0
23.8
24.4
Random (K = half)
29.0
24.5
26.8
Retrieval (K = 4)
52.3
25.0
38.6
5.7. Effect of Retriever
To explore the impact of retrieval quality on RAP perfor- mance, we conduct experiments on HR-Bench 8K using LLaVA-ov-0.SB with SigLIP and VisRAG. Due the lim- ited text input length of SigLIP, utilize MILM extract noun phrases from the query to compute relevance with
35-
35-
FCP
Rocalned Rate (hl)
Redeived Rete
3.0
image crops.
15-
1
As shown in Table 8, we evaluate the retrieval quality of SigLIP (Zhai et al., 2023) and VisRAG (Yu et al., 2024), find- ing that VG achieves superior retrieval performance. Notably, our RAP significantly enhances performance even with the relatively weaker retriever (SigLIP); for instance, it delivers a 9.1% overall enhancement on HR-Bench 8K.
180
260
369
400
580
25
50
75
100
128
150 178 2m1
Number of Retrieved Imaage Crop
Numbher of Retrieved Image Crap (K)
(a)
(b)
Figure 5. Analyzing the distribution for selecting K using our RAP. (a) The distribution of K selected by RAP, where "All" denotes the total number of image crops in the original image. (b) The distribution of K corresponding to different task types.
Table 8. Analyzing the relationship between RAP performance and retriever using V.LaVA-ov-O.5B with IIP and VisRAG. The "Params." refer to the total parameters of the retrievers.
swer this question, we visualize the distribution of the num- ber retrieved image crops (K) for ALaVA-v1.5-7B w/ RAP on HR-Bench 8K. As shown in Figure 5(a), our RAP effectively reduces the number of image crops, resulting +21.7% accuracy improvement. Additionally, for the FSP task, the K selected by our RAP is smaller, while for the FCP task, it is widely distributed across the range cor- responding to larger K (e.g., K 2 60). The experiment results demonstrate that our RAP can provide accurate K, thereby effectively reducing the image resolution.
Bence 8K
Method
Params.
FSP
FCP
Arg.
LLvVA-ov-0.BB
47.3
38.3
42.8
w/ RAP -
883M
65.0
38.8
$T.9
w/ RAP (VisRAG)
3.34B
81.8
45.3
63.5
5.8. Why Does Our Method Work?
Reviewing the design principles RAP: Retrieve image crops related to the query to reduce the image resolution in- put the MILM, thereby enabling the MI.LM perceive images more accurately. To explore the underlying mecha- nism of RAP, we perform experiments that help address the following questions:
6. Conclusion
In this paper, propose novel training-free framework Retrieval-Augmenned Perception (RAP) to enhance HR image understanding in MLLMs. We empirically demon- strated the effectiveness and universality RAP on several widely used MLLM benchmarks. From the results, we mainly conclude that: (1) Retrieving image crops relevant to the query can result in significant improvements; (2) Maintaining the relative spatial relationships of the retrieved image crops is essential, particularly for tasks that rely on positional information; (3) The number of image crops that need to be retained varies across different task types. In our future work. we will explore more token compression techniques further enhance HR perception and efficiency.
I) Is it truly necessary to retrieve image crops relevant to the query? we compare randomly retained image crops with query-relevan image crops using LLaVA-v1.5-4 on HR-Bench 8K. As shown in Table 9, randomly retained K and half of the image crops, comparing them with K image crops retrieved through VisRAG that are relevant to the query. The results indicate that retaining query-relevant image crops is necessary.
2) Can RAP accurately select an appropriate K? To an-

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Impact Statement
Chen, Z., Wang, W. Tian, H., Ye, S., Gao, Z., Cui, E.,, Tong, W., Hu, K., Luo, J., Ma, Z.,, al. How far are gpt- 4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint, 2024c. URL https: http//aaxvvvvo/// wwwpiiioorgg/baae.. ababbbs22444. 6821.
Our work introduces a training-free framework, Retrieval- Augmented Perception (RAP), which significantly improves the ability of multimodal large language models (MLLMs) to perceive and reason over high-resolution (HR) images. Specifically, by combining retrieval techniques with effi- cient spatial-awareness and adaptive search, RAP enables MLLMs to capture finer visual details and spatial relation- ships in images resolutions previously unattainable with standard approaches.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. URL https: // //openreview net forumorriiooo www.e/ooomidd/i/cc////oaao.aa FNTTy.
Acknowledgements
Ge, C.,, Cheng, S., Wang, Z., Yuan, J., Gao, Y., Song, J., Song, S., Huang, G., and Zheng, B. Convllava: Hi- erarchical backbones as visual encoder for large mul- timodal models. arXiv preprint, 2024. URL https: //arxiv. httpa/ab/220..1111
This project is supported by the National Key Research and Development Program of China (2033FFC2005700) the National Natural Science Foundation of China (Grant No. U23A20318, 62276195 and 62225113), the Science and Technology Major Project Hubei Province (Grant No. 2024BAB046), the Foundation for Innovative Research Groups of Hubei Province (Grant No. 2024AFA017). This project is supported by the National Research Foundation, Singapore, under its NRF Professorship Award No. NRF- P2024-001.
Gu, S., Lugmayr, A., Danelljan, M., Fritsche, M., Lamour, J., and Timofte, Div8k: Diverse 8k resolution image dataset. In ICCVW, 2019. URL
https://leeexplor www.:eeeeeeeeee.......ooooo wrw.eeo//aaabstc.t.t.e.... document/9021977
Han, X., Wu, Z., Huang, P. X., Zhang, X., Zhu, M., Li, Y., Zhao, Y., and Davis, L. S. Automatic spatially-aware fashion concept discovery. In ICCV, pp. 1463-1471, 2017.
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadal- lah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint, 2024. URL https: // /ttp///avvv/////////eee/eaeee.ea org./oorg/oooo abs/ 2404. 14219.
Hiippala, T., Alikhani, M., Haverinen, J., Kalliokoski, T. Logacheva, E., Orekhova, S., Tuomainen, A., Stone, M., and Bateman, J. A. Ai2d-rst: multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661-688 2021.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint, 2023. URL https:/ aariiiivv.. org/ vrrg/bbo abs/ /bss/308....... 12966.
Hong, W., Wang, W., Ding, M., Yu, W., Lv, Q., Wang, Y., Cheng, Y., Huang, S., Ji, J., Xue, Z., et al. Cogvlm2: Vi- sual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
Chang, Y., Narang, M., Suzuki, H., Cao, G., Gao, J., and Bisk, Y. Webqa: Multihop and multimodal qa. In Pro- ceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 16495-16504, 2022.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., al. Gpt-4o system card. arXiv preprint, 2024.
Chen, K., Thapa, R., Chalamala, R., Athiwaratkun, B., Song, S. I.,, and Zou, J. Dragonfly: Multi-resolution zoom supercharges large visual-language model. arXiv preprint, 2024a. URL https::///// http://xxxvvvv///// org. abs/ 2406. 00977.
Jin, B., Yoon, J., Han, J., and Arik, S. O. Long-conter Ilms meet rag: Overcoming challenges for long inputs in rag. arXiv preprint, 2024.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, I., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-.,, et al. Segment anything. In ICCV, 2023. URL
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z.,, Duan, H., Wang, J.. Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? arXiv preprint, 2024b. URL https: ptttpp:///axii....// arxiv.org.........oo.ooo........ abs/2403. 20330.
/ttps/////pppcccsss/eeeee thece.....t.......ee thecvf. com//conteet////////////ttt/t/t Segment Berg paper html.
10

---

Retricval-Augmented Perception: High-Resolltion Image Perception Meets Visual RAG
Li, B., Zhang, Y., Guo. D., Zhang, R., Li, E,, Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., al. Llava- onevision: Easy visual task transfer. arXiv preprint, 2024a.
Masry, A., Do X. L., Tan, J. Q. Joty, S., and Hoque, E. Chartqa: benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, 2022.
Li, Y., Zhang, Y., Wang, C.. Zhong. Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the poten- tial multi-modalin vision language models. arXiv preprint, 2024b. URL hwt.h:tt/////e /ttpss///avvvv../ www.orggaaaooo. arg/abs/ abs/ 2403. 18814.
Mathew, M., Karatzas, D., and Jawahar, C. Docyqa: A dataset for vqa on document images. In WACV, 2021. URL
https: http:/::///pppaccccesee www.we.hecccccccm com/ content/wACv20211 A Datasaste__t for
Barrrtt___o__________ VQA_ oot__o____oooouuueeeettteeee
Document Images_
Li, Z., Yang, B., Liu, Q.. Ma, Z., Zhang, S., Yang. J., Sun, Y., Liu, Y., and Bai, X. Monkey: Image resolution and text label are important things for large multi-modal mod- els. In CVPR, 2024c. https::::// / thecvf com/ http://ttteee///2////////////// hhtpm/htmmm..///////////////mm/t Monkey_Tmager Trinkee_Tmaae__eeollttitoo 2024 _wl_Arre__m__atttt_t_eeee
WACV_ pacV_2201_paaer.htte
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. ICML, 2021. URL https: proceed ngs ml pww.spp.ss/ee..................e vv1.//v11/////// 39/ra/orrd22....
Text_
_VP_
Shen, H.,, Zhao, K., Zhao, T., Xu, R., Zhang, Z., Zhu, M., and Yin, J. Zoomeye: Enhancing multimodal Ilms with human-like zooming capabilities through tree-based im- age exploration. arXiv preprint arXiv:2411.16044 2024.
Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Liu, H. Li C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge. 2024b. URL io/blog/
Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, 2019. URL https:/ /ttpps://textteee /ttxttqx.ttt.tt org/
2024-01- 001--01-300
Tian, Z., Wen, Z., Wu, Z., Song, Y., Tang, J., Li, D., and Zhang, N. L. Emotion- -awate multimodal pre-training for image-grounded emotional response generation. In Database Systems for Advanced Applications: 27th Inter- national Conference, DASFAA 2022, Virtual Event, April 11-14. 2022, Proceedings, Part III, pp. 3-19, Berlin, Hei- delberg, 2022. Springer- Verlag, ISBN 978-3-031-0011 1-00128-------- 4. doi: 10.
Liu, H., You, Q. Han, X., Wang. Y., Zhai, B., Liu, Y., Tao, Y., Huang, H., He, R., and Yang, H. Infimm- hd: A leap forward in high-resolution multimodal un- derstanding. arXiv preprint, 2024c. URL https: //arxiv. wwwxiv.orrg//.ooooooooooo abss/4453 01487.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In CVPR, 2022.
URL https:
doi . org / 10 007 9900//78---33
URL https: htttps:///penaaceess: thecvf. com/ Convvii 2tt_he__2220______22222222
Tian, Z., Xie, Z., Lin, F., and Song, Y. multi-vier meta- learning approach for multi-modal response generation. In Proceedings of the ACM Web Conference 2023, WWW
html.
'23, pp. 1938-1947, New York, NY, USA, 2023, Associa- tion for Computing Machinery. ISBN 9781 9781450394161. doi: 10. 00.1145/3543307.558885542 URI. ht. //doi. /oi. 1ttp/0..115/355430 /50.//333335557733333333 3583448.
Liu, Z. Dong, Y., Liu, Z., Hu, w., Lu, J., and Rao, Y. Oryx milm: On-demand at arbitrary resolution. arXiv preprint arXiv:2409. 12961, 2024d.
understanding
Tong, P., Brown, E., Wu, P., Woo, S., IYER, A. J. V., Akula, S. C., Yang, S., Yang. J., Middepogu, M., Wang, Z., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. Advances in Neural Information Processing Systems, 37:873 10-87356, 2024.
Lu, H., Liu, W. Zhang. B., Wang. B., Dong. K., Liu, B., Sun, J., Ren, T., Li, Z.,, Yang, H., Sun, Y., Deng, C., Xu, H. Xie, Z. and Ruan, C. Deepseek-vl: Towards real-world vision-language understanding. 2024. URL wtgg/bb//////ooooooogggoooeoeeb/ abs 2403. 05525.
attpps:/aat /arivv..//iii/////a///a/eaa org/
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint, 2023a. URL https: wwwp..oog/bb////0//..o0
Luo, G., Zhou, Y., Zhang, Y., Zheng, X., Sun, X., and Ji, R. Feast your eyes: Mixture-of of-uur--o-rrsolutioon adaptation for multimodal large language models. arXiv preprint, 2024. URL httph:tt/////..t.. //tpx/aavvvv//////////////..wwvv or.. htggabbs/400) 03003.

---

Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint, 2023b. URL https: //arxiv.or org/ abs/ / 2307 02888.
Young, A., Chen, B., Li, C., Huang, C.,, Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., al. Yi: Open foundation models by 01. ai. arXiv preprint, 2024. URL https: /arpssi//axxvv.oo/
Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z.., Wang, S., Han, X., Liu, Z.,, al. Visrag: Vision- based vetrieval-augmennn generation multi-modality documents. arXiv preprint, 2024.
Wang, W., Lv, Q. Yu, W. Hong, W., Qi, J., Wang, Y., Ji, J., Yang. Z., Zhao, L.,, Song, X., Xu, J., Xu, B., Li, J., Dong. Y., Ding, M., and Tang, Cogvim: Visual expert for pretrained language models, 2023. URL https: //arxiv. org/ abs/ / 2211..1111111 03079.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF international conference on com- puter vision, pp. 11975-11986, 2023.
Wang, W., Ding, L., Shen, L., Luo, Y., Hu, H., and Tao, D. Wisdom: Improving multimodal sentiment analysis by fusing contextual world knowledge. In Proceedings of the
Zhang, Y., Liu, Y., Guo, Z., Zhang, Y., Yang. X., Chen, C., Song, J., Zheng, B., Yao, Y., Liu, Z., et al. Llava-uhd v2: milm integrating high-resolution feature pyramid via hierarchical window transformer. arXiv preprint, 2024a.
32nd ACM International Conference on Multimedia, MM
'24, pp. 2282-2291, New York, NY, USA, 2024, Associa- tion for Computing Machinery. ISBN 9798400706868. URL //doi.
doi: 10. 01114//666447733614440 org/ / 114411455000 101111111364540 3631403.
Zhang, Y.-F. Wen, Qu, Fu, C., Wang, X., Zhang, Z.. Wang, L., and Jin, R. Beyond llava-hd: Diving into high- resolution large multimodal models. arXiv preprint, 2024b. URL fttps://arxtie org/ wwwg/bs//////ooooooooooooooeeeee 2406. 08487.
Wang, W., Ding, L.. Zeng, M.. Zhou, X., Shen, L., Luo, Y., Yu, W., and Tao, D. Divide, conquer and combine: A training-free free framework for high-resolution image percep- tion in multimodal large language models. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7907-7915, 2025.
Zhang, Y--F,, Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., al. Mme- realworld: Could your multimodal IIm challenge high- resolution real-world scenarios that are difficult for hu- mans? arXiv preprint, 2024c
Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., and Zhang. X. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint, 2023. URL https: /wtxxv.xvvvv./////// org/ /bs/ 2312.06109,
Zhao, Y., Yin, H.,, Zeng, B., Wang. H., Shi, T., Lyu, C., Wang, L., Luo, W., and Zhang, K. Marco-ol: Towards open reasoning models for open-ended solutions, 2024a. arXiv preprint.
Wu, P. and Xie, S. V Guided visual search core mechanism in multimodal llms. In CVPR, 2024.
Xia, P., Zhu, K., Li, H., Wang, T., Shi, W., Wang, S., Zhang, L., Zou, J., and Yao, H. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint, 2024a.
Xia, P.. Zhu, K., Li, H., Zhu, H., Li, Y., Li, G., Zhang, L., and Yao, H. Rule: Reliable multimodal rag for factuality in medical vision language models. In EMNLP, pp. 1081- 1093, 2024b.
Yao, H., Huang, J., WI, W. Zhang, J., Wang, Y., Liu, S., Wang, Y., Song, Y., Feng, H., Shen, L., et al. Mulberry: Empowering milm with ol-kike reasoning and reflection via collective monte carlo tree search. arXiv preprint, 2024.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint, 2023.
12

---

Retrieval-Augmented Perception: High-Resolltion Image Perception Meets Visual RAG
Appendix
This appendix presents a detailed description of the proposed Rerrieval-uugmentte Perception (RAP), along with additional results from comprehensive experiments, ablation studies, and case analyses. The structure of the appendix is summarized as follows.
Appendix A provides the implementation details of RAP, including preprocessing and hyperparameter settings. Specifi- cally, we introduce the specific implementation of the Spatial-Awarened Layoul in Appendix A. Appendix A.2 presents the details of RE-Search, including the number of search steps and termination conditions. Appendix A.3 outlines the algorithmic workflow of the proposed RAP. Finally, the prompt used for generation based confidence calculation is provided in Appendix A.4.
Appendix provides additional experimental results, including the experiment results on widely used benchmarks (Appendix B. with more powerful MLLMs (Appendix B.2), the impact of inference scaling (Appendix B.3), the influence of hyperparameters (Appendix B.4), and comprehensive comparison with search-based methods (Appendix B.5).
Appendix C provides a qualitative analysis of the proposed RAP and the current SOTA methods (Wang et al., 2025; Shen et al., 2024). Appendix C. presents qualitative analysis examples on the one-grained single-instance perception task, while Appendix C.2 illustrates examples on the fine- grained pross-instance perception task.
Appendix D presents and discusses the limitations of our RAP, providing directions for future research.
A. Additional Implement Details
Due to space constraints in the main paper, additional implementation details are provided in this section. In Appendix A.1 and Appendix A.2, we elaborate on the implementation of Spattal--Aarennss Layout and Retrieved-Exploviiee Search, respectively. Building upon these components, Appendix A.3 presents the complete algorithmic workflow of Retrieval- Augmented Perception (RAP).
A.1. Implement Details of Spatial-Awareness Layout
Given an input image I, it is first partitioned into smaller image crops based on a predefined crop size, which corresponds to the preferred resolution of the retriver. To ensure that the image size are divisible by the crop size and to prevent potential loss of visual information, padding is applied to the original image / necessary. Next, only non-zero image crops (referred to as valid image crops) are retained eliminate potential interference in the subsequent semantic similarity computation with the query. The retriever, specifically VisRAG (Yu et al., 2024) in this implementation, is then utilized to compute the cosine similarity between the user- provided query and each valid image crop. Based on the given K, the top K image crops with the highest similarity scores are selected. Subsequently, the rows and columns containing the selected crops are retained, forming a compressed matrix M'. Finally, using M', the corresponding image crops in I are mapped to construct a new transformed image l'
A.2. Implement Details of Retrieved-E -xplooratt-xloooatioo Search
In Retrieved -iteeveed-xlloratioon Search (RE-Search), a given HR image I is designated as the root node for the search process. The semantic similarity between I and the query is computed, along with an assessment of whether the image contains sufficient information for the MLLM to generate an appropriate response to the query. Subsequently, different proportions of image crops are retained based on the predefined retention ratio set P. Specifically, for each node, 25%, 50%, and75% of the image crops are preserved. The REward function is then applied to the retained image crops, and corresponding child nodes are created. These child nodes are added to the list of candidate nodes O for further exploration. Throughout the search process, the algorithm continuously tracks and maintains the optimal node identified thus far. The search process terminates if the current search step exceeds the predefined maximum search steps, which is set to 200 by default, or when the answering confidence e of the current node surpasses a specified threshold r. We set T 0.6 throughout the paper.
A.3. Complete of Algorithm Workflow
With the above notations and definitions in place, we provide the complete algorithm workflow in Algorithm 2.
13

---

Retricval-Augmented Perception: High-Resolltion Image Perception Meets Visual RAG
Algorithm 2 Relrieval-Augmeented Perception
Require: HR image I, Retriever R, Retention ratio P, Max steps max, Layout import Layoi from Algorithm
function REward( q, I, d)
V: (ui,..., J < Divide image into image crops
g - s(g. V)
h -Po(p (q),
2) f (1 w) g w.
4 (1-b).
b)-
return
end function
function Redrieval
{VI, ..., ) Divide image into image crops
Initialize V, +- 0
Initialize M2 +-
for idæ to [P] do
S - sq,, V)
V', M < V,,
V. + V. U [V']
M, + M, (M}
end for return Vs, M,. end function
f + RE.ard((.11.00 1,0) to + Node(u I = f, 0)
Initialize + {to}
hoptimal <- to S+0 /*Current step */
while O is not empty and S < max, do
Extract all confidence F <- of: 0)
idx + arg min(F)
4. + [
Remove f, from O
5 < 5
if ff > then return Loptimal-U
end if
** Retrieval*/
V. M2 & TedrievalSubNoode())
* Exploration*/
for ida to |Val do
I. Spatial (idr), M, [idz]) REwarddy, Ts, l.)
t.+ + Node(v = L,, fsr d t. .d + 1)
O x OU (1sti}
if t.+1.f > toptimal.f then
toptimal + ts+1
end if
end for end while
14

---

Retrieval-Augmentee Perception: High-Resolution Image Perception Meets Visual RAG
A.4. Prompt for Generation-basel Confidence Scores
To improve the applicability of our RAP method closed-source MLLMs, we investigate straightforward alternative that leverages generation-bast based confidence scores. Specifically, employ the prompt shown in Table 10 to prompt the MLLM to generate confidence score conditioned the given imput image and question.
Table 10. Prompt template generation-bast confidence scores. "[question]" placeholder meant be replaced with specific question from the dataset.
Prompt Template for Generation-base Confidence Score Question: [question]
Could you answer the question based on the available visual information? Return only a JSON object with a numerical confidence score (0 10) of "Yes" like {"Yes": x).
B. More Experiment Result
B.1. More Experiment Results on Widely used Benchmarks
We conduct additional experiments on five widely used benchmarks: Doc (Mathew et al., 2021), ChartQA (Masry et al., 2022), Te (Singh et al., 2019), A12D (Hiippala et al., 2021), and MMStar (Chen et al., 2024b). As shown in Table 11, incorporating RAP led to performance of 1.8% and 2.1% 1 on LLaVA-v1.5 7B and 13B, respectively. We also observed taht RAP brings more notable improvements on higher-resolution images. For example, on Voc, which has an average resolution of 1599 x 1241, RAP improved performance by 4.7% and 2.3% for LLaVA-v1.5 7B and 13B, respectively.
Table 11. Comparison of the RAP against the baseline MLLM on five widely used benchmarks.
DocVQA
ChartQA
TextVQA
A12D
MMStar
LLaVA-vI.5-7B
21.5
18.2
45.8
54.9
30.3
w/ RAP
26.2
18.5
46.8
55.1
33.1
LL
23.7
T8.5
49.0
60.2
32.8
w/ RAP
26.0
23.2
50.0
60.9
34.4
B.2. More Experiment Results with Powerful MILMS
To further demonstrate the effectiveness and generalizability of our RAP, we conduct experiments on HR-Bench using several advanced MILMs, including Oryx-1.5-7B, CogVL.--LLama3119:. Cambrian-8B and LLaVA-ov-72B. As shown in Table 12, our RAP consistently boosts performance across all models. These results underscore the robustness of our RAP across a wide range of model architectures, highlighting its potential as a universal enhancement for high-resolution image perception.
Table 12. Comparison of RAP with Advanced MLLMs HR-Bench.
HR-Bench 4K
IIR-Bench 8K
FSP
FCP
Avg.
FSP
FCP
Avg.
Oryx-1.5-7B (Liu et al., 2024d)
62.0
50.5
56.3
53.8
45.3
49.5
w/ RAP
80.8
52.5
66.6
77.3
45.2
61.3
VogVLM-LLaaaa--1 (Hong et al., 2024)
69.3
48.8
59.1
53.3
45.0
49.1
w/ RAP
82.5
48.3
65.4
76.3
43.7
60.0
Cambrian-8B (Tong et al., 2024)
45.5
45.5
45.5
30.8
45.0
37.9
w/ RAP
70.0
44.3
57.1
66.5
44.8
55.6
L VVV--72-72B (Li et aT. 2024a)
76.8
757.5
67.7
70.5
55.3
62.9
w/ RAP
91.5
60.0
75.8
89.3
54.5
71.9
15

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
B.3. RAP Performance and Inference Computation Scale
To analyze the performance changes with different search steps, plot the performance of RE-Search steps. We conduct experiments on HR-Bench 8K using LLaVA-v1.5 7B & 13B, LLaVA-ov-O.5B. To accurately analyze the relationship between search steps and performance, the search process. This forces the model to perform a fixed number of steps and selects the K' with the lowest cost as the final output. As shown in Figure 6, we observe that increasing the number of search steps improves the performance,
set
to prevent early termination due to threshold constraints during
LLaVA-ov-0.58
O LLaVA-v1.5-7B
LLaVA-v1.5-13B
85
45
65
Accuracy (%)
Accuracy
75
40
Accuracy
55
65
50
35
35
1
10
20
30
40
50
200
10
30
40
50
200
30
10
30
Step
Step
Step
(a) FSP
(b) FCP
(c) Average
Figure 6. Performance vs. RE-Search steps HR-Bench 8K. (a) Fine-grained Single-instance Perception (FSP); (b) Fine-grained
Cross-instance Perception (FCP); (c) Overall Performance.
especially on the FCP task. However, the gains are marginal for LLaVA-v1.5-7B but more pronounced for stronger models like LLaVA-ov-0.5B and LLaVA-v1.5-13B5 Our analysis reveals that the FCP task requires consideration of the spatial relationships between image crops and their spatial combinations, making capabilities result in more noticeable performance improvement with increased search steps.
B.4. Effect of bias b
In the RE-Search, we to balance the cost 9(-) and heuristic function h(-) in different depth. In Eq. 6, we use b the bias value control the influence of depth on u. A smaller b indicates greater influence of depth on w, while larger b reduces this influence.
LLaVA-ov-0,5B
LLaVA-v1.5-7B
LLaVA-v1.5-13B LLaVA.v1.5-13B
LLaVA-ov-0.5B w/ RAP -L RAP
82
78
76
Accuracy Sar (%6)
(%))
.58
Accuracy
56
72
54
70
37
50-
45
40
40
31
35
30
30
0.2
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
(a) FSP
(b) FCP
(c) Average
Figure 7. Impact of bias value b, illustrating how the accuracy changes when varying bias value b.
As shown Figure 7, our RAP not sensitive to the value of and it consistently outperforms the baseline across all
16

---

Retricval-Augmentee Perception: High-Resolution Image Perception Meets Visual RAG
configurations. Furthermore, we observe that smaller values of b lead to better results for all models. Therefore, ensure fair comparison, set b 0.2 by default.
B.5. Compared with Other HR Processing Methods
We compare our RAP with HR processing methods DC - and Zoom Eye. DC is training-free framework to enhance MLLM understanding of HR images by partitioning images, generating textual descriptions for image crops, and integrating them for improved perception. Zoom Eye, tree search algorithm, is designed to effectively navigate the hierarchical and visual structures of images to extract relevant information.
As shown in Table 13, compared with the baseline, all HR processing methods bring the average performance gains. Among all these methods, our RAP achieves the relatively better formance on most tasks. For instance, RAP achieve an accuracy of 73.8% and 72.3% on HR-Bench 4K and HR- BR--Bench 8K, respectively, using LLaVA-v1.5-7B. representing improvements of 6.0% and 6.8% compared Zoom Eye. These results can prove the superiority of our RAP.
Table 13. Performance comparison between RAP and other HR methods. We conduct experiments on V Bench and HR-Bench using LLaVA-v1.5 7B and 13B. The "A(") represents the performance gains of HR methods against the baselines.
V
Bench
HR-Bench 4K
HR-Bench 8K
Method
A(t)
Attrihute
Spatial
Overall
FSP
FCP
Overall
FSP
FCP
Overall
LLaVA-v1.5-7B
43.5
56.6
48.7
38.5
33.8
36.1
33.0
31.3
32.1
Dw
49.6
59.2
51.6
45.3
37.0
41.1
36.5
33.3
34.9
+2.5
-w/ Zoom Eye
83.5
82.9
83.3
67.8
38.8
53.3
65.5
36.0
50.8
+22.5
-w/ RAP
90.4
96.1
91.1
73.8
40.5
57.1
72.3
35.3
53.8
+27.0
LLaVA-v1.5-13B
41.7
55.3
47.1
45.2
41.3
43.3
37.5
38.0
37.8
-w/ D
54.8
56.6
57.3
52.0
51.0
51.5
40.0
41.0
40.5
+7.1
-w/ Zoom Eye
87.5
81.6
85.3
73.0
43.0
58.0
67.3
45.5
56.4
+23.9
-w/ RAP
89.6
90.8
89.8
74.3
46.0
60.1
76.5
42.0
59.3
+27.0
C. Case Study
C.1. Qualitative Examples of Fine-grained Single-instance Perception Task
Figure illustrates two instances where incorporating different HR processing methods (DC Zoom Eye and our RAP) on LLaVA-v1.5-13B In the first example, the critical information "08-26" in the image lies exactly at the boundary of two image crops. Zoom Eye retains only part of it, leading to the loss of critical information. In contrast, our RAP, leaveraging RE-Search, accrately preserves the critical information and provides correct response. In the second example, DC initially searches along an incorrect path, resulting in an erroneous final answer. contrast, our RAP method accurately retrieves the cup on the ground, thereby providing the correct answer.
Query: What the primary material the cup lying the ground?
Query: What the number written the sign visible the background image?
B 18-28
Zoom Eye
Our RAP
DC
Our RAP
Search result:
Retrieval resuit:
Search result:
Retrieval result:
Answer: 08-28
X
Answer: 08-26 df
Answer: Glass X
Answer: Cardboard
Figure 8. Qualitative examples of Fine-grained Single-instance Perception task. We conduct experiments on HR-Bench 4K using LLaVA-v1.5-132 with processing methods.
17

---

Retricval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
C.2. Qualitative Examples of Fine-grained Cross-instance Perception Task
Figure presents two examples demonstrating the performance of different HR processing methods (DC*, Zoom Eye, and RAP) applied LLa VLv..-11..-13.. In the first example, Zoom Eye fails to consider the spatial relationships between image crops, leading to incorrect search result and enrоneous response. In contrast, our RAP effectively preserves the relative positions between image crops, enabling the generation of a correct answer. In the second example, multiple image crops are required for accurate reasoning. However, DC retrieves only a single image crop based on the keyword "chair" from the query, resulting in an incorrect answer. contrast, our RAP accurately retains the critical image crops, thereby producing the correct answer.
Query: Where is the small stone cairn located relative waterfall?
Query: What direction the chairs facing in the image?
Eye Search result:
DC
Our RAP
Retrieval result:
Search result:
Retricval result:
Answer: To the left X
Answer: At the buttom right
Answer: Facing the window
Answer: Facing each other -
Figure Qualitative examples Fine-grained Cross-instance Perception task. We conduct experiments on HR-Bench 4K using LLaVA-v1.5 13B with HR processing methods.
D. Limitation Discussion
Admittedly, the proposed RAP has limitations, despite its promising performance on HR benchmarks. In particular, the proposed method relies on external retriever for plausible results. In our future work, will strive address this limitation by investigating the model's internal visual perception adaptively select key image crops.
18


Reviews:
[{"review_id": "HWwy7FPSBq", "reviewer": "Reviewer_h1HT", "summary": "The paper works on a key challenge in the area of MLLMs -- the perception of high-resolution (HR) images. Centering on this significant problem, this paper leverages RAG to enhance MLLM’s ability to perceive HR images. The paper first explores the impact of the layout and the number of retrieved image crops on the model’s perception capability and further proposes Retrieval-Augmented Perception (RAP) to improve MLLM’s perception of HR images. RAP retrieves image crops while preserving spatial relationships and dynamically select the optimal number of retrieved image crops via Retrieved-Exploration Search (RE-Search). To the best of my knowledge, this is the first study towards exploiting the benefits of visual RAG for the challenging HR perception tasks. Experiments achieve an improvement of 24% on average on HR benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate Reviewer h1HT's insightful comments and suggestions.\n\n> **Q1:** Replace confidence score calculation in RE-Search.\n\n**R:** The reviewer's point is well taken. We clarify that currently, APIs like OpenAI's [R1] provide the `logprobs` parameter, which returns the log probabilities of output tokens and can be used as confidence scores in *RE-Search*.\n\nAdmittedly, there are also some closed-source models that do not support `logprobs`. Here, to echo the reviewer's concern,  we explore a simple alternative approach using **generation-based confidence scores**. Specifically, we design a scoring prompt that asks the model to evaluate whether the given image contains sufficient information to answer the question. The constructed scoring prompt is as follows:\n\n```Question: {}\\nCould you answer the question based on the available visual information? Return only a JSON object with a numerical confidence score (0~10) of \"Yes\"  like {\"Yes\": x}.``` \n\nIn the implementation, we rescale the final score to the range $[0,1]$. We conduct additional experiments on HR-Bench using LLaVA-ov-0.5B. The experimental results are shown in the table below. Compared to the baseline, using *RAP* with generation-based confidence scores achieved an average improvement of 7.4%, demonstrating that generation-based confidence scores through the model can also lead to significant performance gains.\n\nThe corresponding results and discussions will be included in the revision.\n\n|                                            | HR-Bench 4K | HR-Bench 8K |\n| ------------------------------------------ | ----------- | ----------- |\n| LLaVA-ov-0.5B                              | 51.5        | 42.8        |\n| **LLaVA-ov-0.5B w/ *RAP* (logit-based confidence score)**  | **61.3**    | **63.5**    |\n| LLaVA-ov-0.5B w/ *RAP* (generation-based confidence score) | 57.0        | 52.1        |\n\n[R1] OpenAI API, https://platform.openai.com/docs/api-reference/completions/create\n\n> **Q2:** Experimental results on general benchmarks.\n\n**R:** We appreciate the reviewer's advice. As suggested, we extend our *RAP* to LLaVA-v1.5 7B and 13B, evaluating it on DocVQA, ChartQA, TextVQA, AI2D, and MMStar. These five benchmarks cover resolutions ranging from 390 to 1600. Across both benchmarks and model scales, *RAP* consistently yields notable performance gains, reinforcing the robustness and generalizability of our approach. We will include the corresponding results on these broader benchmarks, along with further analysis, in the revised version.\n\n|  | DocVQA   | ChartQA  | TextVQA  | AI2D | MMStar |\n| -- | -- | -- | -- | -- | -- |\n| LLaVA-v1.5-7B  | 21.5     | 18.2     | 45.8 | 54.9     | 30.3     |\n| **LLaVA-v1.5-7B w/ *RAP***     | **26.2** | **18.5** | **46.8** | **55.1** | **33.1** |\n| LLaVA-v1.5-13B | 23.7 | 18.5 | 49.0 | 60.2 | 32.8  |\n| **LLaVA-v1.5-13B w/ *RAP***     | **26.0** | **23.2** | **50.0** | **60.9** | **34.4** |\n\n\n> **Q3:** Why does the FCP task still underperform despite preserving spatial relationships?\n\n**R:**  We thank the reviewer for the comments. We would like to clarify that the experiments in Table 1 were conducted using a fixed value of $K$. Our findings indicate that **the choice of $K$ significantly affects final performance**—using a fixed $K$ leads to suboptimal results on ***FCP*** compared to the baseline. To address this, we propose *RE-Search*, a method for adaptively selecting an appropriate $K$. As shown in Table 4, with *RE-Search* , both ***FSP*** and ***FCP*** outperform the baseline, demonstrating the effectiveness of our approach.", "claims_and_evidence": "It looks to me that the paper mainly presents the following key claims:\n\n1.\tThis paper finds that applying RAG to MLLMs for HR image perception requires preserving the original spatial information of the retrieved image crops and that the number of retrieved image crops needed varies across different tasks. The specific experimental support is provided in Section 3 named Pilot Study.\n\n2.\tBased on the above findings, this paper proposes a training-free framework, RAP, which maintains the relative positional relationships between retrieved image crops through Spatial-Awareness Layout and dynamically selects the appropriate top-K image crops using RE-Search. The authors show plenty of results in Tables 2 and 3, which convincingly show that RAP significantly enhances MLLM’s perception of hr images.", "methods_and_evaluation_criteria": "The proposed method is suitable for the task. The experiments conducted on the HR-Bench and MME-RealWorld are reasonable, as to my knowledge, many SOTA related works in this area also used the same benchmarks to evaluate the effectiveness of their methods.\n\n[1] Wang et al. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. 2024.\n\n[2] Shen et al. ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration. 2024.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments validate the effectiveness of the proposed RAP method on both HR-Bench and MME-RealWorld. Although the authors claim to have evaluated on general benchmarks, the average resolution of MME-RealWorld is 2000×1500, which still falls under high resolution. It remains unclear whether the method is effective on more general MLLM benchmarks (e.g., MMStar).", "relation_to_broader_scientific_literature": "RAP provides a novel and efficient method for the HR image perception of MLLMs. The method leverages the core idea of RAG to enhance the MLLM’s ability to perceive HR images.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-structured and clearly explains the motivation and insights in an intuitive manner, tackling a highly significant problem. The authors start with a pilot study, proposing three questions and challenges, which then motivate the main methodology design. This logic makes it easy to grasp its main idea.\n2. The authors introduce RAP, which achieves superior performance on high-resolution benchmarks and is designed with the merits of easy integration into existing open-source MLLMs. I believe that this could potentially be of interest to a broad multimodal audience.\n\nWeaknesses:\n\n1. It looks to me that RAP relies on the model’s confidence scores in RE-Search, making it seemingly unsuitable for closed-source models. Are there other solutions that can replace the calculation of confidence scores in the RE-Search? Any more discussions on the alternative solution would be appreciated.\n\n2. Although the authors claim to have validated their method on general benchmarks for standard MLLM scenarios, they only conducted experiments on the general benchmark MME-RealWorld, which has an average resolution of 2000×1500. In fact, it should be noted that some existing related studies (e.g. (Wang et al., 2024b) in the paper ) not only conduct experiments on HR benchmarks but also evaluate on benchmarks with normal resolution images, thus validating the generalization of the method. There is, however, a lack of experimental evidence demonstrating its applicability to more common resolutions (e.g., MMStar).", "other_comments_or_suggestions": "Please address: \n\n1. Alternative computation of the confidence scores in RE-Search?\n\n2. More experiments on more common resolution to show its generalization capability. I may consider raising my score, if the authors could show the effectiveness of their method and provide additional experiments on general benchmarks like MMStar.", "questions_for_authors": "One more minor question is in Table 1, I wonder why the FCP task still performs worse than the baseline even after preserving the spatial relationships of image crops?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lQbDntArfI", "reviewer": "Reviewer_tave", "summary": "This paper introduces a retrieval-augmented perception method for MLLMs, which retrieve and fuses relevant image crops from the full high-resolution image.\nSpecifically, a apatial-awareness layout is proposed, which is to maintain the relative positional relationships of the image crops.\nIn addition, a retrieved-exploration search dynamically selects the optimal number of crops based on the text-to-crop similarity and the model's confidence.\nIn experiments, the proposed method outperforms others by a large margin in both fine-grained single-instance perception and fine-grained cross-instance perception tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate Reviewer tave's constructive comments and positive support.\n\n> **Q1:** The concern of the comparison methods are comprehensive enough.\n\n**R**: We'd like to thank the reviewer for the advice. In fact, building on established works such as $DC^2$ [R1] and ZoomEye [R2], we conduct experiments on $V^*$, HR-Bench, and the widely used benchmark MME-RealWorld. Our study includes a comprehensive comparison with **12 widely adopted MLLMs**, and we further demonstrate the efficiency of our *RAP* method against leading state-of-the-art approaches (see Table 5 in our paper).\n\nTo echo the reviewer's concern, here we further strengthened our contributions by including additional experiments:\n\n- We kindly refer the reviewer to our response to `Reviewer h1HT (Q2)` for additional experimental results on **five widely used benchmarks**: DocVQA, ChartQA, TextVQA, AI2D, and MMStar.\n- We respectfully refer the reviewer to our response to `Reviewer TfPd (Q2)` for additional evaluations involving **five state-of-the-art MLLMs**, including LLaVA-ov-7B/72B, Oryx-1.5-7B, Cambrian-8B, and CogVLM-Llama3-19B.\n\n[R1] Wang W, Ding L, Zeng M, et al. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models[C]. In AAAI 2025.\n\n[R2] Shen H, Zhao K, Zhao T, et al. ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration[J]. arXiv preprint arXiv:2411.16044, 2024.\n\n\n\n> **Q2:** The effect of crop size.\n\n**R:**  We appreciate the reviewer for the valuable advice. We'd like to clarify that we use the default crop size of $448\\times 448$ recommended by VisRAG in our *RAP*. To further investigate the impact of crop size, here we perform additional experiments on HR-Bench 8K using LLaVA-ov-0.5B and show the results below. Our findings reveal that while variations in crop size result in relatively minor differences, all configurations of our *RAP* method yield substantial performance gains over the baseline. \n\n|                             | HR-Bench 8K |\n| --------------------------- | ----------- |\n| LLaVA-ov-0.5B               | 42.8        |\n| LLaVA-ov-0.5B w/ *RAP* ($224\\times 224$)    | 61.4        |\n| **LLaVA-ov-0.5B w/ *RAP* ($448\\times448$)** | **63.5**    |\n| LLaVA-ov-0.5B w/ *RAP* ($896 \\times 896$)   | 60.0        |\n\n\n\n> **Q3:** An analysis of failure cases is missing.\n\n**R:** We'd like to thank the reviewer for pointing out this issue. Admittedly, while *RAP* demonstrates strong capability in accurately retrieving key visual information, its performance can degrade when the input question lacks critical object-specific details. A detailed failure case is available at the following anonymous link:\n\nhttps://anonymous.4open.science/r/RAP_Case-1D48/Failure_Case_Study.md\n\nIn our revised version, we will incorporate a comprehensive analysis of this case to further clarify the limitations and potential directions for improvement.", "claims_and_evidence": "In the spatial-awareness layout, the compressed mapping is reversable so the proposed method can handle cross-instance perception task.\n\nThe paper appears to have successfully addressed the computational complexity limitations of search-based methods by progressively narrow down the search space with the proposed RE-Search inspired by A* algorithm rather than a simple divide and conquer approach or a complex search algorithm, in MLLMs' HR image perceptual capabilities.", "methods_and_evaluation_criteria": "The proposed method is evaluated on V∗ Bench and HR-Bench, showing improved performance by large margin with efficiency.", "theoretical_claims": "The proposed RE-Search consists of two reward functions g and h inspired from A* search seems reasonable. \n- g: similarity between current patches and the query.\n- h: MLLM's confidence that the model can answer from given patches.", "experimental_designs_or_analyses": "I'm uncertain whether the comparison methods are comprehensive enough, because I am not familiar with the HR image preception in MLLMs, but the types and quantities of comparison targets appear to be suitable, according to the `Related Work` section written by the authors.\n\nThe experimental results in the appendix effectively complement the experimental results presented in the main text.\n\nHowever, the effect of crop size is not shown.", "relation_to_broader_scientific_literature": "I believe this research could provide important contributions not only for HR image processing but also for extending into video perception.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "Section3 `Pilot Study` helps readers understand the intuition behind the proposed method more easily.\nFor example, Table1 shows that preserving relative position information is necessary to improve both FSP and FCP tasks.", "other_comments_or_suggestions": "An analysis of failure cases is missing.\nIt is difficult to pick a specific case, but like all other methods the proposed method also have its pros and cons so it may have some shortcomings compared to existing methods. \nAnalyzing such cases will help for readers a deeper understanding of the proposed method.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGuJjUPqRX", "reviewer": "Reviewer_qqJL", "summary": "This paper proposes Retrieval-Augmented Perception (RAP), a training-free framework that enhances high-resolution image perception in multimodal large language models by leveraging RAG. RAP retrieves and fuses relevant image crops while maintaining their spatial relationships using a Spatial-Awareness Layout and dynamically determines the optimal number of retrieved crops through Retrieved-Exploration Search (RE-Search). Experiments show that RAP brings consistent improvement on HR benchmarks and general MLLM tasks.\n\n## update after rebuttal\nThanks for answering my questions. I will keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the thoughtful comments and suggestions, as well as the positive support.\n\n> **Q1:** The concern about the extra retriever increasing computation.\n\n**R:** We appreciate the reviewer's comments. Although our *RAP* includes an additional retriever module, it still delivers a significant performance gain despite the overhead. As shown in the table below, our evaluation on HR-Bench 8K demonstrates that integrating *RAP* with LLaVA-ov-0.5B, using VisRAG as the retriever, results in a total model size only 3.3 billion parameters. Despite this lightweight configuration, it achieves an impressive 20.7% improvement in accuracy. Notably, **LLaVA-ov-0.5B w/ *RAP* matches the performance of LLaVA-ov-72B while using approximately 22 times fewer parameters**, highlighting *RAP*'s efficiency, scalability, and strong potential for high-resolution visual understanding.\n\n|                 | Params | Throughput | Accuracy |\n| --------------- | ------ | ---------- | -------- |\n| LLaVA-ov-0.5B   | 0.5B   | **40.0**   | 42.8     |\n| LLaVA-ov-0.5B w/ ***RAP (VisRAG)*** | 3.3B   | 20.0       | **63.5** |\n| LLaVA-ov-72B    | 72B    | 8.7        | 62.9     |\n\n\n\n> **Q2:** Replace confidence score calculation in *RE-Search*.\n\n**R:** We appreciate the reviewer's point. To clarify, the OpenAI API [R1] currently offers a `logprobs` parameter, which returns the log probabilities of output tokens — an essential feature we leverage in our *RE-Search* module to compute confidence scores. \n\nBeyond this, we also explore an alternative approach using **generation-based confidence scoring**. Specifically, we prompt the MLLM to self-assess whether the image provides sufficient information to answer the question. The constructed prompt is as follows:\n\n```Question: {}\\nCould you answer the question based on the available visual information? Return only a JSON object with a numerical confidence score (0~10) of \"Yes\"  like {\"Yes\": x}.```\n\nWe normalize the final score to the range $ [0,1] $ for consistency. As shown in the table below, incorporating this generation-based score into *RAP* resulted in a **7.4%** performance gain over the baseline, demonstrating its effectiveness.\n\nThe relevant results and discussions will be incorporated into the revised version.\n\n|            | HR-Bench 4K | HR-Bench 8K |\n| ---------------------- | ----------- | ----------- |\n| LLaVA-ov-0.5B                              | 51.5        | 42.8        |\n| **LLaVA-ov-0.5B w/ *RAP* (logit-based confidence score)**  | **61.3**    | **63.5**    |\n| LLaVA-ov-0.5B w/ *RAP* (generation-based confidence score) | 57.0        | 52.1        |\n\n\n[R1] OpenAI API, https://platform.openai.com/docs/api-reference/completions/create\n\n\n\n> **Q3:** What is the fundamental difference between *RAP* and search-based methods?\n\n**R:** We thank the reviewer for the constructive comments. We would like to clarify the main differences between our *RAP* and search-based methods as follows:\n\n- **Search-based methods yield high-resolution inputs with multiple crops, while RAP lowers resolution by keeping only key spatially-aware crops** \n\n  Search-based methods model high-resolution images using a tree structure and apply search algorithms to identify key image crops. However, when a question requires information from multiple crops, methods like $DC^2$ tend to select the LCA (Lowest Common Ancestor), which is an image containing all relevant areas. This results in extremely high-resolution inputs. In contrast, our ***RAP*** employs a *Spatial-Awareness Layout* to selectively retain only the key image crops, effectively reducing image resolution while maintaining their spatial relationships (see Figure 4 in our paper).\n  \n- **Search-based methods typically follow a top-down strategy, while *RAP* starts directly from low-resolution image crops** \n\n  Search-based methods often treat the high-resolution image as the root node, splitting it into sub-images (such as a $2\\times 2$ grid) to generate child nodes. However, high-resolution root images can mislead the MLLM by providing incorrect search cues at the outset, resulting in inefficient paths and potential errors. Our ***RAP*** mitigates this issue by starting directly with low-resolution image crops, enabling more efficient and accurate retrieval of relevant content (see Table 5 in our paper).\n\n> **Q4:** Essential References Not Discussed.\n\n**R:** Thanks for pointing out the work on SV-RAG and MMed-RAG. SV-RAG applies RAG to multimodal long-document understanding, while MMed-RAG enhances the factuality of Med-MLLMs. In our revised version, we will include these two works in the discussion of Multimodal RAG in the Related Work section.", "claims_and_evidence": "The claims are supported by clear and convincing evidence according to experimental results.", "methods_and_evaluation_criteria": "The proposed method enhances the perception of high-resolution images by MLLMs through retrieval augmentation, without requiring additional training. As a plug-in, it can theoretically be applied to any MLLMs. This is meaningful for the current MLLM's perception of high-resolution images. However, it may incur additional inference overhead.", "theoretical_claims": "The paper does not present particularly complex techniques, with its effectiveness mainly demonstrated through experimental results. The ablation study in Table 4 and Figure 4 effectively emphasizes the validity of the method's design.", "experimental_designs_or_analyses": "The experiments are conducted on high-resolution benchmarks and general benchmarks, with LLaVA-v1.5 7B & 13B and LLaVA-ov-0.5B as baselines for comparison. The metrics include accuracy and throughput. The experiments confirm the validity and effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "The method seems to be similar to RAG applied to LLMs [a], enhancing the ability of LLMs to perceive long-context in NLP.\n\n[a] Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG", "essential_references_not_discussed": "Although it can be considered as concurrent work, the authors are encouraged to include the latest papers [a, b] from ICLR2025 in the related work section.\n[a] SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding\n[b] MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models", "other_strengths_and_weaknesses": "Strengths:\n+ The paper explores applying RAG to MLLMs for the perception of high-resolution images. Through comprehensive experiments, it investigates the impact of the layout of retrieved image crops and the number of retrieved image crops on the final performance.\n+ This paper contributes to the ability of MLLMs to perceive high-resolution images.\n+ Experimental results show the effectiveness of the proposed methods.\n\nWeaknesses:\n- The method requires the introduction of an additional retriever, which consumes certain computational resources.\n- Since the method relies on the model's confidence score regarding whether currently retrieved image crops can answer the question, it cannot be directly applied to closed-source models.", "other_comments_or_suggestions": "Please consider to add citations to the papers mentioned above.", "questions_for_authors": "Although the authors emphasize that RAP is a RAG-based method, it also uses search techniques to enhance MLLMs' perception of high-resolution images. What is the fundamental difference between RAP and search-based methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9ugbdi3TD", "reviewer": "Reviewer_TfPd", "summary": "- The paper introduces a RAG-based approach to handle High-Resolution visual reasoning with MLLMs.\n- The authors develop a Retrieval-Augmented Perception (RAP) framework to find the important image crops required for answering a given query based on the query-crop similarity and using those for inference.\nRAP is an iterative process with two stages: (I) RE-Search (to find the optimal number of crops using confidence scores about the cropped image to answer the query from multiple MLLM feed-forward passes) and (ii) Spatial-Awareness Layout (to lay out the most similar crops while maintaining their relative positioning from the original image).\n- The authors present an extensive analysis of factors affecting performance, including the number of retrieved image crops and their layout.\n- The authors test their framework on the MME-Realworld, V*, and HR-Bench benchmarks and show improvements with different base MLLMs from the LLaVA series.\n\n## update after rebuttal\n\nI am raising my score to accept based on the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments and suggestions.\n\n> **Q1:** Experiments on widely used benchmarks.\n\n**R:** We appreciate the reviewer's suggestion. As suggested, we conduct additional experiments on five widely used benchmarks: **DocVQA, ChartQA, TextVQA, AI2D, and MMStar**. As shown in the table below, incorporating *RAP* led to performance of 1.8% and 2.1% on LLaVA-v1.5 7B and 13B, respectively. We also observed that *RAP* brings more notable improvements on higher-resolution images. For example, on DocVQA, which has an average resolution of $1599\\times 1241$, *RAP* improved performance by 4.7% and 2.3% for LLaVA-v1.5 7B and 13B, respectively.\n\n|  | DocVQA   | ChartQA  | TextVQA  | AI2D | MMStar |\n| -- | -- | -- | -- | -- | -- |\n| LLaVA-v1.5-7B  | 21.5     | 18.2     | 45.8 | 54.9     | 30.3     |\n| **LLaVA-v1.5-7B w/ *RAP***     | **26.2** | **18.5** | **46.8** | **55.1** | **33.1** |\n| LLaVA-v1.5-13B | 23.7 | 18.5 | 49.0 | 60.2 | 32.8  |\n| **LLaVA-v1.5-13B w/ *RAP***     | **26.0** | **23.2** | **50.0** | **60.9** | **34.4** |\n\n> **Q2:** Experiments with more powerful MLLMs. \n\n**R:** We thank the reviewer for the constructive advice. To further demonstrate the effectiveness and generalizability of our *RAP*, we conduct experiments on HR-Bench using several advanced MLLMs, including **Oryx-1.5-7B, CogVLM-LLama3-19B, Cambrian-8B, LLaVA-ov-7B and 72B**. As shown in the table below, our *RAP* consistently boosts performance across all models. These results underscore the robustness of our *RAP* across a wide range of model architectures, highlighting its potential as a universal enhancement for high-resolution image perception. We will include the corresponding results and discussions in the revised version.\n\n| | HR-Bench 4K | HR-Bench 8K |\n| -- | -- | -- |\n| Oryx-1.5-7B | 56.3 | 49.5  |\n| **Oryx-1.5-7B w/ *RAP***   | **66.6**  | **61.3** |\n| CogVLM-LLama3-19B | 59.1  | 49.1 |\n| **CogVLM-LLama3-19B w/ *RAP*** | **65.4**    | **60.0** |\n| Cambrian-8B | 45.5 | 37.9 |\n| **Cambrian-8B w/ *RAP***  | **57.1**    | **55.6** |\n| LLaVA-ov-7B  | 63.0 | 49.3 |\n| **LLaVA-ov-7B w/ *RAP*** | **70.3** | **63.8** |\n| LLaVA-ov-72B | 67.1  | 62.9 |\n| **LLaVA-ov-72B w/ *RAP***  | **75.8**  | **71.9** |\n\n> **Q3:** Comparison with native high-res MLLMs.\n\n**R:** We appreciate the reviewer for the comments. We'd like to clarify that due to limitations in visual encoders and LLMs' long-context handling, current MLLMs struggle with high-resolution (HR) image perception. Existing native high-res MLLMs mainly address this through two strategies:\n\n- **Splitting HR images into multiple image crops**\n\n  These methods split HR images into smaller crops, encode them separately, and concatenate the features for the LLM. Oryx uses OryxViT to handle arbitrary resolutions, but processing 8K images produces up to 300K visual tokens, **heavily taxing the LLM's long-context capacity**. In practice, this often led to erroneous outputs, so we downsampled to 2K resolution (as recommended), which unfortunately resulted in loss of important visual details.\n\n- **Using HR visual encoder**\n\n  Another line of work explores HR visual encoders designed to handle HR images. For instance, ConvNeXt-L supports inputs up to $768\\times 768$ resolution. However, such encoders still **downsample 8K images to align with pretraining resolutions, limiting their effectiveness on HR images**. Even Cambrian, a multi-encoder framework, only supports resolutions up to $384\\times 384$, necessitating downsampling for 8K inputs and resulting in significant loss of critical visual information.\n\nInspired by RAG's success in improving long-context capabilities in LLMs, we adapt it to MLLMs for HR image perception. *RAP* retrieves and preserves only the most relevant image crops, reducing input resolution while retaining essential information. Our *RAP* can be integrated into any MLLMs, significantly enhancing their perception of HR images. For example, LLaVA-v1.5 7B with *RAP* reaches 53.8% accuracy on HR-Bench 8K, outperforming Oryx-1.5-7B (49.5%) and Cambrian (37.9%), validating *RAP*'s effectiveness.\n\n> **Q4:** The practical usefulness of our *RAP*.\n\n**R:** We'd like to thank the reviewer for the comments. Here, to robustly demonstrate the practical usefulness of our *RAP*, we evaluate its impact on both throughput and accuracy. As shown in the table below, LLaVA-ov-0.5B equipped with *RAP* achieves performance comparable to the much larger LLaVA-ov-72B, yet operates with **2.3 times higher throughput and requires 22 times fewer parameters**. These results highlight *RAP*'s potential for efficient, high-performance deployment, particularly in resource-constrained environments such as edge devices. \n\n|  | Params | Throughput | Accuracy |\n| -- | - | -- | -- |\n| LLaVA-ov-0.5B   | 0.5B   | **40.0**   | 42.8     |\n|  **LLaVA-ov-0.5B w/ *RAP (VisRAG)*** | 3.3B   | 20.0  | **63.5** |\n| LLaVA-ov-72B    | 72B    | 8.7  | 62.9 |", "claims_and_evidence": "- The authors claim RAP improves performance on high-resolution benchmarks over standard MLLMs built with fixed-resolution visual encoders, which are supported by experiments.\n- The authors claim RAP is more efficient than other search-based retrieval methods, which is also well supported.", "methods_and_evaluation_criteria": "- The authors evaluate on high res benchmarks which is aligned with their RAG-based framework design motivation", "theoretical_claims": "- There are no theoretical claims made in the paper, IMO.", "experimental_designs_or_analyses": "- The authors analyze the different components of their framework:\n   - Their SL and RE-Search techniques bring on improvements along with the VisRAG method.\n- The authors use the LLaVA-1.5-7B/13B and LLaVA-OV-0.5B models to test their method and show improvements. However, the mentioned models are not the best, and the community rarely uses these for their applications. To show the practical effectiveness of the approach, the authors should consider testing it with more powerful MLLMs like LLaVA-OV-7B/72B and Cambrian-8B/34B.\n - Related to the above point, Cambrian is a multi-encoder framework that uses spatial compression on CLIP-ConvNeXT features, which are better for high-res images, so this is an important experiment, IMO.", "relation_to_broader_scientific_literature": "- High-resolution visual reasoning is an important problem and highly relevant to the current literature as also reflected in the design of benchmarks like V* and HR-Bench.", "essential_references_not_discussed": "Nope", "other_strengths_and_weaknesses": "- Did the authors test their framework on other widely benchmarks like DocVQA, ChartQA, TextVQA, AI2D, or any ocr/document-related tasks?\n- The authors should compare their results to a native high-res MLLM like Oryx-MLLM (https://github.com/Oryx-mllm/Oryx).\n- The authors show CogVLM has high performance in Fig. 1, but they don't include it in Tab.2 or share results with RAP with CogVLM as the base MLLM. why? I believe a more thorough evaluation with different benchmarks/models is important to show the complete effectiveness of the method.\n- The authors do not report the throughput for the original LLaVA-1.5 in Tab. 5 and add other better general MLLMs like LLaVA-ov-72B/CogVLM/Oryx-MLLM to the mix for a better understanding of RAP's practical usefulness.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenge of enabling Multimodal Large Language Models (MLLMs) to effectively perceive high-resolution images. The authors propose Retrieval-Augmented Perception (RAP), a training-free framework that adapts principles from Retrieval-Augmented Generation (RAG). RAP operates by retrieving relevant image crops based on a textual query, fusing them while preserving spatial context via a proposed Spatial-Awareness Layout, and dynamically selecting the optimal number of crops using a Retrieved-Exploration Search (RE-Search) mechanism based on model confidence and retrieval scores.

Experimental results demonstrate substantial performance improvements on established high-resolution benchmarks (e.g., V* Bench, HR-Bench) when applying RAP to existing MLLMs. The paper effectively motivates the RAP design through initial pilot studies and presents the methodology clearly. The overall assessment indicates that RAP is a novel and practical framework offering a valuable contribution to high-resolution visual understanding in MLLMs.

Main Weaknesses: The RE-Search component relies on accessing model confidence scores (typically derived from logits), limiting its direct applicability to closed-source models that do not provide such access. Reviewers raised concerns regarding the initial scope of experiments, focusing primarily on specific high-resolution benchmarks and a limited set of base MLLMs. However, the authors provided substantial additional experiments during the rebuttal phase addressing these concerns by evaluating on broader MLLM benchmarks and more diverse, state-of-the-art models. A detailed comparison and differentiation from related search-based methods and native high-resolution MLLM architectures needed further clarification.

Overall, the strengths outweigh the weaknesses, and the reviewers are largely positive, so I recommend acceptance. 

There are some clear areas for improvement though:
- Provide a thorough analysis and validation of the proposed generation-based confidence scoring mechanism as an alternative for closed-source models. This should include its correlation with logit-based scores and robustness across different prompt formulations.
- Incorporate the extensive supplementary experiments conducted during the rebuttal (evaluations on additional benchmarks like DocVQA/MMStar, results with models like Oryx/Cambrian/CogVLM, failure case analysis, crop size ablations, and refined efficiency metrics) into the main paper to fully demonstrate the method's robustness, generalizability, and practical trade-offs.
- It would also be interesting to check whether the retrieval is indeed enough to answer the questions (e.g., building on something like Sufficient Context: A New Lens on Retrieval Augmented Generation Systems, https://openreview.net/forum?id=Jjr2Odj8DJ). In the MLLM setting, this is still a very important consideration.
Decision: Accept (oral)