VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning
Authors: Qiuchen Wang
Ruixue Ding
Yu Zeng
Zehui Chen
Lin Chen
Shihang Wang
Pengjun Xie
Fei Huang
Feng Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Image Understanding
Retrieval Augmented Generation
Vision-language Models
Primary area: applications

Abstract: Effectively retrieving, reasoning and understanding visually rich information remains a challenge for traditional Retrieval-Augmented Generation (RAG) methods. On the one hand, traditional text-based methods cannot handle visual-related information. On the other hand, current vision-based RAG approaches are often limited by fixed pipelines and frequently struggle to reason effectively due to the insufficient activation of the fundamental capabilities of models. As reinforcement learning (RL) has been proven to be beneficial for model reasoning, we introduce VRAG-RL, a novel RL framework tailored for complex reasoning across visually rich information. With this framework, VLMs interact with search engines, autonomously sampling single-turn or multi-turn reasoning trajectories with the help of visual perception tokens and undergoing continual optimization based on these samples. Our approach highlights key limitations of RL in RAG domains: (i) Prior Multi-modal RAG approaches tend to merely incorporate images into the context, leading to insufficient reasoning token allocation and neglecting visual-specific perception; and (ii) When models interact with search engines, their queries often fail to retrieve relevant information due to the inability to articulate requirements, thereby leading to suboptimal performance. To address these challenges, we define an action space tailored for visually rich inputs, with actions including cropping and scaling, allowing the model to gather information from a coarse-to-fine perspective. Furthermore, to bridge the gap between users' original inquiries and the retriever, we employ a simple yet effective reward that integrates query rewriting and retrieval performance with a model-based reward. Our VRAG-RL optimizes VLMs for RAG tasks using specially designed RL strategies, aligning the model with real-world applications. Extensive experiments on diverse and challenging benchmarks show that our VRAG-RL outperforms existing methods by 20\% (Qwen2.5-VL-7B) and 30\% (Qwen2.5-VL-3B), demonstrating the effectiveness of our approach. The code is available at https://github.com/Alibaba-NLP/VRAG.

VRAG-RL: Empower Vision-Pereeption-Basee RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning
Qiuchen Wang', Ruixue Ding, Yu Zeng', Zehui Chen', Lin Chen', Shihang Wang, Pengjun Xie, Fei Huang, Feng Zhao?
'MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC qiuchenwang @ matl.ustc.edu.cn (aaao956@utt.edu...
Abstract
Effectively retrieving, reasoning and understanding visually rich information re- mains a challenge for traditional Retrieval- Auggmeted Generation (RAG) methods. On the one hand, traditional text-based methods cannot handle visual-related in- formation. On the other hand, current vision-based RAG approaches are often limited by fixed pipelines and frequently struggle to reason effectively due to the insufficient activation of the fundamental capabilities of models. As reinforcement learning (RL) has been proven to be beneficial for model reasoning, we introduce VRAG-RL, a novel RL framework tailored for complex reasoning across visually rich information. With this framework, VLMs interact with search engines, au- tonomously sampling single-turn or multi-turn reasoning trajectories with the help of visual perception tokens and undergoing continual optimization based on these samples. Our approach highlights key limitations RL in RAG domains: (i) Prior Multi-modal RAG approaches tend to merely incorporate images into the context, leading to insufficient reasoning token allocation and neglecting visual-specific perception; and (ii) When models interact with search engines, their queries often fail to retrieve relevant information due to the inability to articulate requirements, thereby leading to suboptimal performance. To address these challenges, define an action space tailored for visually rich inputs, with actions including cropping and scaling, allowing the model to gather information from coarse-to-fine perspective. Furthermore, to bridge the gap between users' original inquiries and the retriever, employ simple yet effective reward that integrates query rewriting and retrieval performance with a model-based reward. Our VRAG-RL optimizes VLMs for RAG tasks using specially designed RL strategies, aligning the model with real- world applications. Extensive experiments on diverse and challenging benchmarks show that our VRAG-RL outperforms existing methods by 20% (Qwen2.5-VL) V-V-7B and 30% (Qwen2.5-VL-3B). demonstrating the effectiveness of our approach. The code is available at http:://ggtuubbcommlllaaaaaaaaaa
Introduction
enables Language Models (LMs) to leverage external information to tackle problems. Due to the limitations of traditional textual RAG methods in handling visually rich information efforts have been made to introduce RAG into the visual domain by integrating Vision- Languag Models (VLMs) [ 7 33 15 34 with search engines. However, current visual RAG methods still fall short in effectively reasoning with search engines and
Retrieval-Augmenter Generation (RAG) 12 18
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Reinforcement Learning Driven Visual Iterative Reasoning Frumework
Update Policy Model
Interaction
Rollout Module:
environment
Trja.
Refenence Model
Tdd.
Instructione & Query
Hi.
Adn. Computiition
Search Engine
f3
Pollcy Medel (VLMs)
Ho.
Reward Fun: (RA6 Specific)
Visuel Perception Fanction
True Yau. Tixe
(b) Visual Perception Action Space
(c) Fine-grained Retrleval-based Reward
Shocked retrieval results from Round to N
Bagion of Drteneat
Tegjetta Hi.b.
Antwer Ervurtie
Bulr-ooodd
Il baced Evilrttie
dalee
bottem and Achion
Oktcome jodaeentt Fase
Pe
Exilate retrieed
Unniined
Crup & Zoen
Pre-defined Vinael Perception Actiona
Pattern Reward
Model-bored Reward
Retrieval-basee Reword
Figure I: Overall Framework of our Reinforcement Learning Framework, (a) demonstrates the interaction process between the model and the external environment, as well the implementation of the GRPO algorithm. (b) shows the proposed visual perception action space which allows the model extract information from coarse-to-fine perspective. (c) is the specially designed reward for RAG, which combines outcome and retrieval performance across the entire sampling process.
understanding complex visual information. Reinforcement Learning (RL) has been recognized an
effective approach for optimizing VLMs complex reasoning tasks [39. 20 14 30 49]. Therefore, RL offers promising approach address the challenges faced by visual RAG methods.
Inspired by these advancements, we introduce VRAG-RL, a novel multimodal RL framework specifically designed for iterative reasoning in visually rich information RAG. Our approach is based on three critical observations: (i) Insufficient activation of reasoning capabilities with visual information. Existing methods underutilize the reasoning potential of VLMs when incorporating visual information. For instance, prior approaches tend to merely embed images into the context without adequately addressing visual-specific perception processes, resulting in insufficient reasoning token allocation and limiting the models' ability to fully leverage visual data for complex reasoning tasks. (ii) Inefficient and disjointed Retrieval. In previous work, limited by the inability to articulate complex requirements, models struggled to retrieve relevant information efficiently, which may lead to repetitive and meaningless interactions, restricting the overall effectiveness. (iii) Inconsistent multi-turn reasoning and unstable training with VLMs. Current RL frameworks for LMs often struggle with maintaining stability and consistency during multi-tum reasoning. Handling complex, multi-step reasoning tasks can be particularly challenging, as models may encounter difficulties in maintaining effective reasoning across interactions with external environments, leading inconsistent performance and suboptimal results. This challenge is further exacerbated for VLMs, which are limited by their cestruction-folloiing and reasoning capabilities.
Building upon these insights, VRAG-RL introduces improvements in various modules: (i) We propose visual perception action space that includes selecting regions of interest and zooming into these areas. VLMs with visual perception tokens in the action space are capable of acquiring information from coarse-to-fine perspective. As shown in Figure [ (b), when dealing with images or charts within documents, VLMs can give higher attention to information-dense areas through the proposed perception tokens. This allows the model more effectively activate reasoning abilities within a limited context length, preventing the overlooking of details. (ii) Furthermore, rather than relying solely on a simple outcome-basse reward, we factor in the effectiveness of the retrieval process as part of the reward structure. In particular, during the interaction between the model and the search engine, retrieving pertinent images promptly enhances the model's ability to address questions effectively, whereas persistently retrieving irrelevant documents adds noise and hampers the reasoning process. As illustrated in (c). by integrating retrieval performance into reward, we establish comprehensive guidance for retreval- augmented generation frameworks. (iii) Inspired by the current
approach and the ReAct paradigm, we model the interaction between the VLMS and the search engine, along with the visual perception action space, as a process of iterative reasoning and tool invocation. Figure (())re((a) illustrates our training pipeline, which supports automatic sampling and integrates the GRPO algorithm. To ensure stability in multi-turn sampling

---

and training, we have carefully designed the sampling strategy including post-pre processing for each interaction, and model- based reward together with the retrieval reward mentioned above guides the model training. Additionally, we have re-annotated existing datasets of visually rich documents and developed a data construction pipeline to efficiently scale data for RI. and SFT.
Our major contributions are as follows:
We propose VRAG-RL, novel reinforcement learning framework tailored for training VLMs to effectively reason, retrieve, and understand visually rich information.
We define visual perception action space that includes selecting, cropping, and scaling regions of interest, allowing VLMs to gather information progressively from coarse- -raained to fine-grained levels. This action space enhances the models' ability to focus on information- dense areas and activates their vision-specific reasoning capabilities more effectively.
We introduce a comprehensive reward structure that integrates retrieval performance and model-based outcome reward. This reward mechanism aligns the model more closely with real-world applications, bridging the gap between users' original intentions and the retriever.
Extensive experiments demonstrate the effectiveness of our method. VRAG-RL VRAG-RLLsggafficantt outperforms strong baselines, achieving over 20% improvement on various benchmarks.
2 VRAG-RL
In this section, drawing on insights and foundational ideas, we present comprehensive description of our VRAG-RI. framework. We start with the formulation of the problem ($2. then introduce the action space designed for visual perception ($ 2.2 and the fine-grained reward specifically defined for the RAG task ($2.3) Finally, we illustrate the model interaction process in the rollout module and the reinforcement learning training implementation of our framework ($2.4).
2.1 Problem Formulation
Given query denoted as q, we have a huge collection of images C = (II I2..... Ix). consisting of N images. Each image contains variety of visually rich elements, such flowcharts, charts, tables, and diverse layouts, derived from real-world documents across multiple domains, including slides and reports. Our goal is to efficiently reason, accurately retrieve the most relevant images, extract valuable information from the complex visual data, and generate the final answer a to the query q.
2.2 Visual Perception Action Integration for Understanding Information-Dent Regions
Previous works merely involved migrating textual RAG to the multi-modal domain, which simply meant inserting images into the context and then reasoning and responding. However, these efforts overlooked the characteristics of image data, where the efficiency of visual perception is closely related image resolution, visual element layouts, information density, and other visually related factors. Motivated by these findings, we introduce dynamic novel visual perception paradigm into VLMs that involves region selection and re-encoding at the token level, as illustrated Figure 2
Definition of Visual Perception Actions. We define the visual perception action space for VLMs by taking into account the specific characteristics of visual information. This enables the model to select regions with high information density or regions relevant to the query for a detailed view, acquiring information from coarse fine perspective. We integrate search queries, answer summaries, and visually specific actions into unified action space to align with the model's pre-training domain.
The policy model To interacts with the environment in the Thought-Action- Obbtrvvttossottoionn (T, O) paradigm. In each interaction, the model generates the next action Ar T (- H+-1) based on the trajectory H.-1 from step ( 1 and earlier. A role-based function is used extract visual perception tokens <region> and </region>, whose main purpose is to select, crop, and zoom in on the region of interest of the image that has already been retrieved in the context:
A x O O,, k E (1.2.....t 1}.
Given w x h image as an observation O₁, a bounding box [Zrmin: Mmins Tmazy Ymnas] within percep- tion tokens can precisely delineate the position of region R. where (Imins Umin) and (Imazs Ymax)
3

---

(a) Truditional RAG with search action only
(b) Our VRRAG-RL with unified visual perception action space
Language Model Decoder
Language Model Decoder
(T, AQ
(T2, A2)
(Tx, Ak)
(Ti, AD
Ol: Retrieved Image
(T², A2)
(Tk. Aa)
Vision Encoder
Vision Encoder
Action Extraction
edlact
re-irput
Mapping
Retrieved
Retrieved
Grop
Image
Image
Region Selection
Inefficient parception and Repetitive netrievol
perception
Figure 2: Comparison between our VRAG-RL and the traditional RAG terms of perception methods. (a) Traditional methods lack effective perception, which easily leads to repetitive and ineffective retrieval calls and suboptimal outcomes. (b) Our VRAG-RL efficient and accurate, enabling the model to perceive information-denss regions from coarse-to-fine perspective.
represent the coordinates of the top-left and bottom-riihh pixels of region R. Some current models' pre-training domains for grounding tasks normalize the coordinates [0, resulting in actual coordinates of (I: x 4. y 4), while other models, such as Qwen2.5VL, directly the original coordinates without normalization. Then we will map the selected region R from the image tokens in context to the Wrow X hrew raw image, and crop this raw image to obtain R:
R = Crop((rrrrar [Imin
Wraw
-ymin
hraw
Fmaz
Wraw Wencoder
ymax hencoder
Braw 1). (2)
Wencoder
hencoder
where (Wraw, hyaw) are the shape of the original image Iraws (Weneoder, hencoder) are determined by the vision encoder such that Wencoder hencoder Pizelsmar- Finally, R is integrated into the context observation: R O.. Actually, the image token embedded in the context does not represent the original size of the image. The maximum pixel size Pixelsmaz for the vision encoder is often considerably smaller than the pixel of visually rich documents found in real-world applications. This is the reason why the region cropped from the original image and scaled within the vision encoder has higher density of vision tokens. This simple yet effective "crop and re-input" strategy enhances visual perception performance by directly increasing perceptual resolution 50 26 37].
Trajectory Data Scaling-Up Based on Multi-Expert Sampling. To effectively train the model, especially smaller-scale models, to learn the utilization of Visual Perception Tokens while retaining their foundational capabilities, we need train them with high-quality data through Supervised Fine- Tuning before applying RL. We propose multi-exper sampling strategy to scale up the trajectory data, aiming to sample diverse interactions within the same reasoning trajectory for each data.
The core idea is to utilize large-scale models 7M to effectively guide the reasoning process and tool selections within trajectory, while smaller expert models TEM annotate coordinate under the guidance of large-scale models. At the 4 interaction between the model and the environment:
H4 A1,
O0-1, Ar, O.),
where H. is the trajectory, representing the sequence of past observations and actions leading up to the current step. The TLM equipped with extensive capacities for understanding and processing complex multi-modes interactions, act pioneers in determining the overarching reasoning pathway:
[T. = TLM (-1 H4-1),
We use a rule-based function to extract action and thought. the action is search, the engine returns the original image as O, Otherwise, each time visual perception token is output, we employ couunding-speciffe expert models to re-locate the coordinates of regions of interest:
A = T Hz-1; Tr),
(5)
where the expert models म FEM benefit from the guidance provided by the large model's thought Tr, leveraging these insights enhance their precision in region localization. The newly generated

---

coordinates of the region of interest Ar will replace the old visual perception tokens A, generated by RLM, and the re-encoded image serves observation Oj:
= Pv
(6)
where Py represents the visual processing function, the selected region will undergo cropping. zooming in, and re-encoding before being inserted into the context.
2.3 Fine-Grained Reward Function Tailored for Enhancing RAG Framework
Unlike traditional RL methods that focus only on output re- sults, VRAG-RL emphasizes optimizing retrieval in RAG, retrieval quality directly affects overall performance. We designed a reward function with three components:
70
63.5
54.1
53.3
501
50
pattern reward, retrieval efficiency reward, and model- based outcome reward, guiding the model to efficiently retrieve information and generate high-quality answers.
42.4
40
Inference w/ Noise Upper Bound
30
of Reference Images
Retrieval Efficiency Reward. As shown in Figure 3 Figure 3: Experiments on the impact of context length on model performance.
when the information is sufficient, excessively long context can interfere with the model. Therefore, the earlier and more comprehensive the retrieval of relevant information, the better the model can construct a coherent and informative context for generating high-quality answers. Inspired by Normalized Discounted Cumulative Gain, and using our predefined relevance of the recalled images, we define:
[Dar
2
1, if d, Dret 0, if d, & Dreet
DCG() = Σ
log (i + 1) 8, =
where d; E Dirj represents stacked retrieved images within the trajectory, Dret is the collection of relevant golden images, 8; the predefined relevance score. We believe that the performance is optimal when all relevant documents retrieved first, the Ideal-DCG is defined as:
[Dral
2ad 1 iDaal Σ
IDCGGDree)= Σ
2d
(8)
loga ++11
(i + 1)
loga(i (
i= i=|Daal++
where Stel = and Synrel = 0 respectively represent the relevance scores of ideally relevant and irrelevant documents. Our Retrieval Efficiency Reward is defined as:
DDG Drat)
T Ret
(9)
IDCG(Dreet
where TRet. the modified NDCG, is directly used as the reward reflect retrieval performance.
Pattern Consistency and Hodel-Based Outcome Rewards. The rule-based pattern reward is designed to encourage the model to follow the reasoning patterns during the interaction process:
Tpa Parse(H),
(10)
where is the generated trajectory. Parse(.) employ action tokens <search> and </search> to extract predefined actions in the action space. This is crucial for reasoning agent with predefined action space, as it helps effectively extract actions and thoughts. Regarding outcome reward, unlike rule-based methods that are prone to falling into local optima, adopt a model-based reward:
Tans ~ TRM (RRM Agolden- Apred).
(11)
where Q represents the input query, Agolenn is the reference golden answer, and Apreat is the answer generated by the VLMss. Based on these inputs, the evaluation model ArM assesses the correctness of the final answer. Please refer to Appendix for the detailed prompt used in the model-based reward.
5

---

Integrated Reward Function. The final reward function is a weighted combination of the three components described above, with weights used balance the contributions of each component:
= -Reet -Ans + 7 TPtt.
(12)
where a + (3 + 1. In practice, we usually set y
the model can effectively learn the pattern after SFT. We set ? = 0.1 when performing RI with cold start to help the model learn the predefined pattern. By integrating these three components into the reward function, our VRAG-RL provides a comprehensive and fine-grained evaluation mechanism that guides the model in optimizing its reasoning and retrieval capabilities in a way that aligns closely with real-world applications.
2.4 Reinforcement Learning Framework with Iterative Reasoning
We apply RL to multimodal RAG agent tasks to enhance the capability of VLMS in retrieving and reasoning. Our RL framework is primarily divided into two parts for discussion: the rollout process for multimodal agent and the reinforcement learning training strategy for multi-turn interactions.
Multi-Round Generation with Search Engine and Visual Perception Actions. As shown in Al- gorithm [ the model interacts with the external environment in multiple turns, where the observation, which is the image, is inserted into the trajectory in the role of the user. This is necessary to align with the model's s pre-training domain, where only the user token can insert image tokens.
Algorithm 1 Interaction of VLM with the External Environment through Iterative Reasoning
Input: Input query x, Policy model Te, External environment V. Maximum iterations T.
Output: Final trajectory y-
I: Initialize rollout sequence y + 0 and action count t +- 0
2: while < T do
3: Generate VLM response sequence
To(- 22, v)
4: Concatenate y to the y sequence with the role of assistant: y <- y y/:
<search>> </search> detected in y, then 6: Extract search query & + Porse(y:) and Retrieve related image 1t = Het(g)
7: else if <region> </region> detected in y then
8: Extract visual perception tokens loc t- Parse(J)) and Processing image I Py (loc, v)
else if <answer> </answer> detected in y, then return final generated trajectory y end if
10:
11:
12:
Concatenate vision tokens 1, to the sequence with the role of user: +- y d
13:
Increment action count ( +
14:
end while
15:
return final generated trajectory y
Training Strategy for Reinforcement Learning in Multi-Step Interactions. We propose RL framework that enables VLM to learn how to interact with search engines and gather visually rich information from coarse-to-fine perspective. The optimization objective is formulated
max groma D.y.a [x, v)] 3DKL [TT 758
x;; VV) || Tier x;; V)]
(13)
where the To is the policy model, Tref is the reference model, DDL. is KL-divergence, and ~
To(-] x; V)= ( x) @ is the rollout process. Our approach implements Group Relative Policy Optimization (GRPO) [3). which optimizes the model's retrieval-augmenned reasoning capability with group-sampled role-play trajectories. Please refer to Appendix [ for more details.
3 Experiments
3.1 Experimental Settings
Datasets, Metric and Baselines. To evaluate the effectiveness of VRAG-RL, we compare our method with the baxt- and vision-based baselines: (1) Vanilla RAG [ uses the original question as query for the search engine, then VLMS perform direct inference. (2) ReAct 48 The model performs rewriting, retrieving, and reasoning the think-then-act paradigm. (3) Search-R1(-
VL) is the baseline adapted from Search-R1 [19]. and the settings are aligned across all experiments

---

Table I: Main Results. The best performance marked in bold. VIA and ViDoSeek mainly focus on reasoning type, while MenchnBenchh focuses on the visual type of reference content. OCR-based (+) RAG and purely visual RAG evaluated with the same prompt and setting.
VIA
VIDOSEEK
MMLONGBRNCH
METHOD
Singlehhpp
Multi-hup
Extraction
Logic
Text
Table
Chart
Figure
Layout
OVERALL
Qwenn2.5-4 S-VL-3B 3B-Instruct
Vanilla
15.1
12.1
5.1
2.5
ReAst [8]
11.8
9.9
5.3
7.4
6.5
3.7
3.9
5.2
2.5
8.4
Search-RI
17.5
13.8
13.3
6.8
Vanilla RAG
19.4
12.2
10.1
17.3
2.2
4.1
5.2
4.7
4.3
13.2
ReAct 4X
15.7
10.9
6.7
14.2
2.7
3.6
3.4
3.1
5.1
10.9
Search RI- VL
19
26.3
20.1
20.1
29.8
8.5
7.8
7.6
21.3
VRAG-RL (Ours)
65.3
38.6
63.1
73.8
22.7
16.1
21.9
21.4
19.5
53.5
Qween2.5 2..on..5-V--22. ----VV----B-----
10.6
24.7
3.3
ReAct HR
21.2
13,3
14.3
21.3
5,9
5.1
7.3
5.5
1.7
15.8
Search-RI
28.4
19.7
6.0
5.9
Vanilla RAG
29.1
17.4
26.4
14.7
15.9
7.6
ReAct 4X
34.8
20.4
27.5
42.1
10.1
12.4
10.2
6.2
7.1
26.9
Search RI- VL
19
48.3
42.3
40.5
50.3
19.9
13.4
12.9
11.4
10.2
37.4
VRAG-RL (Ours)
69.3
43.1
60.6
74.8
26.1
26.3
24.8
25.9
21.2
57.1
to ensure fairness. We evaluate method three challenging, visually rich benchmarks: ViDoSeek [1]. Slide VQA [40] and MMLongBench [29]. The model-based evaluation metric is binary or 1, indicating the accuracy of the model's responses. Please refer to Appendix E and | for more details.
Training and Inference Setups. We conducted SFT and RL on Ilama-factory respectively. We use full parameter fine-tuning and cosine learning scheduler with a warmup ratio of 0.1 during SFT. When training with the GRPO algorithm, we set the group size to 5 and the coefficient for the KL loss is typically set to 0.01, but if we perform cold start, we set it to the KL loss constraint the model. During training and inference, we built a search engine from database of approximately ~ 70k visual documents. All the experiments are conducted on 8 NVIDIA A100 80G GPUs. Please refer to Appendix G for detailed hyperparameters used in our paper.
[53 and verl [38]
disable
3.2 Results
Main Results. As shown in Table compared to purely visual methods, DCR-based methods exhibit significant limitations on visually intensive benchmarks. On the one hand, visual information inherently contains elements that cannot be represented by text, such element positions, layout, and color, etc. On the other hand, the perceptual capabilities of OCR models are considerably inferior to those of the current advanced VLMs, which restricts the overall performance ceiling of the framework. Visual- -based methods have proven to be a more elegant solution compared to DCR-based methods, especially in tasks related visual understanding. For prompt-based baselines of vision domain, Vanilla RAG and ReAct exhibit poor performance, far behind RL-based baselines and our method on various benchmarks. The 7B model, compared to the 3B model, possesses superior perception and understanding capabilities, exhibiting strong performance across various datasets. For RL-based baselines, method also performs better than search-R1-VS (34.7 + 57.1) and Qwen2.5-Ve ----3---Insttt (21.3 + 53.5). The evaluation results on Slide VQA and ViDoSeck demonstrate our model's 's significant improvement in reasoning capabilities across various reasoning tasks. Furthermore, as MMLongBench includes multiple visual elements, which indicates the model's improvement in visual perception capabilities, this phenomenon is related to our proposed visual perception action space. The results across various benchmarks prove the effectiveness and generalization of our method in the retrieval and reasoning of visually rich information.
both Vwen225-VV--7B-Inssrnc
Approach Ablations. As shown in Table taking Qwen2.5-VL.ce -L---BB-IIsstuee decompose the key components of VRAG-RL to examine the impact of different rewards and action space on performance separately. In macro view, removing each module results in clear drop in the accuracy, which validates the power of our RAG-specific reward and Visual-perception action space.
example,

---

The action space module we defined shows cer- tain degree of improvement on different bases, which proves the effectiveness of the visual perception-based strategy. Consistent with the findings demonstrated in MMLongBencc in Figure the visual perception action space we introduced has generally enhanced the frame-
Table 2: Ablation study on three benchmarks.
REWARD
ACTION SPACE
Vanilla
RAG-Specific
Search
FisualPPereepie
Accuracy
47.2
54.9
57.1
work's performance, particularly improving high-density visual information. Furthermore, ablation experiments the reward model further demonstrate that retrieving relevant information is pre- requisite for high-quality generation, highlighting the role of high-quality retrieval in RAG, which proves the importance of our RAG-specifie reward. Comparisons and analyses of experiments across different settings collectively demonstrate the effectiveness and generalization of our modules, and their combination comprehensively enhances end-to-end performance from various perspectives.
3.3
Analysis
Better retrieval facilitates high-quality generation. Our VRAG-RL framework significantly enhances the retrieval efficiency, which is crucial for constructing coherent and informative context for high-quality generation. As demonstrated in Figure 3 the context length has a substantial impact on model performance. When the context is too long. it can introduce noise and interfere with the model's ability to generate accurate answers. In contrast, when relevant information is retrieved early and comprehensively, the model can build a more focused and informative context. As shown in Figure 4 our model is more effective at retrieving relevant information compared to traditional prompt-based rewrite methods. Our approach provides the vision model with a better context for generating high-qualit, answers.
VRAG-RL
ReAct
Direct Retrieval
70
Recalle1
60
50
40
SlideVQA
ViDoSeek MMLongBench
Figure 4: The retrieval performance of tradi- tional prompt-based RAG and our approach.
Text
Visual perception action space provides fine-grained perspective. The visual perception action space introduced in our framework further enhances understanding by allowing the model focus on information- dense regions of images. Figure 5 iillustrase the relative performance comparison between our approach with visual perception action space and various baselines, from which we can observe that VRAG-RL not only performs well textual tasks but also shows noticeable im- provements in tasks requiring visual perception abilities, particularly Layout, Chart, and Figure. This is particularly important given the current limitations in computational resources, especially considering that VLMMs are highly memory- -entory-ntensive. Using this dynamic resolution strategy, the model can achieve more detailed perception within the
Toble
Layout
Chart
Figure Vonill - w/o Visuri Penception w/ Visua Perception
Figure 5: Relative perfor- mance on MMLongBench. constraints of limited computational resources, rather than simply maximizing the resolution of the original image. Our method achieves improvement in perceptual abilities while optimizing resource utilization. Perhaps this human-like way of thinking and acting is the key to AGI.
Reinforcement learning helps the model to perform multi- step reasoning effectively. One major challenge of the prompt-based method is that the number of interactions in- creases, the model's 's capability to follow instructions weakens. However, pre-training with SFT helps the model reason in a pre-defined pattern compared to cold start, but also impacts the model's 's inherent foundational capabilities some extent. To further explore the activation of multi-turn turn reasoning abilities in models by RL, we compared the iterative reasoning performance of models with and without RL, as shown Table For our method with action space, effective actions are crucial for interacting with the external environment. The Invalid Action Rate indicates incorrect action responses, which include not only pattern errors but also hallucinations caused by wrong cropping, answering before retrieval, and so Incfficient reasoning often includes repeated
Table 3: Average Finish Rate (%) and Average Invalid Action Rate (%).
Method
Invalid Action
Finish Rate 1
SFT RL
9,4
84.2
5.1
97.1
8

---

meaningless searches, leading to a decrease in the finish rate. Our method with RL effectively reduces the invalid rate and increases the finish rate. guides the model make optimal decisions each step of the reasoning process, enabling it to flexibly adjust strategies when faced with different types of out-of-domain visual information, thereby better completing complex reasoning tasks.
Model-based reward offers more stable training compared to rule-based reward. Previous works often use EM as the reward, which is too strict. Unlike short answers for data-related questions, it is difficult for the model's responses to exactly match the golden answer, resulting in inefficient training. However, using recall as a reward may lead to misjudgments and cause models to hack the function, resulting in repetitive responses that destabilize training. In contrast, model-bbsed reward leverages an evaluation model to assess the quality and relevance of generated responses in a more flexible manner. This approach not only aligns better with real-world applications but also provides a more stable and effective training signal, as demonstrated in Appendix A The model-based reward thus enables VRAG-RL to achieve more robust performance across visual reasoning tasks.
Time efficiency. As shown in Figure 6 our method's multi-turn interaction with external environ- ments can lead to increased latency. The latency of vanilla RAG remains consistent, as it only performs a single search and provides an answer. ReAct RAG, a prompt-bane based method, also demonstrates multi-turn interaction capabilities due to the fundamental rea- soning abilities of the model. However, it is limited to only two defined actions: answer and search. Due the lack of sufficient perception capabilities, it often falls into repetitive search loops. Our approach equips the model with a visual perception space that can effectively understand visually rich images. The model can quickly extract answers after retrieval, thus avoiding ineffective searches. Despite the increase in latency, the overall performance improves due to the higher quality of generated answers, making the trade-off between latency and accuracy highly beneficial for visually rich retrieval and understanding tasks.
Vonitl RAG
Arswer Search Perception
ReAct RAG
VRAG-RL
Avg. Action Count
Figure 6: Latency Analysis on Generation.
Case Study. In Figure 7 and & (Appendix H). we list the trajectories of our VRAG-RL to illustrate how our model reasons and interacts with the environment. These cases highlight two challenges in visually rich information RAG: (1) accurately retrieving relevant images, and (2) the reference information often requires higher-resolution perception. In Figure 7 we can observe that the model demonstrated reflective capability, and eventually identified subtle clues the relevant images. Moreover, as shown in Figure 8 the model engages in visual perception actions only when required, showcasing human-likk like reasoning instead of simply replicating patterns from its training data.
4 Related Work
Vision-based Retrieval-augmeened Generation. RAG demonstrates significant advantages in
addressing knowledge-intensss problems 22
12 2
Traditional text-based RAG methods typically
involve designing different agents to interact with search engines [45] 5 6 44 25
32
21 [0].
However, with the widespread adoption of electronic documents, knowledge is no longer confined to text. Recently, there has been an increasing amount of research on OCR- free retrieval methods that directly align textual queries with images
[5T Ii. Furthermore, more and more work is focusing on
multimodal RAG agents 41 g 16] 24 46). enabling more accurate retrieval and extraction of visual information. Our work builds upon these developments by incorporating visual perception actions into visual-based RAG, effectively activating the reasoning and understanding capabilities of VLMs.
Reinforcement Learning with Large Models. Reasoning capabilities are crucial for models to effectively address complex problems, and RL has been proven to be a powerful approach to enhance these capabilities [S]. Previous work applied RL in the training of LLMs [3 43 B5] 36 27). Additionally. more and more works aim to use RL to enhance the reasoning capabilities of VLMs
3 30 28 52 Recent advancements have seen RL being widely applied to the training of large model-drives agents 42). These agents, especially RAG agents, require robust multi-step reasoning capabilities to interact effectively with external environments [17 23 However, there is still a scarcity of RL frameworks specifically tailored for multimodal iterative reasoning, which is essential

---

for handling visually rich information. Our work aims to fill this gap by introducing novel RL framework that enables VLMs perform iterative reasoning with visual perception actions, thereby enhancing their reasoning capabilities in complex, multi-modal retrieval-augment reasoning tasks.
5 Conclusion and Future Work
In this paper, we introduce VRAG-RL, a novel reinforcement learning framework tailored for complex reasoning across visually rich information. Our approach enables Vision Language Models to interact with search engines more effectively, significantly enhancing their reasoning and retrieval capabilities. Extensive evaluations information reasoning, retrieval, and understanding with our model. For future work, we plan to introduce more actions that mimic how humans handle complex information, allowing the model to focus more on deep thinking. Additionally, we aim to reduce hallucinations by leveraging more advanced models, further improving the accuracy and reliability of our framework.
various benchmarks have demonstrated significant advantages in visual
Acknowledgments
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC. The Al-driven experiments, simulations and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
10

---

References
[1] Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K. Wang. P., Wang. S., Tang, J.,
al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1399 (2025)
[2] Chen, J., Lin, H., Han, X., Sun, L.: Benchmarking large language models in retrieval-augmente generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 17754-17762 (2024)
[3] Chen, I.,, Li, I.., Zhao, H., Song, Y., Vinci: R1-v: Reinforcing super generalization ability in vision-language models with less than $3. http:: http:///huub..co//eee-ggeee///// (2025), accessed: 2025-02-02
[4] Chen, M., Li, T., Sun, H., Zhou, Y., Zhu, C., Wang, H., Pan, J.Z., Zhang, W., Chen, H., Yang, F., et al.: Research: Learning to reason with search for ilms via reinforcement learning. arXiv (2025)
preprint arXiv:2503.
[5] Chen, Z. Liu, K., Wang, Q.. Liu, J., Zhang, W., Chen, K., Zhao, F: Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.2018: (2024)
[6] Chen, Z., Liu, K., Wang. Q. Zhang, W., Liu, J., Lin, D., Chen, K., Zhao, F:: Agent-flan. Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.1 12881 (2024)
[7] Chen, Z., Wang, W., Cao. Y., Liu, Y., Gao. Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)
[8] Cho, J., Mahata, D., Irsoy, O., He, Y., Bansal, M.: M3docrag: Multi-modal retrieval is what you need for mult multi-document understanding. arXiv preprint arXiv:2411 1.:1:211.000 1.04952..222 (2024)
[9] Du, Y., Li, C., Guo, R., Yin, X., Liu, W., Zhou, J., Bai, Y.. Yu, Z., Yang. Y., Dang, Q. al.: Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941 (2020)
[10] Fang, Y., Sun, T., Shi, Y., Gu, X.: Attentionrag: Attention-guided context pruning in retrieval- augmented generation. arXiv preprint arXiv:2503. 10720 (2025)
[11] Faysse, M.. Sibille. H., Wu, T., Omrani. B., Viaud, G., Hudelot, C., Colombo, P.: Colpali: Efficient document retrieval with vision language models. In: The Thirteenth International Conference Learning Representations, pp. 1-26 (2025)
[12] Gao, Y., Xiong, Y., Gao, X., Jia, Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H., Wang, H.: Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2, (2023)
[13] Guo, D., Yang. D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P., Bi, X., al.: Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 1 (2025)
[14] Huang, W., Jia, B., Zhai, Z.., Cao, S., Ye, Z.., Zhao, F. Xu, Z., Hu, Y., Lin, S.: Vision- rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749 (2025)
[15] Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al.: Openai ol system card. arXiv preprint arXiv:2412.16720 (2024)
[16] Jiang, D., Zhang, R., Guo, Z., Wu, Y., Lei, J., Qiu, P.. Lu, P., Chen, Z., Fu, C.. Song, G., et al.: Mmscarch: Benchmarking the potential of large models multi-modal search engines. arXiv preprint arXiv:2409. 12959 (2024)
[17] Jiang, P., Lin, J., Cao, L., Tian, R., Kang, S., Wang, 7., Sun, J., Han, J.: Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiy preprint arXiv:2503, .00223 (2025)

---

[18] Jin, B., Yoon, J., Han, J., Arik, S.O.: Long- Ilms meet rag: Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983 (2024)
[19] Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang. D., Zamani, H., Han, J.: Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint T0V::50330091.5 (2025)
[20] Kaelbling, L.P., Littman, M.L., Moore, A. ..: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237-285 (1996)
[21] Lee, C.. Roy, R., Xu, M., Raiman, J., Shoeybi, M.. Catanzaro, B., Ping, W.: Nv-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint AXXv:2445517422. (2024)
[22] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t. Rocktäschel, T., et al.: Retrieval-augmenntd generation for knowledge-intensivee nlp tasks. Advances in Neural Information Processing Systems 33, 9459-9474 (2020)
[23] Li, X., Dong, G., Jin, J., Zhang, Y., Zhou, Y., Zhu, Y., Zhang, P., Dou, Z.: Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501 .53666 (2025)
[24] Li, Y., Li, Y., Wang, X., Jiang, Y., Zhang. Z., Zheng, X., Wang, H., Zheng, H.T., Huang, F,, Zhou, J., et al.: Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint TeXXx:2211.009990 (2024)
[25] Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., Zhang, M.: Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308. 002281 (2023)
[26] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306 (2024)
[27] Liu, R., Wang, J., Shi, Y., Xie, Z., An, C., Zhang, K., Zhao, J., Gu, X., Lin, L., Hu, W., et al.: Attention as a compass: Efficient exploration for process-supervieed in reasoning models. (2025)
arXiv preprint wxXiv:2509.6660
[28) Liu, Z., Sun, Z., Zang, Y., Dong, X., Cao, Y., Duan, H., Lin, D., Wang, J.: Visual-rft: Visual reinforcement fine-tuning, arXiv preprint arXiv:2503.01785 (2025)
[29] Ma, Y., Zang. Y., Chen, L.,, Chen, M., Jiao, Y., Li, X., Lu, X., Liu, Z., Ma, Y., Dong, X., et al.: Mmlongbench-docc; Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.0152. (2024)
[30] Meng, F., Du, L.. Liu, Z., Zhou, Z., Lu, Q., Fu, D., Shi, B., Wang, W., He, J., Zhang, K., et al.: Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.0744
(2025)
[31] Meng, Y., Xia, M., Chen, D.: Simpo: Simple preference optimization with a reference-free reward, Advances in Neural Information Processing Systems 37, 124198-124235 (2024)
[32] Moreira, G.d.S.P. Osmulski, R., Xu, M., Ak, R., Schifferer, B., Oldridge, E.: Ny-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.1583) (2024)
[33] OpenAl: Hello gpt-4o. https:// wwtp//ndwweeeeloogoggeeeoeoee http:p///ggaaaaaoo//////aaaoooaa (2024)
[34] Pichai, S., Hassabis, D., Kavukcuoglu, K.: Introducing gemini 2.0: our new ai model for the agentic era (2024)
[35] Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference
optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems 36, 53728-53741 (2023)
[36] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint uuXiv:1707.0644. (2017)
12

---

[37] Shao, H., Qian, S., Xiao, H.. Song, G., Zong, Z. Wang, L., Liu, Y., Li. H.: Visual cot: Advancing multi modal language models with comprehensive dataset and benchmark for reasoning. Advances in Neural Information Processing Systems 37, 8612-8641 (2024)
t-thoughtt
[38] Sheng, G., Zhang, C., Ye, Z. Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., Wu, C.: Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19255 (2024)
[39] Sutton, R.S., Barto, A.G., et al.: Reinforcement learning. Journal of Cognitive Neuroscience 11(1), 126-134 (1999)
[40] Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T. Saito, I., Saito. K.: Slidevqa: A dataset for document visual question answering on multiple images. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 13636-13645 (2023)
[41] Wang, Q., Ding, R., Chen, Z., Wu, W., Wang, S., Xie, P., Zhao, F: Vidorag: Visual docu- ment aatrieval-augmentee generation via dynamic iterative reasoning agents. arXiv preprint arXiv:2502.1 18017 (2025)
[42] Wang. Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen. M.N., Liu, L.,, et al.: Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20033 (2025)
[43] Williams, R.J.: Simple statistical madiient-folloiigg algorithms for connectionist reinforcement learning. Machine Learning 8, 226 (1992)
[44] Wu, J., Yin, W., Jiang, Y., Wang, Z., Xi, Z., Fang, R., Zhang. L., He, Y., Zhou, D., Xie, P., et al.: Webwalker: Benchmarking iims in web traversal. arXiv preprint arXiv:2501.077 .07557 (2025)
[45] Wu, W., Huang, S., Jiang, Y., Xie, P., Huang, F., Zhao, H.: Unfolding the headline: Iterative self- questioning for news retrieval and timeline summarization. arXiv preprint arXiv:2501.008: (2025)
[46] Xia, P., Zhu, K., Li. H., Zhu, H., Li, Y., Li, G., Zhang, L., Yao, H.: Rule: Reliable multimodal rag for factuality in medical vision language models. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1081-1093 (2024)
[47] Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang. J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z. Qiu, Z.: Qwen2.5 technical report. arXiv preprint 151 (2024)
[48] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: React: Synergizing reasoning and acting in language models. In: The Eleventh International Conference on Learning Representations. pp. 1-33 (2023)
[49] Yu, E., Lin, K., Zhao, L.. Yin. J., Wei, Y., Peng. Y., Wei, H., Sun, J., Han, C., Ge, Z., et al:: Perception-rl: Pioneering perception policy with reinforcement learning. arXiv preprint arXiv:2504.07954 (2025)
[50] Yu, R., Ma, X., Wang, X.: Introducing visual perception token into multimodal large language model. arXiv preprint arXiv:2502.17433 (2025)
[51] Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., al.: Visrag: Vision-based preprint arXiv:2410. 10594 (2024)
augmentedd generation on multi-modality documents. arXiv
[52] Zeng, Y., Huang, W., Huang. S., Bao, X., Qi. Y., Zhao, Y., Wang, Q., Chen, L., Chen, Z., Chen, H., et al.: Agentic jigsaw interaction learning for enhancing visual perception and reasoning in lasion-language models. arXiv preprint arXiv:2510.01300 (2025)
[53] Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., Ma, Y.: Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.1337 (2024)
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper includes potential limitations of the proposed method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Theoretical results are accompanied by detailed assumptions and complete proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed instructions for reproducing the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The paper provide data and detailed instructions for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httaa//gggieee//eeuuabiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The paper provides a detailed description of the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experimental results were conducted multiple times and averaged, provid- ing data and error analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides the details of them.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
The research adheres to all guidelines outlined in the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses both the potential positive and negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve high-risk data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper describe the usage of LLMs and VLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurr/ /ttp://///////oofeeeeee///////// for what should or should not be described.
20

---

Appendix
A Model-Based Reward
We employ model-based reward evaluate the quality and relevance of generated responses. Specifically, we utilize Dwen2.5-7B-Insttrus [47] as our reward model. This model deployed on NVIDIA A100 GPUs to enable efficient batch evaluation. The prompt used for the reward model is illustrated in Figure T2 Given the input query, reference answer, and generated response, the reward model assesses the correctness of the generated response and outputs binary value (0 or 1) to represent the accuracy of the answer. Compared to the rule-based reward like exact match (EM) Recall, used in previous work [9]]] our model-based reward provides more flexible and comprehensive evaluation of the generated response. This leads to higher training efficiency and better generalization to diverse datasets.
The implementation of the search engine
To effectively support the retrieval -aggeeted generation tasks our VRAG-RL framework, we implemented OCR-based and vision-based pipeline separately. The vision-based retriever is built upon the state-of-the-an embedding model ColPali [[]], which specifically designed for aligning textual queries with images. For the textual retrieval pipeline, we employ the PP-OCR [9] to extract text from images. We utilize the Llama- Index ensure an efficient indexing and querying mechanism for large-scale image datasets. In our experiments, we deployed the search engine on single NVIDIA A100 80G GPU, allowing us to handle large-scale queries efficiently. The use of batch querying further optimizes the retrieval speed, making it suitable for real-time applications.
C Reinforcement Learning Framework with GRPO
Our framework implements the Group Relative Policy Optimization (GRPO), which leverages the average reward of multiple sampled outputs as baseline rather than relying on a learned value function. The policy model is optimized by maximizing the following objective function:
(C)Po(!) =EEND. (
1 0
min
To(s,, Weisee V) mou(s1, Mervees V)
As.es
laxe (v..e) 1-1:ftn,e)) mml
molye aas Me.sces V.)
-e.1+t))...)
Ar..) TTx [sel]med
clip
Taulii (m, Mi. 17)
where rollout module samples a group of trajectories (11, "iu.,,,..... ,.) from the reference policy Tee for each input question æ by interacting with the external environment V. Á represent the advantage, computed based on the relative rewards of outputs within each group.
D Expert Trajectories Collection
Data Collection. To train our model effectively, we collected expert trajectories using Qwen- VL-max-latest for prompt-basee based data collection. Specifically, utilized the React-based prompt to gather data, ensuring that the model could perform complex reasoning tasks. During the data collection process, whenever grounding required to focus specific regions of interest within images, we employed Qwen2.5VI to perform the grounding tasks. This was done under the guidance of the historical trajectories.
Data Proportions. To ensure that our model could perform diverse multi-step reasoning during Reinforcement Learning (RL), we carefully balanced the training data. Specifically, we balanced the trajectories based on the number of steps (2-6) and the types of actions involved (search and perception). This approach ensured that the model was exposed a wide range of reasoning tasks and could learn to handle different types of interactions with the environment effectively.
21

---

E Dataset Information
We evaluate our method on three visually rich document datasets: VQi ViDoSeck, and MMLongbench.
1. SQ 40 is dataset for document visual question answering focused on understand- ing slides. It contains over 2,600 slide decks with more than 52,000 slide images and 14,500 questions that require complex reasoning skills such as single-hop, multi-hop. and numerical reasoning. The dataset is designed to support various reasoning types and includes annotated arithmetic expressions for numerical questions to enhance reasoning capabilities.
2. ViDoSeek [41] is dataset specifically designed for visually rich document retrieval-rease reason- answer tasks. It aims to evaluate the performance of RAG systems on large-scale document collections. Unlike traditional VQA datasets that focus single images documents, ViDoSeek contains queries with unique answers across a collection of approximately 6,000 images, covering diverse content types such as text, charts, tables, and layouts. This dataset provides more comprehensive and challenging benchmark for evaluating the retrieval and reasoning capabilities of RAG models in real-world scenarios.
3. MMLongbench [29] is dataset designed to evaluate the document understanding capabili- ties of VLMs with an emphasis on long-com long-context, multi-modal documents composed of text, images, charts, tables, and layout structures.
Table Statistics of datasets.
Datuset
Questions
Corpus Size
Visual Elements
SlideVOA-Tee
2020
8000
Text, Chart, Table, Layout
Slide -rain
12268
44359
Text, Chart, Table, Layout
ViDoSeck
1142
5400
Text, Chart, Table, Layout
BMLooggeench
847
6492
Text, Chart, Figure, Table, Layout
F
Compared Baselines
Here we detailedly introduce the baselines we compare with and our re-produce details.
1. Vanilla RAG. There two types of Vanilla RAG: text-based and visual-based. Text-based Vanilla RAG uses text as the retrieval corpus, which is reflected in text search engines and text modality generation. During the retrieval phase, it directly uses the original question to search for relevant text, which is then inserted into the context to answer the question. Visual-based Vanilla RAG uses images as the corpus. During the retrieval phase, it directly uses the original question to search for relevant images, which are then inserted into the context answer the question.
2. ReAct RAG 48
The method incorporates Chain-of of- Thought (COT) prompting RAG agent tasks with format of a Thought-Actinn Obbievvtionaa loop. The main difference between text-based and uisual-based approaches lies in the retrieval corpus of the search engine and the modality of the information inserted.
3. Search-R1 [19]. The method introduces multi-turn reasoning RL into the text RAG. We used our framework for reproducing, which includes multi-turn interactions and rule-based rewards.
4. Search-R1-VL. This is vision-based baseline implemented on our framework based on search-R1. We used the same reward and post-process methods and trained models based cold start with the same dataset VRAG-RL.
G Hyperparameters
The detailed hyperparameters we use during training are shown Table 5 and Table 6 identical hyperparameters for different models.
We employ
22

---

Table 5: Key hyperparameters for SFT.
Table 6: Key hyperparameters for RL.
Name
Value
Name
Value
Finetuning type
Full
Number of agent groups
Freeze vision tower
True
Warmup steps ratio Mini batch size
0.285
Freeze multi-modal projector
True
64
Freeze language model
False
Micro batch size per GPU I Learning rate (Actor)
Cutoff len
16384
1.0e-6
Epochs
3
KL loss coefficient
0.01 (optional)
Batch size
16
Tensor model parallel size
Gradient accumulation steps
2
Total epochs
Learning rate
1.0e-5
Max prompt length
8192
LR scheduler type
cosine 0.1
Max response length
2048
Warmup ratio
GPU memory utilization
0.6
H
Case Study
In Figure 7 and 8 we list the trajectories of our VRAG-RL to illustrate how model reasons and interacts with the environment. These cases highlight two challenges visually rich information RAG: (1) accurately retrieving relevant images, and (2) the reference information often requires higher-resolution perception. In Figure 7 we can observe that the model demonstrated reflective capability, and eventually identified subtle clues in the relevant images. Moreover, shown in Figure R8 the model engages in visual perception actions only when required, showcasing human-like reasoning instead of simply replicating patterns from its training data.
I Limitations
Despite our best efforts, this paper still has some limitations: (1) Retriever Accuracy: The accuracy of the current retriever still needs improvement to better support the RAG system. (2) Base Model Capability: Enhancing the capabilities of the base model can contribute to the overall improvement of the system. (3) Real-time Requirements: The current multi-turn interactions may not meet the requirements high real-time applications.
Ethics Statement
Our research complies with the NeurlPS Code of Ethics. We used publicly available datasets (as detailed in Appendix E)), avoiding ethical concerns related to privacy, confidentiality, misuse of personal biological information. The VRAG-RL framework aims to improve visual information understanding for beneficial applications. We emphasize responsible use and remain vigilant against misuse.
K Prompts
In this section, illustrate all the prompts used in paper. Part of our prompts are taken from Search-RI [19
K.1
Vanilla RAG Prompt
See Figure
K.2
Search-R1 Prompt
See Figure 10
23

---

K.3
ReAct RAG Prompt
ReAct RAG uses the same prompt as Search-R1, shown in Figure I0
K.4
VRAG-RL Prompt
See Figure 9
K.5
Model-based Reward Prompt
See Figure 12
24

---

Question: How many more Dugwells are there than Shallow Tubewells in the years of India's s Energy Divide whose map was prepared by IWMI based on Minor Irrigation Census?
<thinlo To answer this question, need to find the specific data related to Dugwells and Shallow Tubewells India's Energy Divide map prepared by IWMI based on Minor Irrigation Census. This requires searching the relevant information.
</thinlo
<search> India's Energy Divide IWMI Minor Irrigation Census Dugwells Shallow Tubewells </search>
Image Retrieval
The image provided map showing the proportion of electric structures India's Energy Divide. It indicates the proportion electric structures using different color codes. However, it does not provide specific numbers for Dugwells and Shallow Tubewells. I need to search for more detailed data Dugwells and Shallow Tubewells from the Minor Irrigation Census.
</thinlo
<search> Detailed Dugwells Shallow Tubewells </search>
Image Retrieval
<thinlo The image contains table with the number of structures, including Dugwells and Shallow Tubewells. I need to crop this part of the image to focus on the relevant data. </thinke
<region [361, 366, 721, 517J </region>
SERAROK SORUCTURE TURP
20040
of
Ramtarr Seria Share:
Fil
humbber of biloed Iripeia Shuctars
11.0
Select & Crop & Zoom in
<thinlo The cropped image provides the specific numbers for Dugwells (9.20 million) and Shallow Tubewells (9.10 million). To find how many more Dugwells there are than Shallow Tubewells, need to subtract the number of Shallow Tubewells from the number of Dugwells.
</thinko
<answer> are 0,10 million (or 100,000) more Dugwells than Shallow Tubewells, </answer>
Figure 7: Case for VRAG-RL.
25

---

Question:
Which antagonist has lower bicavailability: the drug that inhibits CYP-450 isoenzymes reduces hepatic blood flow or Famotidine?
<thinle To onswer this question, need to identify the H2 antagonist that inhibits several CYP- 450 isoenzymes and reduces hepatic blood flow. Then, I will compare its bioavailability with that Famotidine.
</thinko <search> H2 antagonist that inhibits several CYP-450 isoenzymes</search
ag
Image Retrieval
uee decrter
<thinlo The image provides information about drug interactions, specifically mentioning Cimetidine's 's effect CYP-450 isoenzymes and hepatic blood flow. This aligns with the search query. need to look the lower right corner the image to confirm this. </think>
<region> [556, 337, 722, 505] </region>
Orug Intorections
Select & Crop Zoom in
thinlo The diagram confirms that Cimetidine inhibits the P-450 enzymes, leading to increased serum concentrations of drugs metabolized by these enzymes. Now, I need to find information on bioavailability Cimetidine and Famotidine determine which has bioavailability.
</think
<search> bioavailability of Cimetidine and Famotidine </search>
Compar man of
Image Retrieval
<thinlo Based the provided table, Cimetidine has bioavailability of 80%, while Famotidine has bioavailability of 40%. Therefore, Famotidine has lower bioavailability compared to Cimetidine. The drug that inhlbits several CYP-450 isoenzymes and reduces hepatic blood flow Cimetidine, indicated in previous information. </think
canswer-Famotidine has lower bioavailability (40%) compared to Cimetidine (80%). Therefore, the H2 ontogonist with lower bioavailability is /anotidine.//nswer>
Figure 8: Case 2 for VRAG-RL.
26

---

VRAG-RI. Prompt.
System Prompt:
Answer the given question. You must conduct reasoning inside <think> and </thinks first every time you get new information. After reasoning, if you find you lack some knowledge, you can call search engine by <search> query </search> and user will return the searched results. Every time you retricve image, you have the option to crop it to obtain a clearer view, the format for coordinates is <bbox>[x1, yl, x2, y2]</bbox>. You can search as many times as your want. If you find no further external knowledge needed, you directly provide the answer inside <answer> and </answer>, without detailed illustrations. For example, <answer> Beijing </answer>.
User Prompt:
Query: (Query Description]
Figure 9: Prompt of VRAG-RL.
Search-R1(-LL) Prompt.
System Prompt:
Answer the given question. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, if you find you lack some knowledge, you can call search engine by <search> query </search> and user will return the searched results. You can search many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer>, without detailed illustrations. For example, <answer> Beijing </answer>.
User Prompt:
Query: (Query Description)
Figure 10: Prompt Search-RI(-VL.) and ReAct RAG.
Vanilla RAG Prompt.
System Prompt:
Answer the given question. You conduct reasoning inside <think> and </think first every time you get new information. After reasoning, you should directly provide the answer inside <answer> and </answer>, without detailed illustrations. For example, <answer> Beijing </answer>.
User Prompt:
Query: (Query Description]
Reference: (Retrieved Images/ / Text Tokens]
Figure 11: Prompt of Vanilla RAG.
27

---

Reward Model Prompt.
System Prompt:
Character Introduction
You are expert evaluation system for question answering chatbot.
You are given the following information:
the query
generated answer
reference answer
Your task is to evaluate the correctness of the generated answer.
Response Format
Your response should be formatted as following: <judge>True or False</judge>
the generated answer correct, please set "judge" True. Otherwise, please set "judge" to False.
Please note that the generated answer may contain additional information beyond the reference answer.
User Prompt:
Query: (Query Description] Reference Answer: (Reference Answer)
Generated Answer: (Generated Answer)
Figure 12: Prompt of Reward Model.
28


Reviews:
[{"review_id": "bzwOAwS0Tf", "reviewer": "Reviewer_FHAe", "summary": "The paper proposes VRAG-RL, a reinforcement learning framework designed to enhance vision–language models (VLMs) by introducing a visual-perception action space (e.g., cropping and zooming) and a composite reward tailored for retrieval-augmented generation (RAG). This reward combines retrieval efficiency, answer correctness based on a model, and consistency with common patterns. The training approach combines supervised fine-tuning from multiple experts with reinforcement learning using a GRPO-based policy gradient. Evaluated on three visually complex QA benchmarks (SlideVQA, ViDoSeek, and MMLongBench) VRAG-RL improves top-line accuracy for a 3B model and a 7B model, compared to the strongest baseline, Search-R1-VL.", "questions": "- Have you tried reward-model swap experiments (e.g., using GPT-4o judge) to verify that gains persist?\n- How sensitive is performance to the α/β/γ weights in Eq 12?\n- Could the visual perception policy hallucinate crops (i.e., propose empty regions)? What proportion of invalid crops remain after RL?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Well-motivated visual action space that reduces context and raises “effective resolution” without inflating input length\n- Clear empirical gains on three diverse benchmarks and two model sizes. Ablations show that each component matters.\t\n- End-to-end agent framework (rollout algo 1) that unifies search, cropping, and answer steps; cases show coherent multi-turn reasoning.\n\nWeaknesses:\n- Because both training and evaluation depend on the same 7B Qwen reward model (binary accuracy), the policy may learn to exploit that grader’s characteristics rather than the underlying task. Adding independent checks (like exact numeric accuracy on SlideVQA) might reduce reward hacking and improve the results.\n- Paper highlights two success cases but does not discuss the failure patterns (e.g., when cropping the wrong region, retrieval recall is low, etc.). Understanding failure modes would guide future work.\n- Recent vision-RL agents such as Vision-R1 and Perception-R1 (2025) are not included. Search-R1-VL is a fair baseline, but comparison to those highly-related methods is essential to judge novelty and delta.", "was_revised": true, "final_justification": "Thank you for the strong rebuttal. The additional experiments, failure case analysis, and Vision-R1 comparison effectively address my concerns and enhance the work’s clarity and rigor. I am therefore increasing my rating.", "rebuttal": "Dear Reviewer FHAe,\n\nWe sincerely appreciate your insightful review and the valuable insights you provided, which are highly beneficial for highlighting our contributions. Your recognition of our work's strengths has been a great encouragement to our team. We have tried our best to address all concerns in the last few days. Please see our responses to each point below:\n\n> **W1: Adding independent checks (like exact numeric accuracy on SlideVQA) might reduce reward hacking and improve the results.**\n\nFollowing your constructive suggestions, we perform consistency checks between our current reward strategy and the rule-based evaluation strategy to verify that reward hacking does not exist. As these two evaluation strategies are completely different, when both strategies show the same performance improvement trend, it indicates that our model is objectively being optimized, rather than relying on hacking the reward. The results are shown in the table below.\n\n| Method | F1 | Accuracy (Binary) |\n| --- | --- | --- |\n| Vanilla RAG | 29.4 | 25.9 |\n| ReAct | 37.1 | 30.9 |\n| Search-R1-VL | 42.9 | 46.7 |\n| VRAG-RL | 53.2 | 62.3 |\n\n\nWe used the F1 metric from the SlideVQA benchmark for additional evaluation. The Pearson coefficient between the two sets of data is calculated to be 0.97, which indicates a strong positive linear correlation between the two metrics. The consistency in their trends demonstrates the effectiveness of our reward and the improvement of our framework.\n\n\n\n> **W2: Analyzing more failure cases (e.g., when cropping the wrong region, retrieval recall is low, etc.) would guide future work.**\n\nThank you for your highly valuable suggestions, which are beneficial in highlighting our contributions. Besides simply giving incorrect answers, we categorize failure patterns into three types and we also provide corresponding solutions to each type of failure case. Some of these issues are commonly encountered by agentic RAG, while others are unique to our VRAG-RL, as follows:\n\n+ The coordinates of the region of interest have a slight bias.\n    - Analysis & Solution: The grounding task in the pre-training domain of Qwen2.5-VL uses actual pixel format (some VLMs use normalized coordinates). Since the patch size in vision embedding is 28*28, a slight bias is quite normal. During training, we use the original coordinates. But during inference, we perform region upsampling as $[\\lfloor x_{min} / 28 \\rfloor \\cdot 28, \\lfloor y_{min} / 28 \\rfloor \\cdot 28,\\lceil x_{max} / 28 \\rceil \\cdot 28,\\lceil y_{max} / 28 \\rceil \\cdot 28]$. This simple yet effective approach slightly expands the region to an integer multiple of the patch size, effectively mitigating the localization hallucination without any additional overhead.\n+ The model selection for cropping regions is irrelevant (or empty). \n    - Analysis & Solution: During the SFT, we scale up more high-quality data by mixing experts and  increase the proportion of visual perception action data. During the RL, we enhance this capability of action. Generally speaking, the trajectory that answers correctly is highly related to the golden reference in its region of interest, which is one of the reasons why reinforcement learning is effective.\n+ Unable to retrieve relevant information or too many search iterations, which is a common issue with agentic RAG.\n    - Analysis & Solution: In our approach, we utilize the Retrieval Efficiency Reward to optimize the model's interaction with search engines, addressing this issue to a certain extent. In practice, due to the limitations of search engine performance, it often fails to retrieve relevant information. When the interaction process becomes too lengthy, we prompt the model to provide answers by summarizing existing information, such as \"Please give me the answer based on context in \\<answer\\>...<\\answer>.\" This method enables the model to offer a possible answer even when its confidence is not high.\n\nWe will discuss these failure patterns and solutions along with more specific cases in the revision, to highlight our contributions and provide potential inspiration for future work.\n\n\n\n> **W3: Comparing with more recent works such as vision-r1 and perception-r1 would help highlight the novelty.**\n\nFollowing your highly valuable suggestions, we have summarized the latest relevant work, as shown in the table below.\n\n| Method | Task | Context | Action Space | Multi-Turn Interaction |\n| --- | --- | --- | --- | --- |\n| Perception-R1 (2025.4) | Detection&OCR  | Single Image | None | ✗ |\n| VLM-R1 (2025.4) | Detection | Single Image | None | ✗ |\n| Vision-R1 (2025.4) | Mainly focus on mathematical geometry QA | Single Image | None | ✗ |\n| VRAG-RL (Ours) | open-domain QA | Multi Images | Visual Perception | ✔ |\n\n\nDuring the rebuttal, we reviewed the latest visual perception related works from preprints over the past three months. Perception-R1 and VLM-R1 are both trained on the Grounding task, with outputs constrained to coordinates and object categories/contents, and evaluation metrics using mIoU, which cannot be transferred to evaluate our general QA task. Vision-R1 is trained on mathematical geometry task and can generalize to general QA tasks. Therefore, within related work above, we can only conduct more comparative experiments with Vision-R1, as shown in the table below.\n\n| Method | Accuracy |\n| --- | --- |\n| Vision-R1 | 24.4 |\n| VRAG-RL | 57.1 |\n\n\nThe Vision-R1 model performs well in single image QA tasks, but its performance is constrained by its limited ability to handle multi-turn tool calls and multi-image understanding. In fact, when it comes to agentic RAG tasks, the model's capabilities expand beyond mere image understanding to include effective retrieval and tool call features. Considering the characteristics of the agentic RAG task, it is essential to optimize both the retrieval and understanding capabilities of the model, which also highlights our core contribution.\n\n> **Q1: Have you tried reward-model swap experiments (e.g., using GPT-4o judge) to verify that gains persist?**\n\nDuring the rebuttal, we conducted additional experiments using GPT-4o as a reward to verify the gains. In our experiments, the Qwen2.5 model with 7B parameters was already capable of determining semantic correctness based on the response and golden answer. The api-based large scale model such as GPT-4o has achieved human-level performance in this simple task. The experiments are shown in the table below.\n\n| RL Reward Model | Overall Evaluation Results |\n| --- | --- |\n| GPT-4o | 57.3 |\n| Qwen2.5-7B-Instruct | 57.1 |\n\n\nFrom the experimental results, the gain is still consistent across different experimental settings, which validates the effectiveness of our original setting. Indeed, as long as the model is provided with sufficiently clear prompts, it becomes quite simple to judge whether an answer is correct or incorrect. This is also the reason why Qwen-7B can effectively serve as our reward model.\n\n> **Q2: How sensitive is performance to the α/β/γ weights in Eq 12?**\n\nWe experimented with many combinations of weights and conducted quantitative analysis during our research. \n\n$r_\\phi = \\alpha \\cdot r_{Ret} + \\beta \\cdot r_{Ans} + \\gamma \\cdot r_{Pat}$\n\nThe experimental results are shown in the table below.\n\n| α | β | γ | Overall Performance |\n| --- | --- | --- | --- |\n| 0.1 | 0.8 | 0.1 | 53.9 |\n| 0.2 | 0.7 | 0.1 | 57.1 |\n| 0.3 | 0.6 | 0.1 | 55.4 |\n\n\nIn practice, for the parameter γ, it can be set to a small weight like 0.1 when the model's instruction-following ability is good after SFT. During the actual implementation, when extracting the final answer and retrieval results, we will conduct a pattern validity check. \n\nFor the parameters α and β, it is better to prioritize the final answer reward and adjust the parameter values within a smaller range. This conclusion can be drawn from observing the results of the first and last lines. When α is too large, performance decreases, as the model tends to search multiple times, resulting in lower confidence in the answers provided. Conversely, when α is too small, the insufficient activation of the model's search capabilities can lead to suboptimal results. Ultimately, we chose the parameter setting of α=0.2 and β=0.7 for the experiment.\n\n> **Q3: About the hallucination of visual perception policy and invalid action rate after RL.**\n\nOur training method effectively teaches the model to identify regions of interest and significantly enhances the model's ability to perform effective actions, as shown in the table below, which is also explained in Table 3 of our manuscript.\n\n| Method | Invalid Action Rate $\\downarrow$ | Finish Rate $\\uparrow$ |\n| --- | --- | --- |\n| SFT | 9.4 | 84.2 |\n| +RL | 5.1 | 97.1 |\n\n\nWe can also compare the consistency between the trajectory annotations made by mixed experts and the regions of interest output by VRAG-RL on the same dataset to verify whether the model is learning to identify the regions of interest. The experimental steps are as follows:\n\n(i) Use mixed-expert sampling and VRAG-RL to perform inference and obtain trajectories respectively.\n\n(ii) For each identical query, we match the same images retrieved and regions of interest within the trajectories.\n\n(iii) Calculate the mAP of the region of interest in the matched images.\n\nWe evaluate the performance of the models after SFT and RL separately. The experimental results are shown in the table below.\n\n| Method | mAP@50 $\\uparrow$ |\n| --- | --- |\n| SFT | 43.1 |\n| +RL | 49.2 |\n\nIn the experiment mentioned above, we validated that our model gradually aligns with the output distribution of large-scale expert models. The model's ability to identify regions of interest progressively improved, confirming the effectiveness of our approach in reducing model hallucinations."}, {"review_id": "m91mtvqv1S", "reviewer": "Reviewer_jJNP", "summary": "The paper proposes VRAG-RL, a reinforcement learning framework to enhance vision-language models (VLMs) for retrieval-augmented generation (RAG) tasks involving visually rich documents. It introduces a visual perception action space and a fine-grained reward function combining retrieval efficiency and answer quality.", "questions": "- The paper employs Discounted Cumulative Gain (DCG) as part of the retrieval reward. While the results show its importance, the motivation for choosing DCG is not thoroughly discussed. Why DCG? Does it empirically outperform alternatives, or was it selected for simplicity?\n- The authors use a re-annotated dataset for VRAG-RL’s SFT/RL training, while baselines (Vanilla RAG, ReAct, Search-R1) likely use original annotations. Does the performance gap in Table 1 partly reflect data quality/quantity differences rather than method superiority? \n- The paper emphasizes RL’s role in optimizing retrieval and reasoning, but how much does VRAG-RL improve over an SFT-only version?\n- The reward coefficients (α, β, γ in Eq. 12) are set heuristically. Was their sensitivity tested?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strength\n- The proposed VRAG-RL achieves significant improvements over baseline methods\n- The visual perception action space (cropping/zooming regions) allows VLMs to focus on information-dense areas\n\n### Weakness\n- Some key components, such as cropping/resizing as visual perception actions and the think-then-answer (ReAct-style) paradigm, are not fundamentally novel. Similar ideas have appeared in prior work (e.g., Search-R1). While the integration of these techniques is well-executed, the paper does not clearly articulate what makes its core formulation (e.g., action space design) uniquely innovative compared to existing approaches.\n- The paper introduces multiple modules (visual perception actions, retrieval reward, re-annotated dataset, multi-round generation), but it’s unclear which one is the primary contribution.", "was_revised": true, "final_justification": "After carefully reviewing the authors' response and the discussion among reviewers, I acknowledge that the technical components of this work are generally solid. However, my concern about the paper's core contribution remains unresolved. While the individual modules are well-motivated and technically sound, they appear to address somewhat separate challenges, and the paper could benefit from a more unified perspective to better highlight its overarching insight. This currently makes it challenging to pinpoint the paper's key message. \nAs a result, I have decided to increase my score and lower my confidence.", "rebuttal": "Dear Reviewer jJNP,\n\nWe sincerely appreciate your insightful review and the valuable insights you provided. Your recognition of our work's strengths has been a great encouragement to our team. We have tried our best to address all concerns in the last few days. Please see our responses to each point below:\n\n> **W1: A more detailed explanation of what makes the core formulation (e.g., action space design) uniquely innovative compared to existing approaches (e.g., Search-R1) would be beneficial.**\n\nAlthough recent works like Search-R1 have utilized ReAct-style in textual tasks, when generalizing to the multimodal tasks, visual-related actions are required. Teaching VLMs these actions is fundamentally different from LLM-based methods. On the other hand, while some current works include crop/resize functions, generalizing them to multi-round interactions and multiple image inputs requires an entirely new paradigm and redesigned data construction pipeline. Below, we will highlight our core innovations one by one.\n\n+ VRAG-RL v.s ReAct-Style Agentic RAG (Search-R1)\n    - Differences: Search-R1 transitions RL from single-turn to multiple interactions, while our work comprehensively migrates RL to multimodal, multi-turn agentic tasks and provides insightful optimizations for RAG tasks.\n    - Our core innovation: (i) Visual-specific action space. In a multi-round scenario, we propose multi-image visual action, which is a completely new paradigm. (ii) RAG-specific reward and trajectory-wise evaluation: We decompose the model's capabilities into retrieval and reasoning. Our reward is calculated based on the entire trajectory, including both retrieval efficiency and the quality of the final answer.\n+ VRAG-RL v.s VLMs with Crop/Grounding/Resize \n    - Differences: The current work is just a single input without interaction, and it only allows for the one image QA task. Our work introduces a novel paradigm called visual specific retrieval-augmented reasoning, which supports multi-turn interaction and multi-image input.\n    - Our core innovation: (i) Scalable multi-expert data construction: We propose Multi-Expert Sampling, which efficiently constructs high-quality multi-turn trajectories by leveraging different capabilities of the expert model. (ii) Multi-modal Multi-turns Training Framework: Recent methods focus only on the understanding of single images, while we propose a novel multimodal multi-round RL training framework from scratch.\n\nWe will update the above discussion in revision according to your insightful review to highlight our contribution, providing more potential insights for the whole community.\n\n> **W2: About the primary contribution in multiple modules proposed.**\n\nOur core idea is to optimize the multimodal agentic RAG task. As the capabilities of the RAG system can be decomposed into retrieval and reasoning, we will detail our key contribution to enhancing these two capabilities by discussing problem identification, the proposed approach, and implementation, as illustrated in the table below.\n\n| Core Contributions | Problem Identification | Novel paradigm/mechanism | Implementation Approach |\n| --- | --- | --- | --- |\n| Multi-modal agentic RL framework | Lack of an effective multimodal, multi-turn RL training framework. | Multimodal multi-round sampling and optimization strategy. | Implemented using GRPO which optimizes the model’s retrieval and reasoning capability. |\n| Novel paradigm for iterative visual reasoning | The text-based reasoning overlooks the sensitivity of visual resolution and information density. | Visual perception action space, allowing VLMs to gather information from coarse-grained to fine-grained. | Multi-Expert data construction pipeline to teach the model the new paradigm. |\n| Fine-grained RAG-specific training strategy | Current agentic training rely solely on outcome rewards. | Comprehensive reward based on the entire trajectory evaluation. | Reward that integrates retrieval efficiency and outcome reward. |\n\n\nIn summary, the multiple modules we propose are closely centered around three insights. Each core contribution addresses a key issue through one or more modules. \n\n> **Q1: Why DCG? Does it empirically outperform alternatives, or was it selected for simplicity?**\n\nWe have explored different metrics based on retrieval performance for reward. As described in Section 2.3, our pilot study indicates that retrieving relevant documents earlier and more completely is beneficial for correctly answering questions. When relevant information cannot be retrieved, the model is unable to produce the correct answer. Even when the model finds relevant information through multiple rounds of search, overly lengthy context can still lead to suboptimal results. Therefore, we need a metric that can comprehensively evaluate retrieval efficiency and completeness to help optimize the model. The table below presents our analysis of various metrics:\n\n| Metric | Focus | Usage scenarios of metrics | Comment |\n| --- | --- | --- | --- |\n| Recall  | Coverage of relevant documents | When minimizing missed relevant documents | Unsuitable. Only include coverage. |\n| MRR | Efficiency of finding the  relevant document\t | When quickly finding the relevant information matters | Unsuitable. Only focus on the ranking without comprehensiveness. |\n| NDCG | Ranking quality with relevance scores\t | When considering both relevance and ranking quality | Suitable. Considering both retrieval efficiency and accuracy. |\n\n\nRecall is based solely on the retrieval results and do not include evaluation of ranking and efficiency. MRR (Mean Reciprocal Rank) only considers the ranking of documents without taking into account those that have not been retrieved. However, we need to consider retrieving every relevant document.\n\nNDCG takes into account both the ranking and accuracy of all relevant documents, making it suitable as a reward for evaluating the efficiency and comprehensiveness of the agentic RAG. In addition, we conducted experiments on VRAG-7B, and the results are shown as follows.\n\n| Method | Overall Performance |\n| --- | --- |\n| Model-based Reward | 52.4 |\n| w/ Recall-based Reward | 53.7 |\n| w/ MRR-based Reward | 53.2 |\n| w/ NDCG-based Reward (Ours) | 57.1 |\n\n\nAs shown in the results, the reward based on Recall and MRR offers a slight gain. However, for RAG-specific tasks, our NDCG-based reward is more suitable as it provides a comprehensive evaluation of retrieval performance.\n\n> **Q2: Regarding the performance gap in the main results shown in Table 1, do they reflect the advantage of the data or the advantage of the method?**\n\nFor the performance of large models, both data and methods are equally important. Our work not only proposes methods for constructing high-quality data but also introduces novel and effective paradigms.\n\nBelow, we will demonstrate the effectiveness of our method with additional experiments:\n\n(i) Demonstrate the effectiveness of our method using the same training data: VRAG-RL and Search-R1-VL are trained with the same data of QA pairs. The performance gap shown in Table 1 of our manuscripts demonstrating the effectiveness of our framework.\n\n(ii) Demonstrate the effectiveness of our proposed paradigm by prompting the same model: We separately evaluate the trajectories constructed using the ReAct-based pipeline and the trajectories constructed using our visual perception paradigm, both with the same model. Through this experiment, we can evaluate the effectiveness of our paradigm without the interference of training data. The experimental results are shown in the table below.\n\n| Method | Accuracy (%) |\n| --- | --- |\n| Vanilla Pipeline | 58.9 |\n| Visual Perception Pipeline | 67.2 |\n\nThe experiments and analysis above have demonstrated the effectiveness of our method through controlling variables. We will update in the revision according to your highly valuable suggestions.\n\n> **Q3: How much does VRAG-RL improve over an SFT-only version before RL?**\n\nThe multimodal RL paradigm we proposed can achieve significant gains, as shown in the experimental results in the table below. \n\n| Method | Invalid Action Rate $\\downarrow$ | Finish Rate $\\uparrow$ | Overall Performance $\\uparrow$ |\n| --- | --- | --- | --- |\n| SFT | 9.4 | 84.2 | 48.9 |\n| +RL | 5.1 | 97.1 | 57.1 |\n\n\nAccording to the results, we believe that the improvement in end-to-end performance mainly stems from the understanding and utilization of tools. In fact, how agentic RAG retrieves more relevant information and avoids repetitive retrievals is the primary issue that needs to be addressed. By effectively utilizing tools, the model can then perform effective reasoning, which is also the fundamental factor that defines its performance upperbound. This is also explained in Section 3.3 and Table 3 of our manuscripts.\n\n> **Q4: About the sensitivity of performance to the α/β/γ weights.**\n\nWe experimented with many combinations of weights and conducted quantitative analysis during our research. \n\n$r_\\phi = \\alpha \\cdot r_{Ret} + \\beta \\cdot r_{Ans} + \\gamma \\cdot r_{Pat}$\n\nThe experimental results are shown in the table below.\n\n| α | β | γ | Overall Performance |\n| --- | --- | --- | --- |\n| 0.1 | 0.8 | 0.1 | 53.9 |\n| 0.2 | 0.7 | 0.1 | 57.1 |\n| 0.3 | 0.6 | 0.1 | 55.4 |\n\n\nIn practice, for the parameter γ, it can be set to a small weight like 0.1 when the model's instruction-following ability is good after SFT. During the actual implementation, when extracting the final answer and retrieval results, we will conduct a pattern validity check. \n\nFor the parameters α and β, when α is too large, performance decreases, as the model tends to search multiple times, resulting in lower confidence in the answers provided. Conversely, when α is too small, the insufficient activation of the model's search capabilities can lead to suboptimal results. We chose the parameter setting of α=0.2 and β=0.7 for the experiment."}, {"review_id": "wtIW4yHm46", "reviewer": "Reviewer_8qoA", "summary": "This paper proposes VRAG-RL, a reinforcement learning approach that trains Vision Language Models to either (1) generate an answer; (2) produce a visual perception action (i.e., cropping); and (3) produce a query to be sent to a search engine. These actions are preceded by instructed \"thinking\" as in a ReAct manner. \n\nThe authors conduct experiments on ViDoSeek, SlideVQA, and MMLongBench with QWen2.5-VL-3B/7B-Instruct models. They show that the VRAG-RL approach performs better than vanilla RAG, ReAct, and Search-R1 substantially.", "questions": "* It seems like only one image is retrieved/cropped in one particular turn, is it possible to extend to multiple images?\n* How is the performance of vanilla RAG/ReACT/Search-R1-VL when multiple images are provided at each turn?  \n* It seems like there is no explicit supervision signal for the visual perception action (cropping) except for the formatting reward. What experimental results suggest that the model is correctly identifying the region of interests other than end-to-end performance? Could you make a case that VRAG-RL is learning to effectively identify region-of-interest?\n* In Figure 6, what’s the x-axis ticks? \n* In Figure 5, what's the scale of the radar chart?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The inclusion of visual perception actions (i.e., cropping region-of-interest), is novel and suitable for the task. \n* The authors show RL can effectively train models to produce visual perception actions along with search queries, which provides a strong baseline for the field. \n* The paper is clearly written. \n\nWeaknesses:\n* Clarity of some results. Figure 6 is missing the x-axis ticks. Figure 5 is not interpretable as there are no scales to the radar chart.\n* Discussion of Top-K retrieval. In RAG approaches it is common to retrieve topK documents to answer the question. It seems like the paper focuses on the Top-1 case in each turn. It is not clear how the approach could extend to the more general TopK scenario. It is also unclear if baseline approaches are all implemented in the Top-1 setting. If so, the comparison could be improved as it is straight forward to extend to the Top-K setting in Vanilla RAG/ReAct (the QWen2.5-VL model family supports that). This could be a limitation for the VRAG-RL approach, but is not discussed by the author.", "was_revised": true, "final_justification": "I maintain a positive view on the strength of the paper and the author's response.", "rebuttal": "Dear Reviewer 8qoA,\n\nWe sincerely appreciate your insightful reviews and constructive suggestions, which are highly beneficial for  enhancing the comprehensiveness of our experimental results and the readability of our manuscripts. Your recognition of the strengths of our work has been a great encouragement to our team. We have made every effort to address all concerns in the past few days. Please see our responses to each point below:\n\n> **W1&Q4: Clarification regarding Fig. 6.**\n\nFor Figure 6, the x-axis represents the average action count, with scales labeled as 1, 2, 3, ... We use the number of answer actions to normalize the count of other actions. \n\nFrom a macro perspective, introducing visual perception actions can make reasoning more efficient, thereby avoiding excessively long and repetitive retrievals. We will update the revision to make the chart more readable according to your valuable suggestions.\n\n> **W1&Q5: Clarification regarding Fig. 5.**\n\nFor Figure 5, the scales of the radar chart represent the normalization of performance relative to the state-of-the-art method (with Visual Perception). Each concentric circle increases by 25% from the center outward. \n\nThis chart indicates that visual perception action indeed provides the model with a fine-grained perspective for visually rich information understanding. We will update the revision to enhance the clarity of the figures according to your valuable suggestions.\n\n> **W2&Q1: How VRAG-RL can be extended to handle top-k inputs as multiple images.**\n\nFrom a macro perspective, for the entire trajectory, VRAG-RL can be seen as sequentially inputting multiple images, making judgments and scaling each image individually. In contrast, inputting multiple images at once can be considered batch inference. Our current sequential process paradigm, which involves reviewing images one by one, can be regarded as a paradigm of test-time scaling.\n\nWe greatly appreciate your insightful perspectives for the multiple images in each turn. We have attempted to extend our Visual Perception Action to support multiple image inputs in a simple yet effective manner: Stitch multiple images together and then let the model output the region of interest. When images are stitched together and input as a single image (with the model being informed that the images are arranged from top to bottom, left to right), the model can identify each image and output the regions of interest at once. \n\nIn the experiment, we attempted input formats including 1x2, 1x3, and 2x2. We employed our original sequential paradigms to prompt the model to select the region of interest of the stitched images. The experimental results are shown in the table below.\n\n| Paradigms | Comment | Top2 (1x2) | Top3 (1x3) | Top4 (2x2) |\n| --- | --- | --- | --- | --- |\n| Select Region of Interest -> Answer | Align with training paradigm | 58.4 | 54.1 | 60.1 |\n\nBy analyzing the case, we think that the 1x3 method causes the input image ratio to differ from the pre-training domain, resulting in a decrease in the model's perception capability. In practical use, we recommend image ratios that are more balanced in terms of length and width, such as 1x2 or 2x2.\n\n> **W2&Q2: The performance of vanilla RAG/ReACT/Search-R1-VL when multiple images are provided at each turn.**\n\nBelow are the experimental results for multiple images provided at each iteration. All experiments utilize the Qwen2.5VL model with the same retrieval corpus, as shown in the table below.\n\n| Method | Top2 | Top3 | Top4 |\n| --- | --- | --- | --- |\n| Vanilla RAG | 32.5 | 36.1 | 37.2 |\n| ReAct | 33.4 | 35.2 | 36.9 |\n| Search-R1-VL | 40.2 | 46.1 | 49.4 |\n\n\nFor the baseline method, we conducted experiments by directly changing the number of input images in each turn. Regarding prompt engineering based methods, when comparing Vanilla RAG and ReAct RAG, we observed that as the number of input images increased, Vanilla demonstrated stronger performance than ReAct. This is because the excessively long context hinders the model's ability to follow instructions, making it challenging to provide coherent tool calls or effective final answers. However, Search-R1-VL addresses this issue; the trained model has a better instruction-following capability. Nonetheless, due to the low density of visual information, there is still a performance gap compared to VRAG-RL.\n\n\n\n> **Q3: A more in-depth discussion about VRAG-RL is learning to effectively identify region-of-interest.**\n\nWe can compare the consistency between the trajectory annotations made by mixed experts and the regions of interest output by VRAG-RL on the same dataset to verify whether the model is learning to identify the regions of interest. The experimental steps are as follows:\n\n(i) Use mixed-expert sampling and VRAG-RL to perform inference and obtain trajectories respectively.\n\n(ii) For each identical query, we match the same images retrieved and regions of interest within the trajectories.\n\n(iii) Calculate the mAP of the region of interest in the matched images.\n\nWe evaluate the performance of the models after SFT and RL separately. The experimental results are shown in the table below.\n\n| Method | mAP@50 $\\uparrow$ |\n| --- | --- |\n| SFT | 43.1 |\n| +RL | 49.2 |\n\nWe used metrics from object detection to verify whether the model can effectively learn to recognize regions of interest. In the experiment mentioned above, we validated that our model gradually aligns with the output distribution of large-scale expert models. The model's ability to identify regions of interest progressively improved, confirming the effectiveness of our approach in reducing model hallucinations.\n\nAlthough there is no supervision for region coordinates in our reward, actually, if the model can correctly focus on the regions of interest, the final answer will also be correct. The reward for the final answer is consistent with the model's ability to identify the region of interest, forming a supervisory signal for the model."}, {"review_id": "eJ4yWYzjCM", "reviewer": "Reviewer_hXKu", "summary": "This paper introduces VRAG-RL, a novel reinforcement learning framework for Vision-Language Models to handle visually rich information. It defines a visual perception action space for coarse-to-fine information gathering and designs a reward structure integrating retrieval efficiency, pattern outcome evaluation. Experiments on three benchmarks show VRAG-RL outperforms existing methods by 20%-30%. The framework enhances model's reasoning and retrieval capabilities through iterative interaction with search engines.", "questions": "1.The paper's ablation experiments on the reward design are insufficient. Table 2 does not clearly distinguish between Vanilla Reward and RAG-Specific Reward, and there are no adequate ablation experiments to demonstrate the effectiveness of the Retrieval Efficiency Reward and Model-Based Reward individually. Additionally, do the claims in 3.3 Analysis and Appendix A that the Model-Based Reward is better lack data support?\n\n2.The paper mentioned α + β + γ = 1 in Section 2.3, but lacks specific values for α and β and balancing strategy for different reward.", "limitations": "The efficiency of training and inferring.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1.The paper introduces Visual Perception Actions, enabling the model to capture fine-grained visual information in images. Ablation experiments validate the effectiveness of this action, demonstrating its critical role in enhancing visual RAG capabilities.\n\n2.The paper introduces Retrieval Efficiency Reward based on NDCG to enhance relevant image recall and reduce redundant searches.\n\n3.Extensive experiments and ablation studies across SlideVQA, ViDoSeek and MMLongBench demonstrate performance improvements over other methods.", "was_revised": true, "final_justification": "I will keep my positive score.", "rebuttal": "Dear Reviewer hXKu,\n\nWe sincerely appreciate your insightful review and the valuable insights you provided. Your constructive suggestions about the experimental analysis and reward function ablation are highly beneficial for highlighting our contributions. Your recognition of our work's strengths has been a great encouragement to our team. We have tried our best to address all concerns in the last few days. Please see our responses to each point below:\n\n> **Q1: A detailed ablation and explanation on distinguishing between Retrieval Efficiency Reward and Model-Based Reward, along with a comprehensive analysis and experimental results to support claims on Model-Based Reward.**\n\nAs shown in the table below, Vanilla Reward represents the vanilla used by Search-R1, which is a rule-based outcome reward (Exact Match). Moreover, we have added more fine-grained ablation studies based on Table 2 in our manuscript to further illustrate the contribution of our retrieval efficiency reward to the overall performance improvement.\n\n| Method | Overall Performance (Acc) |\n| --- | --- |\n| Vanilla Reward | 49.3 |\n| Model-based Reward | 52.4 |\n| Model-based Reward + Retrieval Efficiency Reward | 57.1 |\n\nTo further illustrate the impact of our RAG-specific reward on overall retrieval performance, we present more detailed experimental results in the table below. The retrieval performance gap between w/ Retrieval Efficiency Reward (α=0.2) and w/o Retrieval Efficiency Reward (α=0) serves as a supplement to Figure 4 in out manuscripts, further demonstrating the effectiveness of our module.\n\n| Method | Retrieval Performance (Recall@1) |\n| --- | --- |\n| w/o Retrieval Efficiency Reward | 55.9 |\n| w/ Retrieval Efficiency Reward | 60.3 |\n\nIn order to support claims on Model-Based Reward, we decompose the model evaluation into three aspects: Invalid Action Rate, Finish Rate, and End-to-End Performance, as shown in the table below.\n\n| Method | Invalid Action Rate $\\downarrow$ | Finish Rate $\\uparrow$ | Overall Performance $\\uparrow$ |\n| --- | --- | --- | --- |\n| RL w/ Rule-based Reward | 9.2 | 81.8 | 49.3 |\n| RL w/ Model-based Reward | 6.7 | 94.2 | 52.4 |\n\nIn our agentic RAG task, the training effect of rule-based reward affects the model's performance by influencing the sampling distribution and sampling efficiency. Specifically, rule-based rewards make the model prefer learning shorter answers and shorter trajectories (which can easily achieve higher rewards) because rule-based rewards cannot effectively evaluate longer and more complex answers. However, in the test set and real-world environment, the model's final Finish Rate may decrease, and the Invalid Action Rate may increase.\n\nApart from the experimental results, during training we analyzed the advantages of model-based rewards compared to rule-based rewards (Recall/Exact Match/ANLS), which are mainly reflected in: (i) The practicality of the final model: The model provides answers in the form of complete sentences or paragraphs rather than short answers consisting of one or two words; (ii) Dataset adaptability: The current datasets contain multiple key points instead of phrases or words, and using model-based rewards allows training on more datasets; (iii) Rule-based rewards are difficult to evaluate at the semantic level; (iv) The model is more prone to hacking rule-based rewards, which can ultimately lead to repetitive outputs or training collapse.\n\n> **Q2: About the specific values for α and β, and the strategy for balancing different rewards.**\n\nWe experimented with many combinations of weights and conducted quantitative analysis during our research. \n\n$r_\\phi = \\alpha \\cdot r_{Ret} + \\beta \\cdot r_{Ans} + \\gamma \\cdot r_{Pat}$\n\nThe experimental results are shown in the table below.\n\n| α | β | Overall Performance |\n| --- | --- | --- |\n| 0.1 | 0.8 | 53.9 |\n| 0.2 | 0.7 | 57.1 |\n| 0.3 | 0.6 | 55.4 |\n\nIn practice, for the parameter γ, it can be set to a small weight like 0.1 when the model's instruction-following ability is good after SFT. During the actual implementation, when extracting the final answer and retrieval results, we will conduct a pattern validity check. \n\nFor the parameters α and β, it is better to prioritize the final answer reward and adjust the parameter values within a smaller range. This conclusion can be drawn from observing the results of the first and last lines. When α is too large, performance decreases, as the model tends to search multiple times, resulting in lower confidence in the answers provided. Conversely, when α is too small, the insufficient activation of the model's search capabilities can lead to suboptimal results. We chose the parameter setting of α=0.2 and β=0.7 for the experiment."}]

Author rebuttal:

Meta-review: The paper proposes a reinforcement learning framework that enhances vision-language models for retrieval-augmented generation (RAG).

Reviewers initially raised concerns about novelty, clarity of contributions, reward design, and missing ablations or comparisons. These issues were largely addressed in the rebuttal through additional experiments (reward-model swaps, sensitivity analysis, new ablations), extended comparisons with Vision-R1, Perception-R1, and VLM-R1, detailed failure case taxonomy, and clarifications of figures and design choices.


The strengths of the paper are consistently highlighted. Reviewer FHAe praised the well-motivated visual action space and clear empirical gains, Reviewer jJNP recognized significant performance improvements and effective use of visual perception actions, Reviewer 8qoA emphasized the novelty of visual perception actions, and Reviewer hXKu commended the integration of retrieval efficiency reward and extensive experiments showing strong performance improvements. All reviewers positively support this paper. Therefore, the AC recommend acceptance.
Decision: Accept (poster)