VGR: Visual Grounded Reasoning
Authors: Jiacong Wang
Zijian Kang
Haochen Wang
LiangXiao
Ya Wang
Jiawen Li
Bohong Wu
Ran Jiao
Haiyong Jiang
ChaoFeng
Jun Xiao
Venue: ICLR 2026 Poster (2026)
Keywords: VLM
MultiModal
Cot
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We presents VGR, an MLLM with fine-grained visual perception, addressing language bias in traditional multimodal reasoning by detecting relevant image regions for precise answers, using reasoning data and an inference pipeline with visual replay.

Abstract: In the field of multimodal chain-of-thought (CoT) reasoning, existing approaches
predominantly rely on reasoning on pure linguistic space, which inherently suffers
from language bias and is largely confined to math or science domains. This narrow
focus limits their ability to handle complex visual reasoning tasks that demand comprehensive understanding of image details. To address these limitations, this paper
introduces VGR, a novel reasoning multimodal large language model (MLLM) that
can replay the visual memory during thinking just like humans. Unlike traditional
MLLMs, VGR first thinks the question and detects relevant regions that may help
solve problems, then, the visual memory from the critical area is extracted to assist
reasoning. To achieve this, we curate a large-scale SFT dataset called VGR-SFT
that contains reasoning data with mixed vision grounding and language deduction.
This teaches VGR to think and actively choose grounding areas for key information before answering, and we propose a dynamic visual memory replay stage to
integrates the corresponding information into the reasoning process, enhancing
multimodel comprehension. Experiments on the LLaVA-NeXT-7B baseline show
that VGR achieves superior performance on multimodal benchmarks requiring
comprehensive image detail understanding. Compared to the baseline, VGR uses
only 30% of the image token count while delivering scores of +4.1 on MMStar,
+7.1 on AI2D, and +12.9 improvement on ChartQA. The data is available at https://huggingface.co/BytedanceDouyinContent/VGR.

Published as a conference paper at ICLR 2026
VGR: VISUAL GROUNDED REASONING
Jiacong Wang 1.+ Zijian Kang²* Haochen Wang" 1.2 Haiyong Jiang" Jiawen Li² Bohong Wu² Ya Wang? Jiao Ran
Xiao Liang2t
Chao Feng2t Jun Xiao
"Scoool of Artificial Intelligence, University of Chinese Academy of Sciences "ByteDance Inc.
Ihaiyong- - jiang, xisojun) Bucas ac.cc chacfeng, zzebytodanee.com
wangjiacong20emali ucas
ABSTRACT
In the field of multimodal chain-of-thought (CoT) reasoning, existing approaches predominantly rely on reasoning on pure linguistic space, which inherently suffers from language bias and is largely confined to math or science domains. This narrow focus limits their ability to handle complex visual reasoning tasks that de- mand comprehensive understanding of image details. To address these limitations, this paper introduces VGR, novel reasoning multimodal large language model (MLLM) that can replay the visual memory during thinking just like humans. Un- like traditional MILMMs, VGR first thinks the question and detects relevant regions that may help solve problems, then, the visual memory from the critical area is extracted to assist reasoning. To achieve this, we curate a large-scale SFT dataset called VGR-SFT that contains reasoning data with mixed vision grounding and language deduction. This teaches VGR to think and actively choose grounding areas for key information before answering, and propose a dynamic visual memory replay stage to integrates the corresponding information into the reasoning process, enhancing multimodel comprehension. Experiments the LLaVA-NeXT-7B baseline show that VGR achieves superior performance on multimodal benchmarks requiring comprehensive image detail understanding. Compared the baseline, VGR uses only 30% of the image token count while delivering scores of +4.1 1 on MMStar, +7.1 on AI2D, and +2..9 improvement on ChartQA. The data is available at https::/nugginggace co/p
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable reasoning capabilities, particularly in complex problem-solving scenarios such as mathematical deduction and scientific analysis. Systems like OpenAI-ol (OpenAl 2024) and DeepSeek-R1 (Guo et al.) 2025a) exemplify this progress, achieving state-of-the-ar performance on benchmarks requiring logical inference and algorithmic thinking, where the crux seems to be large-scale Reinforcement Learning (RL) (Sutton et al. 1998
Wang et al. 2025c Hao et al. 2025 with verifiable rewards (Shao et al. 2024b).
Recent advancements in multimodal reasoning have sought to extend these capabilities to vision- language tasks. often by distilling knowledge from powerful LLMs into multimodal architec- tures (Huang et al., 2025 Dong et al.] 2025) Yang et al. 2025: Wang ct al.) 20255; Dong et al.) 2024 Ren et al. (225). While promising results have emerged in math and science domains, studies consistently reveal critical limitation: language bias (Jiang et al., (2025) Wang et al., 2024d
Xu al. 2024), i.e., over-reliance on linguistic priors leads to systematic performance drops in perception-heavy tasks.
To address this limitation, we propose Visual Grounded Reasoning (VGR). Instead of reasoning solely in the linguistic space, we argue that models should perform targeted visual analysis during reasoning to identify key regions of interest that are directly relevant to the question. VGR extends the conventional text-only chain-of-thougle to multimodal reasoning trace, allowing the model to
Equal Contribution. t Project Lead. 1 Corresponding Author.

---

Published as a conference paper at ICLR 2026
selectively retrieve visual memory on demand, thereby enhancing the accuracy and interpretability of multimodal reasoning. This approach mirrors human cognition: we reason not only through language but also by recalling and mentally simulating visual content.
Specifically, we design a novel self-driven visual memory replay module to retrieve and replay the visual memory. The selective replay is controlled by the model via predefined special signal: when the model requires visual grounding during reasoning, it generates a replay signal, prompting VGR to fetch corresponding visual tokens from the visual memory to augment its reasoning process. We im- plement VGR with a novel architecture, the visual memory is constructed from visual representations of nigh-resolution crops, and pooling strategy is adopted further enhance efficiency, the whole design makes the model even more efficient than conventional MLLMs.
To learn this format of instruction, we construct a new visual grounded reasoning dataset embedded with visual clues, marking the attempt to explicitly model visual region attention in multimodal
reasoning. Unlike prior works that either rely on text-only chains-of- offtou-of-thought (Xu et al. 2024 Wang et al.
2024d) for multimodal tasks or enforce rigid multi-turn interactions (Wu & Xie 2024; Shha
ct al. 2024a| Qi et al., 2024 Xiao et al. 2024), our dataset empowers models to autonomously attend to arbitrary visual regions during reasoning. Notably, all grounding areas in the dataset are voluntarily generated by the model itself, avoiding manual annotation bias. To construct this dataset, we first use an existing model to generate cold-start dataset, which is then refined via a rejection sampling pipeline and further expanded using annotations from a custom-traiin -arsine-traieed annotation model.
We conduct extensive experiments this novel framework, results under fair comparison show that our method outperforms the baseline on multiple datasets, such as +6.4 on the MMStar (Chen et al.) 2024) and 1 ChartQA (Masry al. 2022), while utilizing only 0.3x visual tokens. These findings not only underscore the effectiveness of visual memory replay in improving visual-linguistic reasoning, but also establish a new paradigm for enhancing computational efficiency in MLLMs.
In summary, our contributions are threefold:
We introduce VGR, a new visual reasoning framework for MLLM, which enables the model dynamically attend to visual content during inference, enhancing reasoning accuracy with fine-graines visual details.
We build the visual grounded reasoning data with visual cues, the dataset empowers models to freely attend arbitrary visual memory during reasoning, contrasting with prior works relying on text only chains of thought or rigid interactions.
Extensive experiments on VGR demonstrate that our model outperforms the LLaVA-NeXT baseline in downstream tasks while using only 0.3x the number of image tokens. As the quantity of image tokens increases, this performance gap becomes even more pronounced.
2 RELATED WORKS
Multimodal Large Language Models. Pioncering MLLM frameworks like Flamingo (Alayrac al. 2022) and BLIP-2 (Li et al. 2023a) establish foundational architectures for cross-modal understanding using cross-attention. Subsequently. LLaVA (Liu et al.] (2023c) emerged as a more efficient, scalable, and modular framework, combining vision encoder with large language model through a simple linear projection layer. Its instruction-tuning paradigm demonstrated competitive performance, emphasizing the power of aligned vision-language supervision. Building on this idea,
recent advancements (Liu et al., 2024at Wang et al. 2024c Bai et al. 2025 Wang et al.) 2024a 2025a Lu et al. 2024| Wu et al. 2024 Zhu al.] 2025 Lei al. 2025] Wang al.) [2024b Gu et al. 2024 2025) push the boundaries of efficiency, scalability, and task complexity. For instance, Qwen2.5-VI. (Bai et al., 2025) integrates dynamic resolution, and Intern VI.3 (Zhu al., (025)
emphasizes the importance of larger-scale native multimodal pretraining. These models serve as strong baselines for a variety of real-world applications.
Reasoning MILMs. The groundbreaking success of advanced reasoning LLMs like OpenAl- ol (OpenAl 2024) and DeepSeek-RI (Guo et al.) 2025a) has inspired efforts to extend such ca- pabilities into multimodal domains. Prior attempts (Wang et al.) 2024d Diang et al. (025) found that CoT prompting even brings performance degradation for perception-head
tasks due to the accumulation of language bias. Therefore, current approaches mainly focus incentivizing MILMMs
2

---

Published as a conference paper at ICLR 2026
to solve difficult math and science problems with image inputs. For instance, Vision-RI (Huang et al. 2025) first leverages MLLMs to generate detailed captions for provided images and then queries DeepSeek-RI (Guo et al. 2025a obtain a dataset for cold initialization. Other approaches, such as VLM-RI (Shen et al. 2025) and Visual-RFT (Liu et al. 2025), directly adopt GRPO (Shao et al.
b024bb for open-ended visual grounding, where RI consistently outperforms SFT. In parallel, another line of work enhances high resolution grounding via supervised CoT training: Zoomeye (Shen et al.
2024 leverages human-like zooming with tree-based exploration, while Chain-of-Spot (Liu et al.
2024c) employs interactive reasoning for spot based visual search. These methods yield notable gains
on benchmarks such as HR-Bench (Wang al. 2025d) and V* Bench (Wu & Xie [024], highlighting the complementary role of CoT style supervision for perception intensive multimodal reasoning tasks. This paper presents an alternative methodological approach that complements existing perspectives. Our framework seeks to incentivize the wwwpndding-tee-answweee
capability in MILMs,, requiring the model to systematically develop two critical competencies: (1) frequent autonomous selection of task-relevant image regions through deliberate focus mechanisms, and (2) contextualized reasoning based on these visually grounded observations.
3 VISUAL GROUNDED REASONING
In this section, we elaborate on the framework and model of VGR, in Figure 1 To unlock the visual grounding reasoning capabilities, we introduce novel visual memory replay mechanism, allowing the model to attend to arbitrary image regions by retrieving corresponding image tokens during the reasoning on the fly.
Visual Grounded Reesoring
Hewe:
Taplay
Visual Encoden
And fndd diffeower
Drignal Image
Taplay
day ditheeir
Briage
tolken) Plepie Sagket
Jinage Toient Mepsaged besesl
[Tee Yavel
Letsltck un oree cwontettlluuutteee
Iv uns..
Model
1800 vilat 876-200 300 sillce cstaar.
Figure 1: Overview framework of our method. On the left side of the figure, we apply the AnyRes strategy to the original image while maintaining visual memory pool that stores detailed visual features. When a visual memory replay signal is detected, VCR retrieves image tokens from this visual memory pool, thereby enriching the visual clues available for reasoning. On the right side of the figure, we present an example of VGR in operation: it enables the MLLM to inspect key regions on demand.
The dynamic visual memory replay module of VGR retrieves image tokens generated by the vision encoder and adapter. Leveraging LLaVA's AnyRes approach for high resolution image encoding, we first resize the input image dimensions H W where H and W divisible by p 336. The resized image E HR MxxWxX x3 is then partitioned into non-overlapping x patches:
P.
P P (i+ 1),p* (j + 1)].
(1)
The corresponding image tokens are processed by the vision-encoder and adapter, yielding token embeddings in the language space:
T F Fadapter (Frision (Ptz)) Ex
where 8 denotes the size of the vision patch and e denotes the channel number of latent features. Like in LLaVA, the image tokens from each crop are flattened to ID sequence and fed in the LLM separately. We further concatenate the feature of each patch representation to unified image feature R 2 for later use, which serves the visual memory.
3

---

Published as a conference paper at ICLR 2026
The dynamic visual memory replay mechanism relies fine grained visual feature for retrieval, to preserve high resolution visual details while maintaining training and inference efficiency, we propose expand-then-comee comp-ces strategy. Specifically, scale up the maximum crop count of LLaVA'ss AnyRes approach from 4 to 16 patches and introduce a vision feature compression layer using 2D pooling. To balance resolution and computational cost, adopt x 2 pooling for snapshot compression and 4 x 4 pooling for high resolution AnyRes token compression empirically.
Compared to the baseline, which employs maximum 2,880 tokens per image (576 tokens per shot, including I snapshot image and 4 AnyRes crops), VGR achieves superior efficiency. Our method uses only 144 tokens for the snapshot and a maximum of 720 tokens for high resolution crops, reducing token usage by 70% while expanding supported resolutions by 5x. This design guarantees VGR to maintain fine grained visual information for retrieval while lowering computational overhead.
To enable the MILM selectively attend to specific visual regions, we introduce replay control signal for the model. Each replay region defined via a grounding area notation: <sot> [x1, yl, x2, y2] <eot>, where [x1, yl] denotes the top-left corner and [x2, y2] the bottom-right corner of the region. The visual tokens will be retrieved once such signal is detection. The MLLM encouraged to generate these signals during inference in demand to extend visual clues.
During inference, VGR monitors the model output and, upon detecting signal token <eot>, parses the preceding content to extract the region coordinates. If valid, the model retrieves image tokens corresponding to this region from the feature map S and appends them after the control signal. Specifically, for region defined by coordinates (#1, /1) and (12,, 32), the dynamic visual memory replay module extracts the corresponding feature patch R 12 the extracted feature map Re1.91.22 is then down-sampled with 2 x pooling and flattened into ID token sequence. They are fed into the LLM immediately following the visual replay signal token.
Implementing supervision for the dynamic visual memory replay is straightforward. We simply add the retrieved image tokens R 222.22 to the training sequence after the replay signal and optimize the model with the standard supervised fine-tuning. The signal tokens as well text tokens are supervised with cross-entropy loss, while all image tokens (both from the original input and the replay regions) are excluded from the loss computation. To further enhance the model's 's region selection capability, we introduce an auxiliary detection loss that encourages accurate area predictions.
The detection loss important because coordinates for retrieval are actually represented as num- bers, Lae operates as straightforward regression task to precisely align spatial locations, since cross-entropy on tokenized boxes may struggle with quantization errors and discontinuous predic- tions. Therefore, combining both allows the model to leverage continuous regression for accurate localization. Specifically, the detection loss is a combination & loss and GloU loss:
Lett L BlaioU,
where 4 Loss measures the absolute difference between predicted bounding box coordinates and ground truth. We set 3 2 following common practices. For bounding box parameterized by center coordinates (Iee yo), width w, and height h, the formula is:
41 = l xel + lūe yel + l w| + +1h ht/
where Se; Hc, à h are predictions. GloU loss is computed by:
tGo = 1 InterArea
C UnionArea
(5)
UnionArea
C
where C' is the smallest box enclosing both predicted and ground truth boxes:
C C= x()): (vē - -v),,
(6)
where x2. = min(a1, ), 2². = max(x2, .2), vr. min(y) 21), = max(y2, The detection
head we utilized is a small MLP that maps the hidden states of <eot> to a 4-dimensional box.
4 VISUAL REASONING DATA CURATION
VGR learns to the visual reasoning through our reasoning data with replay signal an visual memory, with the proposed three stage data construction pipeline shown in Figure The cold start data is generated with an existing large instruction model and further refined with reject sampling. Then, we train annotation model annotate data from more domains.
4

---

Published as a conference paper at ICLR 2026
Originol Instruction bata
with Parsaiing
with Tectnol Ireatatio
Rowrdt5rg
Rama
224
546 MLU
153k
Figure 2: Overview framework of VGR data pipeline. We use existing large MLLM to annotate initial cold-start data, and train a smaller annotator model to scale up the amount of training data. A reject sampling and refinement pipeline is adopted improve the data quality.
Quectioo:
this juticial book?
Assuer:
queation tion asic the book Jusaccal.
Jusicial books are related to Taw. The
What the 1lashy layer the truit wall collec?
A. Locale
Lasatoon
Ouestion:
C. Stons
side the bookrace?
R. Pesorerp option's letter from
"International Taxation Lnciictes It dests with fox-related togal watters poremed by Lave The puo iiher <tots (12, 340, 182,
Ansver the question using single verd
directly.
Answer:
Ansver:
4091xcot>
the question side is the bookcase?", ue Tank an the
<thünks
Pocasing the aipnifkcon region cssts toa, cabjorsto
The fruit wall is the pertears wnich tas thrre layere:
epocarp (nuter), mesocarp (widete), and enderarp (lener). To
in Foundetton Prees, kaom legx
is a custicial bock, Xlst, being part of the viot=ib, 237, voli
Interaaiinnll
dlagron, the aerocarp 15 cieara Tabeted the sidste in serrass, not the treat vall. Optior (ezzcarpi is the auter layer endscars. the correct </tsing>
seed- compartennt
"Concepts Tnetight Gertas" tapt les
books and other itees on the right side the imago. This, the bootcous on the Finel arower: Right
(Sture) the
aistesres.
w/thiake
Figure 3: Example of training data in VGR-SFT.
4.1 COLD-START WITH INSTRUCTION MODEL
The initial instruction data with replay capabilities is generated using an existing vision language model. Specifically, given image and corresponding question, the model is prompted to generate both reasoning chain and an answer. Concurrently, we require the model to localize all key regions in the image relevant the answer and explicitly reference these regions before describing their content. These key regions are designated as replayed areas during training. We adopt a 72B huge MLLM (Bai et al. 2025) as the cold-start model, due its exceptional instruction following capabilities, output diversity, and strong performance in both object detection and visual reasoning tasks. To standardize the annotation format, we prompt the model to encode detection results in JSON, which includes bounding boxes and semantic labels for each key region.
4.2 REJECT SAMPLING
Following the recent advances in RL (uuo et al. 2025a). we propose a similar reject sampling pipeline for valid data selection. First, employ Format Verification to ensure answer parseability. This involves two checks: (1) verifying that answers can be extracted by locating the designated "Final Answer" section; (2) ensuring bounding boxes and labels are formatted in valid JSON. Next, Correctness Verification assesses the accuracy of answers derived from reasoning chains. For closed ended tasks (c.g., OCR, MCQ), we use ANLS (Average Normalized Levenshtein Similarity) to quantify correctness by comparing generated answers with ground truths. For open ended tasks, we leverage a MILMM to semantically align reasoning chains with reference answers. Incorrect answers are discarded, while inaccurate ones undergo rewriting: the final answer is replaced with ground truth, and reasoning chains for open ended tasks are iteratively revised for coherence. Finally, Visual Grounding Verification validates the correctness of visual replay areas. During data preparation, each visual replay area is annotated with a bounding box and semantic label. We crop these areas and use MLLM to check alignment between cropped content and annotated labels. Additionally, we intentionally expand bounding box areas to encourage the trained model to retain contextual information during reasoning, enhancing its ability handle complex visual semantic dependencies.
5

---

Published as a conference paper at ICLR 2026
4.3 DATA SCALING WITH ANNOTATION MODEL
During the reject sampling, we notice the cold-start data generated by the existing instruction model exhibits high rejection rate and slow generation speed. To address these limitations, we train an annotation model using the cold-start data that passes the reject sampling pipeline. Empirically, we adopt a smaller 14B MLLM (Zhu et al. 2025) as annotation model, we augment it with the cold-start data, we also use reasoning data from the Open-R1 distilled dataset (Face) 2025) to generalize the conventional reasoning ability to our visual reasoning task. With the knowledge and pattern learned from cold-start data, the smaller annotation model significantly improves the pass rate from 14% in cold-start to 40% and also speedup annotation for 3.2x times. This allows us to scale the amount of training data with low cost.
4.3.1 TRAINING DATA
In the last step, we refine the annotated data that passes through the reject sampling pipeline, MLLM is used to revise the reasoning chains, enhancing reasoning robustness. The refinement aligns the data with our predefined template while eliminating ambiguous redundant content. The refined data is subsequently utilized to train the final reasoning model, ensuring its capacity to generate structured and coherent responses. The final training data VGR-SFT is curated from VA's official training data, which aligns strictly with the baseline in fair data comparison, where the composition of the data is shown in Table and examples are shown in Figure 3 You can find more details in the Appendix D
Table 1: The number of data generated from each dataset.
Method
Data Size
Data Type
A12D (Kembhavi et al. 2016
12.5k
ScienceQA
LLaVA-COCO (Liu et al.
2023b
12.3k
General VQA
GQA (Hudson & Manning
2019
39.2k
General VQA
QhartQA (Masry et al. 2022
11.2k
OCR
DVQA (Katie et al. 2018)
25.2k
OCR
Voc (Mathew el al.
2021
6.0k
OCR
OCRVQA MMishra et al.
2019
51.6k
OCR
Total
158.1k
5 EXPERIMENTS
5.1 EXPERIMENT SETTINGS
We validate VGR on LLaVA-NeXT (Lu et al. 2024b) setting following the recent practices, which is a
well-known and fully open-sourced baseline for multimodal training from scratch. The visual encoder
is CLIP-ViT-L/44G 336 (Radford et al. 2021) and the base LLM is Vicuna-v 1.5 series (Chiang al. 2023), including 7B and 13B versions. Following (Liu al.) 2024b), VGR has two training procedures: pre-training and supervised fine-tuning, The pre-training data is LLaVA -58K (Liu et al. 2024a), while the fine-tuning data the combination of LLaVA-NeXT-770K (Liu et al., 2024b) and our self-constructed 158K data. Notably, to ensure a fair comparison, all datasets constructed are derived from the original SFT data of VA-Nexx without introducing any additional data. The LR for pre-training stage is set to le-5 and 2e-5 for fine-tuning stage with Vicuna-7B. We set the learning rate ViT to 1/10 the base learning rate follow the LLaVA-NeXT's setting.
5.2 COMPARISON WITH EXISTING METHODS
We compare our VGR with a wide range of existing vision-language models on various benchmarks,
including MMStar (Chen et al. 2024); ChartQA (Masry et al.] 2022); Doc (Mathew al.
b02Tb: TeX (Singh et al. 2019%; InfoQA (Mathew et al. 2022); A12D (Kembhavi et al. 2016): Real RealWorldQA (Grok 2024); POPE (Li et al. 2023b). For clarity,
note that Sample represents
the image token compression or downsampling scheme used, while Vtoken indicates the maximum
number of image patch tokens. The top results highlighted in bold. All results are drawn either

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with existing vision-language models on various vision-language benchmarks.
Method
DownSample #Vtoken
LLM
MMS*
Chart
Doc
Text
Info
A12D
RWQA
POPE
Qwen-VLL VL-CChat--CB (Bai et all
2023
Cross-Altn
1024
Qwen-7B
34.5
66.3
62.6
61.5
57.7
49.3
74.9
Visual CaT (Shao ela 20245
Vicuma
66.9
86.5
DeepSeck-V (Gi al. 2024
Conv2D
576
DeepSeck-78 7B
40.5
59.1
64.9
65.3
54.2
85.6
LLaVA-v1 era 20250
576
Vicuna-BB
33.1
18.2
28.1
46.1
25.8
54.8
54.8
85.9
LLaVA-NeXI-7B (Liu 20MI
No
2880
Vicuna7BB
37.6
54,8
77.4
64.9
37.1
66.6
57.8
86.5
NeX
2x214x4
864
Vicuna 7B
37.2
58.7
70.2
60.5
34.7
68.5
56.8
87.8
VGR-7B
2x2 2x214x4
864
Vicuna-BB
41.7
67.7
73.7
63.9
39.8
73.7
59.8
88.2
VGR-7B
2x212x2
3024
Vicuna-7B
43.6
72.8
79.9
65.9
42.9
73.4
87.8
Table Ablations on different backbone and high resolution benchmark. Abbreviations in the table correspond to the following models: Qwen2.5 refers to Qwen2.5-7B-Instruuee siglip to SigLIP- 10400M//44:33 384, Intern ViT to Intern Wwt-11000///144448820000 and Vicuna to Vicuna-7B-v1.5.
CLT CLIP-ViT-L/4@3336
Training Settings
V* Bench
HR-Bench8K
MMStar
ChariQ
TexiVQA
RWQA
A12D
Swen2.5+SSiLL..
55,5
44,9
$1.6
64.0
62.0
75.6
63.3
Owen2.5+SSgLIIPa.
67.5 12.0
56.11
11.2
2.5
74.2*
10.2
3,5
77.9t 2.3
65.41
2.1
Wwen2.5+InternVi VIT
56.0
45,2
51.4
75.2
68.2
76.0
61.7
69.8*
13.8
58,31
13.1
5..21
2.9
3.5
79.4f
3,4
64.9t 3.2
Vicuna+CIIIP
56.4
41.1
37.2
58.7
60.5
68.5
56.8
Vicuna+Cl.P+VGGR
67.71 11.3
52.91 11.8
43.6
1..6
72.8
14.1
6..91
73.4*
4.9
59.51 2.7
from the original papers from the official reproduction results reported by LMMs-Eval Zhang et al., 2024), whereas our results are consistently obtained using LMMs- In particular, denotes reproduction setting with a maximum of 20 local images using LLaVA-Nee. (Liu et al. 2024b)
and visual memory feature pooling (2x2 for the hase crop and 4x4 for local crops), with replay visual memory features further processed by 2x2 pooling.
As shown in Table 2 our VGR consistently outperforms strong alternatives, including Qwen-VL Chat (Bai al., 2023), Visual CoT (Shao et al. 2024a), DeepSeek--L--T (Lu al. 2024). LLaVA- v1.5-7B (Liu et al. 2023a), and LLaVA-NeXT-7B (Liu et al., 2024b). In particular, it achieves the best results on benchmarks requiring fine- mna-grained comprehension of high-resolution images. Moreover, when taking into account the average number of visual tokens, VGR delivers superior performance with 0.3 x visual tokens compared to the original LLLVVA-eeT, suggesting that focusing the model on specific regions is substantially more effective than merely increasing the number of visual tokens. As the number of image tokens further increases, this performance gap becomes even more pronounced. The comparison between VGR and the Zoomeye method in terms of performance and inference cost is available in Appendix A.2
5.3 ABLATION STUDIES
Ablations on more backbones and dataset. In Table 3 we present experiments designed to further verify the generalizability of the VGR. We replaced key model modules (including the visual encoder/ViT backbone and base language model/LLM) while strictly upholding experimental fairness: we used the same dataset throughout (with no additional data introduced) and only adjusted the LLM and ViT backbones. In the table, the last section (configured with ficuna-7B-vll and CLIP- ViT-L/14 336) aligns with the original architecture of LLa LLaVA-NeeT--7 (serving baseline). The experimental results confirm that our VGR framework not only delivers consistent performance improvements across diverse visual encoders, base L.Mss, but also improve fine grained high resolution benchmarks (V* Bench (Wu & Xie 2024) and HR-Bench 8K (Wang et al. 2025d)) that need visual grounding capabilities.
Ablations different data formulations. The ablation study in Table 4 demonstrates that the visual reasoning capacity requires both grounding boxes and reasoning. When either component is removed. whether by eliminating visual memory (w/o Memory) disabling the reasoning process (w/o Reasoning), performance consistently degrades. This suggests that while each component makes partial contribution, the two components can complement each other to achieve the best.
7

---

Published as a conference paper at ICLR 2026
Table 4: Ablations on different data formulations. 'indicates our reproduction on same setting with pooling. "w/o Memory" indicates that only the reasoning process is preserved without grounding and replay. "w/o Reasoning" means we remove the reasoning process.
Method
MMStar
ChartQA
DocVQA
TextVQA
InfoVQA
A12D
RWQA
POPE
LLaVA-NeXT-7BB
37.2
58.7
70.2
60.5
34.7
68.5
56.8
87.8
VGR-7B
41.7
67.7
73.7
63.9
39.8
73.7
59.8
88.2
w/o Memory
39.7
66.2
73.2
39.3
72.7
60.6
87.5
w/o Reasoning
39.3
59.6
72.5
61.9
38.5
72.8
59.3
87.8
Table 5: Ablations on public available CoT data. We evaluate different dataset on our setting
(indicated by 1), an extra post-training stage is added for LLaVA-Cor and MMPR following their recommendations.
Data
SFT
Post-Train
MMStar
ChartQA
DocVQA
ScienceQA
LLVAA-NeXTI
770K
37.2
58.7
70.2
70.3
LLaVA-CoT
770K
100K
39.6
58.8
64.4
76.5
MMPR
770K
660K
40.7
55.1
68.3
82.1
VGR-7B
770K + 158K
41.7
67.7
73.7
70.4
Compare with public available CoT data. We compare the effectiveness of our reasoning data (which explicitly utilizes regions of interest) against vanilla reasoning datasets such as LLaVA- CoT (Xu et al. 2024) and MMPR (Wang al. 2024d). As shown in Table [5] the model with region-offinterest guidance focuses more on relevant visual areas, leading to improved overall performance. In contrast, direct adoption of complex reasoning datasets like (Xu et al. 2024 Wang et al. 2024d) yields results even worse than the baseline. This may stem from the accumulation of language bias during multimodal reasoning, highlighting that appropriately integrating visual features of regions of interest significantly aids accurate inference.
Ablations on detection loss. Table dh, we study the effectiveness of the auxiliary detection loss. Since boxes are represented by floating point coordinates normalized to the [0, 1] range. Lett operates as a straightforward regression task precisely align spatial locations, since cross entropy on tokenized boxes may struggle with quantization errors and discontinuous predictions. Therefore, combining both allows the model leverage continuous regression for accurate localization.
Ablations on dynamic visual memory replay. In Table we systematically evaluate the efficacy of dynamic visual memory replay through ablation studies. Results show that excluding dynamic visual memory replay where the model merely outputs regions of interest without incorporating corresponding image features into the LLM input sequence leads to significantly limited performance improvements. This highlights the critical gain from integrating image features of boundary regions into the reasoning process, as it enables the model to leverage fine grained visual details for more accurate predictions.
Ablations on different reasoning data type. In 6 we analyze the performance differences across reasoning data of varying types. Using raw data after annotation during supervised fine-tuning introduces longer contexts, but this also makes the model prone to make mistakes, which does not benefit general question answering. In contrast, after summarizing and condensing them into relatively shorter rewritten data, the data and reasoning process are less confusing, therefore enabling the model develop stronger grounded reasoning abilities.
Ablations on diffferntt replayed strategies. In Table 7 we investigate the trade-off between pooling performance by differentiating the pooling steps for base images, local images, and Replay visual memory, as these components exhibit distinct levels of importance. The base image, which encapsulates the most comprehensive understanding of the entire visual content, demands a balance between global context and spatial detail. Local crops. while useful, often contain redundant information due to overlapping regions, justifying coarser pooling. Dynamic visual memory replay, however, represent specific regions of interest (Rols) critical to task task-solving and thus require finer- grained feature preservation compared to standard local crops. Empirically, the optimal configuration employs 2x2 pooling for both base and Replay visual memory retain critical details, while applying
8

---

Published as a conference paper at ICLR 2026
Table Ablations on each component, including (a) the introduction of detection loss, (b) whether to apply the dynamic visual memory replay after predicting bounding boxes, and (c) the type of reasoning data. By default, we enable detection loss and dynamic visual memory replay with
maximum of 20 local crops.
(a) Detection loss.
(b) Visual memory replay.
(c) Reasoning Data Type.
Lde: MMStar
ChartQA
DocVQA
Replay MMStar ChariQA
Type
MMStar
ChartQA
A12D
39.8
65.5
72.8
39.7
66.2
Annotated
40.7
64.5
71.7
41.7
67.7
73.7
41.7
67.7
Rewritten
41.7
67.7
73.7
Table 7: Ablations different replayed strategies. Different pooling strides for each type of image are important. We utilize different pooling size for the final image feature and Replayed image feature. We also use visual memory reduce the costs of training and inference.
Base
Local Replayed
#Crops
#Memory
#Vtoken
MMStar
ChartQA
Doc VQA
TextVQA
InfoQA
2880
37.2
58.7
60.5
34.7
2x2
4x4
2x2
Yes
288+20
37.5
53.4
52.0
57.0
30,1
2x2
4x4
2x2
20
Yes
864+100
41.7
67.7
73.7
63.9
39,8
2x2
4x4
2x2
20
No
864+360
41.2
65.9
74.0
63.6
40.1
2x2
2x2
2x2
20
Yes
3024+100
43.6
72.8
79.9
65.9
42.9
4x4 pooling to local images to mitigate redundancy without significant information loss. In rows and 4, we present the experimental results comparing approaches for obtaining replay visual features: one using visual memory and the other cropping images via bounding boxes before re-inputting them into the VIT. The results demonstrate that our use of visual memory not only achieves comparable performance but also reduces training and inference costs by utilizing fewer image tokens.
5.4 TEST-TIME REPLAY TOKEN SCALING.
To further improve the performance, we investigate the possibility of test-time replay token scaling. Specifically, adopt a larger image cropping scheme during testing generate more image tokens while keeping the pooling strategy unchanged. The results shown in Table inbleaaiidiicate that a further scaling of tokens is also helpful, and this phenomenon is especially prominent in OCR-related tasks. We set 64 as the maximum number of cropped images, in practice, most images in the fine-tuning data do not reach such resolution level.
Table 8: Test Time Image Tokens Scaling. We apply a larger image resolution cropping scheme during testing to obtain more image tokens. Other setting is same as the Table 7
Base
Local
Replayed
#Crops
#Vtoken
MMStar
ChartQA
DocVQA TextVQa InfoQA
2x2
4x4
2x2
20
864+100
41.7
67.7
73.7
63.9
39.8
2x2
4x4
2x2
64
2592+400
42.9
67.9
76.3
63.9
42.9
CONCLUSION
In this work, we propose VGR for enhanced multimodal comprehension. VGR enables MLLMs to reason visual clues and selectively attend to informative regions on demand. To achieve this, we introduce a selective feature replay module, which allows the model to focus on crucial regions, thereby enhancing fine- ane-ggriiedd comprehension particularly for small regions in high- resolution inputs. We also curate a large-scale reasoning dataset, VGR-SFT, which for the first time integrates visual information into dense reasoning tasks. Extensive experiments VGR demonstrate considerable improvements across multiple benchmarks, validating the effectiveness our approach.
Discussion. Our method has limitations that warrant future research. First, VGR is currently implemented on LLaVA Liu et al. (2023b) architecture, exploring stronger visual encoders and LLMs could further enhance performance. Another avenue is integrating reinforcement learning (RL), more generalized and diverse reasoning process may be achievable with RL.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (62476262, 62271461 62306297,622 (,300299,62306296), the Beijing Nova Program, Beijing Natural Science Foundation (4242053, L.220096,, the Postdoctoral Fellowship Program of CPSF (GZB2025041)), and the Fundamental Research Funds for the Central Universities.
ETHICS STATEMENT
Our research is grounded in ethical practices, with particular attention paid to the responsible use of data. All datasets employed in this study are publicly available and well-established within the computer vision community. Specifically, our benchmarking was conducted on LLaVA (Liu et al. 2023b). Our use this data is in accordance with their provided licenses and intended academic purpose.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of the research presented in this paper. To this end, comprehensive implementation details for our models and experiments are provided in Appendix, including the training procedures and all hyperparameters used. Furthermore, upon acceptance of this paper, all source code, datasets, and trained model checkpoints will be made publicly available.
REFERENCES
Captitapisste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision- Langual model for understanding, localization, text reading, and beyond. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Gc, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Lin Chen, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et
we on the right way evaluating large vision-language models? arXiv preprint 2oXiv:2403220338 2024.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang. Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405. 19092. 2024.
Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. arXiv preprint arXiv:2501.05952 2025.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. UtLattps://gittut com/hugan http/hugiinggg ace/ actpa/cc///pppee open-rl
Xingyu Fu, Mingian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang. Refocus: Visual editing as chain of thought for structured image understanding. arXiv preprint arXiv:2501.055 054552, 2025.
Grok. Grok- 1.5 vision preview, 2024. URL /ttL/ttp://xx.a//l///////
Xin Gu, Heng Fan, Yan Huang, Tiejian Luo, and Liho Zhang. Context-guided spatio-temporal vidco grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18330-18339, 2024.
Xin Gu, Yaojie Shen, Chenxi Luo, Tiejian Luo, Yan Huang, Yuewei Lin, Heng Fan, and Libo Zhang Know- ing your target: Target-aware transformer makes better spatio-temporal video grounding. arXiv preprint arXiv:2502.1 168, 2025.
10

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, Deepscek-rl: Incentivizing reasoning capability in llms via reinforeement lcarning. arXiv preprint ar Xiv:2500 12948, 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-v1 technical report. arXiv preprint arXiv:2505.07062, 2025b.
Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. Rethinking entropy interventions in rivr: An entropy change perspective. arXiv preprint arXiv:2510.10150 2025.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zeulemoyer, Noah Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models, Advances in Neural Information Processing Systems, 0-3399338-337777 2024.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503. 250304990,,, 2025.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thougle in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.0962 2025.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEF conference on computer vision and pattern recognition, pp- 5648-5656, 2018.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram worth a dozen images. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings. Part IV 14, pp. 335-251. Springer, 2016.
Weixian Lei, Jiacong Wang. Haochen Wang. Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with single transformer. arXiv preprint 2504v:2504. 10462, 2025.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image eneoders and large language models. International conference on machine learning. pp. 19730-1972 19742. PMLR, 2023a.
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large lasion-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural
information processing systems, 36:34892-34916, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL /tt.itts////aaa...gggg.a 1o/ 2001og//224--202 2011:///220000000111100 aaavv--aett//aaae
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint a788::250..01785, 2025.
Zayan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models, arXiv preprint arXiv:2403.1 12966, 2024c.
11

---

Published as a conference paper at ICLR 2026
Haoyu Lu, Wen Liu, Bo Zhang. Bingxuan Wang, Kai Dong. Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. et al. Deepseok-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05555, 2024.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Info- graphievqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
OpenAI. Openai-ol, 2024.
Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: Train large vision-language models diving into details through chain of manipulations. arXiv preprint 2rXiv:2402.0422 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Zhongwei Ren, Zhicheng Huang. Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin, Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. (6374-26388) 2024.
Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. Videoworld: Exploring knowledge learning from unlabeled videos. arXiv preprint arXiv:2501.09781, 2025.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and henchmark for ohain-of-thought reasoning. Advances in Neural Information Processing Syxtems, 17:8612-8642. 2024a.
Zhihong Shao, Poiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Li Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2xXv:22022.03300. 2024b.
Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, and Jianwei Yin. Zoomeye: Enhancing multimodal Ilms with human-like zooming capabilities through tree-based image exploration. arXiv preprint arXiv:241 16044, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl rl-style large vision- languaee model. arXiv preprint arXiv:2504.07615 2025.
Amanpreet Singh, Vivek Natarajan, Meet Shah. Yu Jiang. Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read, In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang Reconstructive visual instruction tuning. arXiv preprint arXiv:2410.09575 2024a.
Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. arXiv preprint arXiv:2504.01 2025a.
Haozhe Wang. Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. V1-rethinker: Incentivizing self-reflection of vision- laiion-lagguge models with reinforcement learning. arXiv preprint arXiv22504.08837 2025b.
12

---

Published as a conference paper at ICLR 2026
Hong Wang. Zhezheng Hao, Jian Luo, Chenxing Wei, Yao Shu, Lei Liu, Qiang Lin, Hande Dong. and Jiawei Chen. Scheduling your Ilm reinforcement learning with reasoning trees. arXiv preprint arXiv:2510.248.. 2025c.
Jiacong Wang, Bohong Wu. Haiyong Jiang, Zhou Xun, Xin Xiao, Haoyuan Guo, and Xiao. World to code: Multi-modal data generation via self-instructed compositional captioning and filtering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4608-4623, 2024b.
Peng Wang, Shuai Bai, Siman Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/1,, 2024c.
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:24) 10442, 2024d.
Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: training-free free framework for high-resolution image perception multimodal large language models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7907-7915, 2025d.
Penghao Wu Saining Xie. V?: Guided visual search as a core mechanism multimodal llms. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13084-13094, 2024.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseck-v12: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412. 10302, 2024.
Xin Xiao, Bohong Wu, Jiacong Wang, Chumyuan Li, Haoyuan Guo, et al. Seeing the image: Prioritizing visual correlation by contrastive alignment. Advunces in Neural Information Processing Systems, 37:30925-30950. 2024.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:241J. 10440, 2024.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang. Yan Deng. Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision; Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, Ziwei Liu. Lmms-eval: Reality check on the evaluation large multimodal models, 2024. [RLsttts:://///// /ttp:::///axviviee/aa
Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaowen Zhang, Yang Liu, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, et al. Chain-of of-cus: Adaptive visual search and zooming for multimodal reasoning via rl. arXiv preprint arXiv:2505. 22555515533 15436, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internw1:: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A MORE ABLATION EXPERIMENTS ANALYSIS OF VGR IN THE MAIN TEXT
In Table of the main text, the experiment incorporating the complete components of VGR achieved the optimal results. We also list the outcomes when either component is removed by eliminating grounding cues ("w/o Grounding")
disabling the reasoning process ("w/o Reasoning"). Comparing the results without grounding (using only reasoning data with visual memory replay) against the baseline LLaVA-NeXTS, (first row) demonstrates that even without grounding. the performance exceeds the original baseline, highlighting the rationality and critical role of our reasoning data construction. Additionally, comparing the results without reasoning (retaining grounding with visual memory replay) against the LLaVA-No VA-NNeT--77B baseline shows that these outcomes also surpass the original baseline, validating the effectiveness and significance of using grounding boxes for image visual memory replay.
In Ta of the main text, the two rows of experimental results ablate the impact of adding detection loss. The results without detection loss still outperform the LLaVA-NeXT-7B baseline, confirming the validity of the other two components: visual memory replay and reasoning data. Table presents ablation results for visual memory replay; removing visual memory replay still yields performance superior the baseline, underscoring the rationale behind the detection loss and reasoning data components. Table 6c ablates reasoning data, and the results without it still exceed the baseline, demonstrating the effectiveness of detection loss and visual memory replay.
More trainging details. As stated in Section 5 of our main text, we follow the exact hyperparameter settings of LLaVA-NeXT, training all data for one epoch. The learning rate le-5 for the pre-training stage and 2e-5 for fine-tuning (Vicuna-7B). The VIT learning rate is set to one-tenth of the base rate. We use a warmup ratio of 0.03, a cosine scheduler, and zero weight decay.
A.1 PERFORMANCE COMPARISON OF VGR-SFT.
As shown in Table 9 we first observe that data scaling is critical for VGR: VGR-SFT trained on 158K data clearly outperforms models Goold-Star/AAnnotatie benchmarks. However, cold-start data scalability is limited by slow annotation speed and high reject rate. To address this, we trained a smaller size annotator model (14B MLLM), where cold-start data training eases format and instruction complexity, making it feasible for smaller models to perform annotation. We selected the 14B MILMM for its strong benchmark performance and compatibility with our training framework, and results confirm its effectiveness: compared to the cold-start baseline Sive2.55-VL-222 our annotator-based models are 9x more efficient (3.2x faster inference, 2.9x higher accuracy, from 14% to 40% pass rate) while enabling us to scale more data with limited resources.
trained on 25K data across multiple
For fair comparison, we used 25K data (from two sources) for VGR training: "VGR Cold Start 25K" refers to data annotated by Qwen2.5- while "Annotator Data 25K" denotes data annotated by the 14B MILMM. The two datasets achieved similar VGR performance, demonstrating the effectiveness of our scaling strategy with smaller annotator model. We will add the full data scaling experiment to the manuscript.
Table 9: Performance Comparison of VGR-SFT.
Model
MMStar
ChartQA DocVQA TexiVQA InfoQA
A12D
LLaVA-NeXT-7B VGR Cold Start 25K
37.2
58.7
70.2
60.5
34.7
68.5
37.4
61.2
72.6
62.3
37.1
70.4
VGR Annotator Data 25K
37.7
60.9
72.8
61.9
36.8
70.5
VGR 158K
41.7
67.7
73.7
63.9
39.8
73.7
A.2 COMPARISON WITH ZOOMEYE.
In Table TO we present a detailed comparison between our VGR framework and ZoomEye (Shen)
al. 2024 along with key methodological distinctions and experimental results. ZoomEye adopts
14

---

Published as a conference paper at ICLR 2026
recursive tree based search paradigm to simulate human like "zooming" behavior: it first splits images into hierarchical sub sub-patches and conducts reasoning through iterative node exploration. Though this training free design is innovative and effective for visual grounding, it incurs substantial computational overhead repeated patch splitting and tree traversal lead to slow reasoning speed, especially for high resolution images that demand deep recursion. By contrast, our VGR leverages selective visual memory replay mechanism, which directly retrieves task-relevant visual tokens from a preconstructed feature pool, thus eliminating the need for recursive search entirely.
Quantitatively, the averaged wall clock time per question for ZoomBye reaches 48.46s on V Bench and 55.52s on HR-Bench 8K. In comparison, our VGR achieves significantly faster inference, with average per-question times of only 7.5s (V* Bench) and 9.2s (HR Bench 8K), highlighting the efficiency advantage our feature retrieval-basee design over ZoomEye's s recursive search approach.
Table 10: Comparison with Zy: Inference Time and Performance on High-Resolutinn Benchmarks.
Model
V* Bench
Time
HR Bench
Time
LLaVA-NeXT-7B
56.4
1.04s
1.11s
LLaVA-NeXT-7B w/ Zoom Eye
71.7
48.5s
44.4
55.5s
VGR 7B
67.7
7.51
52.9
9.23 s
B MORE EXPERIMENTS OF VGR ON DIFFERENT MLLMS
B. ABLATIONS ON DIFFERENT POOLING STRATEGIES.
Table 11: Ablations on different pooling strategies. Different pooling strides for each type of image important. We utilize 2x2 for Base image feature and Replayed image feature, and 4x4 for Local images feature. It is noted that above the horizontal line are the results for the original data without VGR-SFT data.
Model
Base Local Replayed Crops
Vtoken
MMStar
ChanQA
DocVQA
TexVVQA InfoQA
I.LVV--Viuna
Vicuna7B
2880
37.6
54,8
77.4
64.9
37.1
2880
40.4
62.2
77.5
66.9
41.3
LLaVA Vicuna7B
2x2
4x4
288
37.5
53.4
52.0
57.0
30.1
LLaVA- Vicuna7B
2x2
4x4
20
864
41.3
60.2
71.7
62.7
38.4
LLaVA L.AVA-Qwen2-17 Qwen2-7B
2x2
864
39.4
49.8
78.2
58.5
39.4
With VGR-SFT data and replay image feuture
2x2
4x4
2x2
20
864+100
41.7
67.7
73.7
63.9
39.8
LLaVA- -VicunaBB
2x2
2x2
2x2
3024+100
41.7
67.7
73.7
63.9
39.8
LLaVA- VVAAViiunal3BB
2x2
2x2
2x2
3024+100
44.6
71.7
78.6
64.9
41.8
LLaVA -wen--2
2x2
2x2
2x2
3024+100
46.9
62.7
61.9
42.5
To illustrate the model performance under different settings, we correct typo and incorporate additional experimental results TT including replacing Vicuna-7B with Qwen2-7B on the LLaVA-NeXT 13B and LLaVA-NeXT baselines. We sincerely apologize for the typo in the main text, where the results of the first two rows were incorrectly stated as the raw baseline and VGR setting.
B.2 How INACCURATE BOUNDING Box REPLAY IMPACTS REASONING.
To directly examine how inaccurate bounding box replay impacts visual grounded reasoning, we evaluate two inference settings: (1) attaching random box image tensor while keeping the text box token correct; (2) replacing both the text box token and the image tensor with a random box. The corresponding results shown in Table [2]
We observe substantial performance degradation in both cases. Incorrect visual features cause large negative effects when the text token is correct, and performance drops further when both text
15

---

Published as a conference paper at ICLR 2026
and visual features are incorrect. This demonstrates that accurate replay predictions play crucial role in reasoning. It's noted that in the first line, does not mean that both the box token and the corresponding image feature are random. Instead. it means that first randomly crop image tensor with a box. At the same time, replace the text token the correct box in the current output II) with the text token corresponding this box.
Table 12: Performance Comparison of inaccurate bounding box.
Model
Box Text Token
Replay Token
V* Bench
DocVQA
MMStar
ChartQA
TexIVQA
A12D
VGR-7B
random
random random
37.4 37.7
61.2
72.6
62.3
37.1
70.4
VGR-7B
right
60.9
72.8
61.9
36.8
70.5
B.3 THE RELATIONSIIIP BETWEEN REPLAY AND GROUNDING
Using Imms-eval, we evaluate VA-Net and VGR on Referring Expression (R)prehension(REC)) of RefCOCo, RefCOCOg, and RefCOCO+, reporting both the standard IoU>0.5 accuracy and the IoU metric. Results are shown in Table 13 The results show consistent improvements over the baseline, suggesting that VGR training and data also enhance grounding performance.
Table 13: Performance Comparison of REC tasks. Where the metric of means acc@0.5, the metric I means the IoU.
Model
RefCOCO_A RefCOCO_ RefCOCOg_A
RefCOCOg_
RefCOCO+_A RefCOCO+
LLaVA-NeXT-7B
0.85
0.72 0.74
0.82
0.69
0.77
0,65
VGR-7B
0.88
0.84
0.71
0.78
0.68
B.4 CONDUCT VGR OTHER BACKBONES.
VGR is unified framework involving both architectural design and data construction, we require a baseline that can be fully trained from scratch with a reasonable implementation cost. L.Ia VA-Next- 7B fits this requirement well. In contrast, the training data for InternVLI and Qwen2.5-VL. is not publicly available. Since VGR requires both pre-training and SFT, a complete reproduction of these data-closed models is not feasible. Nevertheless, as reported in Table 3 of the main paper, we have already evaluated VGR on comparable baselines such as Intern ViT and Qwen2.5 LLM, where we still observe substantial improvements.
To further address some conern about generalization, we conducted an additional analysis experiment. We applied a post-training stage on Intern VL.3mB using I.L.VA-Next-770 data and obtained results comparable to those reported in its paper (although these data were already used during its internal training). As a comparison, we then applied the VGR style post-training stage Intern VL3-8B under the same setting, Note that this experiment includes only post-training rather than the full multi-stage VGR pipeline in our paper, and is therefore an analysis-orienter probe experiment. The results are shown in Table [4] Even without the full multi-stage pipeline, VGR-style post-training yields improvements on key benchmarks VQo, ChartQA, TextVQA), indicating that the proposed strategy is transferable across different architectures.
Table 14: Performance Comparison of serve VGR only in post train stage Intern VL3-8B.
Data Training
Inference Style DocVQA MMStar
CharlQA
TextVQA
A12D
LLaVA-NeXT-710
SFT
91.6
65.2
86.4
80.9
83.0
LLAVA-NAYT-770K--GR--FT VGR
92.8
63.9
87.8
81.5
85.3
16

---

Published as a conference paper at ICLR 2026
B.5 INFERENCE BEHAVIOR AND COST WITH VGR.
The increased latency stems primarily from two factors. First, LLM inference time is largely determined by the next token generation mechanism, making the runtime closely tied output length. Our baseline LLaVA-NeXT does not perform any reasoning and only generates short answers, allowing complete inference quickly. In contrast, models using CoT or thinking-style answers must generate longer reasoning chains, which naturally increases latency. The inference times reported the paper were measured using H20 GPUs and the Lmms-eval framework. Second, we computed the average number of generated tokens, replay times, run time for VGR-7B on different tasks, as shown in Table 15
VGR requires visual feature replay and longer outputs during answering, which contributes to the increased inference time. Nonetheless, thanks to rapid developments in the open-source ecosys- cem-such as vLLM and sglang, more efficient LLM inference framework is becoming increasingly accessible. We will continue to follow and adopt these technologies to further improve the inference efficiency of our model.
Table 15: Inference behavior and cost with VGR.
Metric
MMStar
V* Bench
ChartQA
DocVQA
Repaly Times
2.0
1.7
2.4
Generated Tokens
98
110
155
113
Run Time
5.84s
7.51s
11.57s
6.51s
we measured the cases where the model failed to produce coordinate text between the special tokens that could be parsed by our rules, produced invalid coordinates. Across various downstream benchmarks, the proportion of valid bounding boxes consistently ranges from 97.5% to 98.95 98.9%, indicating that only few corner cases fail after VGR training. Second, trained variant of VGR- 7B that uses absolute pixel coordinates instead of normalized (0-1) relative coordinates, and compared with the original setting in Table T6 Benefit from the global template construction enabled by LLaVA's any-resolution design and the visual feature memory mechanism, the absolute-coordinate version performs only slightly worse than the original setting.
Table 16: Performance Comparison using absolute pixel coordinates instead of normalized (0-1) relative coordinates.
Model
Box Type
DoA
MMStar
ChartQA
TextVQA
A12D
RWQA
VGR-7B w/normalized
73.7
41.7
67.7
63.9
73.7
59.8
VGR-7B
wo/normalized
71.9
40.9
64.2
62.8
70.5
56.23
C THE CASE OF VGR
Figure 4 shows a case in VGR-SFT with different formulations, after being processed separately, these three types of data are used to train VGR. The corresponding results are w/o Memory, VGR-7B and w/o Reasoning in Table 6c.
We also present a case study in Figure to illustrate how VGR's 's design contributes its success in such scenarios.
D REASONING DATA PIPELINE
D.1 DETAILS ON DATA CONSTRUCTION
In this section, we claborate the details on data curation.
Reject Sampling. During the reject sampling, we implement two verification steps with MLLM from online API, which is Doubaol.5. 1.L (Guo al. 2025b) in our implementation. We did not
17

---

Published as a conference paper at ICLR 2026
Osestion:
Question: this judicial book?
this judicial. book?
Is this judicial book?
Answer:
Asswer:
Answer:
<thinio-
<sot> 15, 337, sest
The question asks the book jusscial, Judicial books related to Lan. The ticle esotald 15, 337, 215)s
ludactal, Judicial books are related to lav. The title esot>10, 15, 337, 215]-eott
question it
lauation
deals axe-related indicates
IInternaioonal
Intematkonl Lasatin
countries tocation geverned
<sot=112, 299, 182, 493) <eot>
Ims. The pibl tster ctnts 112, 199, 187, (60) cents Foundation ton Press, knnwn for lesat textbooks, further supportine jusicial.
deals with fat-re lated Legal matters across countries as taxation is govemne by laws, The publisher <sst>112 (12, 399, 182,
the -sotv 18, 0, 337, 80 dplleot> "Concepts and Inssghts Series" ieplies it is on edocal Lonal book for lee-related audiences.
<sot= 337, 56 -est
International
Final unswers Yes
Final. attemrts Yes
15 Foundation Press, textbooks, turther supporting this 15 jidicial book, Also, being part ot the <sot=l8, 0, 337, abl
legal
International
"Cancepts Toxights Series" implies it
audiences.
</thinis-
Finsl. answer: Yes
Case w/o Memory
Case with Memory and Reasoning
Case w/o Reasoning
Figure 4: Example of training data in VGR-SFT in different formulations.
Becation:
Long Training
bosshee Baoka an 38131
dreaed Truth:
laok, cuis bar chart provided The chart stan the franchuse
Stort Tratntia lata after Revrittes
voute the lest right sice the char. The teight of cast Leaking 2012 bur, laboled "Frashhte whise tollinn callars.
ethirns
pravicus
Ne asted fand the estirated
value the Milusudon Backa on 2821.
idwatiify the tar
right which the
for for san
and the 201
The yeasts The chart lataled
the
thare are
in arstacionn sates that nogt
tos,
oillaret. her cap
up
fonustag region
castire that
bartheeeyy this har water. Theeess intigecly The
value carr Reeeeting to this bar is 1855
Final senwer: to22
catermuee extirated
duska 2013
over thee. sote oaes ser reperenting scecitic your,
selted the valanres The tauges of the 2822 our alages with aas auttans seectity costatanneee
(2003) stigntly
11.0, contintirg treee and acs cocy
(667 clearly nacies 1826 akoicattnn the franchate Value that
this har
Pissl drower 1825
Example generated by our annotation model. We distill core information and the chain-of- thought from long redundant reasoning with reject sampling and rewriting.
Figure 5:
adjust the parameters of Doubao1.5-1 VL and instead used the official default settings, with a maximum response length of 8192. The close-soureed online MLLM is fast and strong, but can not be modified for our specific task and is expensive, therefore we only use this model for well-defined task like checking, filtering and rewriting. The prompts for remote requests are detailed in Table 17 where two distinct prompts are designed for correctness verification of open-ended problems and grounding area verification, respectively. To process responses from the commercial model, we use a simple parser to convert the output into an integer ranging from O to 5. A threshold of 3 is applied to filter out noisy data, ensuring the quality of the dataset.
Data Rewriting. The data rewriting strategy is introduced to address amendable errors. First, during the reject sampling phase, we perform ground-truth-aligned rewriting to reconcile the generated answers with the ground-truth annotations in our training data. To avoid absurd change in "Final Answer", we also use the MLLM to align the reasoning chain with final answer. Second, we introduce format and reasoning process rewriting for all reasoning data, ensuring all data matches the same
18

---

Published as a conference paper at ICLR 2026
Think step by step and answer following question, you need reference the key with "[xl_x2.y1.22""
bounding-box formut and give the finul answer with "Tinal answer:".
What is the color of the man's belmet? white (B) red (C) yellow (D) blue
Answer with the option's letter from given choices.
final answer is: The yellow, answer is
Turn
Figure 6: Example of how VGR's s design contributes to the visual cot.
format, mitigate confusion in the reasoning chains, reduce information leak before the replay and avoid failed answer extraction. The prompts are shown in Table 17
Annotator Training. We train our annotator with type of data, cold-start data that includes visual reasoning generated by previous steps, and Open-R1 data (Face 2025) from pure-text reasoning chain distilled from Deepseek-R1 (Guo et al. (025a)). We use the learning rate 1e 5, batch size of 128 and trained the model for 6000 steps. During annotator training, we use different prompts for these two types of data, which reduce confusion for the model, the prompt shown in Table 17 As shown in the next paragraph, we use the combination of these two prompts to generate more diversified answers.
Annotation Generation. We employ distinct prompts to guide cold-start model and annotation model in generating training data, as outlined Table 18] For the cold-start model, we provide a highly detailed prompt with illustrative example to ensure data quality and format consistency. For the annotator, we use a hybrid prompt that integrates visual grounding and Open-R1 prompts, enabling the generation of complex multi-step reasoning akin DeepSeek-RI behavior.
D.2 VISUALIZATION DATA AND VGR
To illustrate the effectiveness and necessity of our data pipeline, we show the differences among each segment of our pipeline. The data examples of cold-start, annotated, and the final data are shown in Figure 7 As shown in the figure, our data curation pipeline is able to improve the data quality step-by-step: the annotated data enhances the reasoning complexity and annotation efficiency, while the training data from the rewriting is more concise and easy to learn. To expose more details, data from the annotation model and refined data are illustrated Figure After training with cold-start data and complex reasoning data from DeepSeek-PP R Face (2025) Guo et al. (2025a), the annotation model can generalize its reasoning ability from text-only to visual reasoning. However, this model still need the reject sampling pipeline and the rewriting
still casily makes various mistakes, module to fix these issues. this example, the reject sampling and checking module expands the bounding- boxes, aligns them with the correct ground truth, and enriches the context. The rewritten module removes duplicate bounding-boxes, reformats the document, and clarifies the explanations. The rewritten short and clean data is especially valuable for smaller-scale models like Vicuna (Chiang et al. 2023) that only supports 4096 tokens. As shown in Table 6e in the Main Paper, short clean data also performs better than long data in our experiments.
In Figure we visualize the responses of VGR on the MMStar and ChartQA benchmarks trained with our data. VGR automatically and accurately locates target regions in the responses, generates correct reasoning based on the content within these regions, and ultimately provides accurate answers.
E FURTHER COMPARISON AND INSIGHTS FROM RELATED WORKS WITH VGR.
19

---

Published as a conference paper at ICLR 2026
We address three distinct novelties of VGR, which differ from previous attempts Visual CoT (Shao et al. 2024a), SketchPad (Hu et al., (024). Refocus (Fu et al. 2025 ):
Native Free-form Visual Reasoning. VGR enables the VLM to retrieve and reuse visual memory dynamically. Compared with Visual CoT, which trains the model with a multiturn dialog with predefined forms, VGR enables the model reuse visual memory during complex visual thinking. This allows VGR for flexible "visual re-reading" at any step of the reasoning chain, interleaving summary, reflection, and visual replay-withou manual cropping interventions. In contrast, Visual CoT can only retrieve the cropped area once and can only think with a static path.
VGR eliminates the need for multi-turn dialogue format guide the model to first output bounding boxes, followed by manual cropping and re-input of relevant visual features and text. Instead, VGR's s visual feature replay not only offers greater flexibility format but also, more importantly, embeds grounding reasoning capabilities within the model itself, enabling the model to spontaneously inspect highly relevant image regions during reasoning, mimicking human- like cognitive processes.
Full Open Solution with Data and Training. While methods like SketchPad and Refocus focus on tool-use capacity on advanced commercial MLLMs (such as GPT-4v/)), VGR provides a complete, open-source pipeline including data construction and training. VGR's s end-to-end data construction pipeline provides the community with a followable, efficient data iteration framework. VGR also employs joint training of detection loss and autoregressive loss, enhancing the model's grounding capabilities. This allows VGR to enable visual reasoning ability for an open- spen-source VLM with no inherent visual capabilities, while SketchPad and Refocus heavily rely on existing capabilities of closed-source MLLMs.
Efficient Visual Memory Architecture. introduces novel architecture for visual memory. Unlike tool-based methods (such as Visual COT, SketchPad, and Refocus), that repeatedly process image crops, VGR pre-computes visual tokens once. By using multi-level pooling, it allocates computation to the most critical regions efficiently. As shown in our experiments, this design saves approximately 70% of tokens compared baseline methods while achieving superior performance.
CogCom and Chain-of-Foeuus (CoF), both are great works and worth discussing, Compared to both of them: VGR proposes a novel architecture with visual memory, which enables visual memory replay during reasoning. Compared to the "grounding then crop" strategy used in CogCom and CoF, the visual memory retrieval operation is more efficient and consistent. VGR pre-computes visual tokens only once, while the crop or zoom-in operations need extra computation views. By using multi-level pooling, VGR further allocates computation to the most critical regions efficiently. As shown in our experiments, this design saves approximately 70% of tokens compared to baseline methods while achieving superior performance.
modified
Compared to CogCom (Qi et al. 2024); CogCom uses predefined operations to retrieve knowledge (e.g., GROUNDING, OCR, CROP AND ZOOMING, CALCULATE). However, it does not reason on the retrieved information. On the other hand, VGR learns the memory retrieval operation during reasoning, which is able to analyze, reflect, and plan further operations based on the retrieval result. This makes VGR more flexible for complex quests.
Compared to Chain-of-Focus (Zhang et al. 2025): Chain-of-Focus is a concurrent work with VGR. Compared with CoF, VGR is able to learn from scratch (from an LLM with vision prior), which is more challenging, while CoF relies on the existing ability of the pre-trained MLLM. Considering that many backbones trained via VGR have not been well-trained on CoT/rrasoning or grounding, VGR makes greater efforts to propose a scalable, high-quality visual reasoning data pipeline. VGR also introduces a detection loss for supervision during training, which effectively improves the grounding and retrieval quality for MI.LMs..
20

---

Published as a conference paper at ICLR 2026
hoper from Anreteto::
gru Eandt cibetity resinced potstt chatt
alss the
torren here rectire the sattir the the larer ary
(section:
Arower the canstion with eingli Io.
wo Openn recnrd CSBOT
You, cereert bort me vatien. The other groan cren't ne here stree centtie
Anower: Tafect motel
Elfieeee is 6.0
the with
greup you, the caicalation raght tnathiantionn
Arower Coldettart Rodel:
sisee quetten about the difference betwees the 18-22 years
rid 22-33
ratas, date
The than the ten tare ieprecetting de mater of paid heurs
aebi caste the altince was
Targer 322 dudi cifferewe
seel duerration The blue ber top wesett mules.
The reaget
catoslace differance
the 22-29 groue Libeled and
Bovrttie Anser for Reed Testxtner
Lhe grea, hess Herennel 3.2
woter
group the watae
repreei with in lerieerl vette for wiles
benee 18-2
parr greas males
velerace we sortetie Sitleeee
Skpp scraty calcelation
19.00
rents certty
agale fresects an the cigutiicaer reginees Fiaal stnee: 6.6
200),
Hapay Oalica wantacer
butay Aepoes waw Traiiag
clearly valune are 1.2 respectiinly conft
exr catocletzan.
veriel betuees the 18-21 years 1ld and 22-29 years ald greeps for Final Mover: 0.0
Figure 7: Example of data from original data, cold-start model, annotator and training set.
Ousstion:
the overall used the suage?
krat the difference between the righest and the lovest green ter?
tptions: At Heutral, 8: scaber, 6: Teroe, 0: Joyful
Arswer the question with stngle vend.
given choiices
Asswer: ethuaks
Ansver:
question
the difference between highest lowest
The lasge shows person koyaking in a river. The individust is wearing
great-bur values in the chars, in the chart, the green bers represent the
and Life- uncicnting satuty precaztians. The water is cala
percenaage
2236, play (b.ss, activv eititary
trees lush, person's pesture and aria on the pasd sugpest costrut and contidence. Insteat, the these elenents, there are no adventure lessure. <tons 19.22, B. 3. 2:24, 65751 storithe persor in the kasak, sppears at cose, which is
8,52, 6-53) <EOT> the green bor value 225 and for 2015, <S0T> 18.52, 8.47, B. 8.56. Subtracting the Towest froe the highest gives 29-25 e/thanko
re the Aala-Pacates regien,
distress tension.
</ttinke
Fanal arower:
Slight Iaganess suppoet
more military sole region Japas chreli indertiss majiee
facing wi
Zo Cobol ieseddd
Fou 000
Figure Example of VGR response MMStar and ChariQA benchmarks.
21

---

Published as a conference paper at ICLR 2026
Table 17: Prompt for VGR reject sampling and data rewriting.
Stage
Prompt
Reject Sampling
Correctness Verification
You are an annolator, your goal is to check the reasoning process is aligned with multimodal question and answer.
You will be given the question, ground truth, the reasoning chain and the original answer. Output integer from to 5: output 5 if the reasoning chain is aligned with the ground truth (even the answer has some mistakes), output 0 the reasoning chain is not aligned with the ground truth.
Question: (question)
Ground truth: {gt)
Reasoning chain: (answer)
Original answer: [final answer]
Visual Grounding Verification
You are an annotator, your goal is to check if the short content description of the bounding box is aligned with the image. I will send you two images: one is the original image and the other is the bounding box area cropped from the original image.
Output integer from 0 to 5, O mcans the content is not aligned with the content, 5 means well aligned. Check if the content from the second image is " "(content)".
Data Rewriting
Ground-Trutth Rewriting
You are an annotator, your goal is to check the reasoning process is aligned with multimodal question and answer, and rewrite the reasoning chain and the answer match the ground truth. You can add more details to the answer, but all information introduced by the ground truth should be covered. given the question, ground truth, and the original answer with the reasoning for reference. Output the answer with the reasoning process: think first, then answer the problem. The final answer that matches the ground truth should be written after "Final answer:".
will be
Question: { question) Ground truth: (gt)
Answer with Reasoning: (answer)
Reasoning Chain Rewriting
You are an annotator. Your goal is check whether the reasoning process aligns with the multimodal question and answer, and rewrite the reasoning chain and the answer to match the ground truth. You can add more details to the answer, but it must cover all the information provided by the ground truth. You need to remove any redundant, confusing, or incorrect information from the original answer. The rewritten answer should be logical and concise. The answer should follow a strict format: all the thinking parts should be enclosed within <think></think> tags, and then state the ground truth starting with "Final answer:". All location information should be enclosed within <sol><eot> tags; the content of <soD<<eoD> includes "bbox 2d" and "label", which are simply copied from the original answer and should NOT changed. You need reference the area before mentioning any information in the area, and each location should be mentioned only once (i.c., duplicate <sot><eot> tags with the same information should be removed).
You will be provided with the question, the ground truth, and the original answer with reasoning for reference. Output the answer along with the reasoning process, make the answer fluent, and do not use the ground truth in the reasoning process. You must reference at least one location with <sot>..<eott the content of <soD><eo>> is copied exactly from the original answer. Think through the problem and the referenced area, and then write the final answer that matches the ground truth after "Final answer:". Only return a single-line final answer, which should strictly conform to the ground truth.
Question: (question) Ground truth: (gt) Answer with Reasoning: (answer)
22

---

Published as a conference paper at ICLR 2026
Table 18: Prompt for VGR model training and data construction.
Stage
Prompt
Annotator Training
Cold-start Data
Think step by step and answer the following question, you need to reference the key area with <sot> <sot>>"bbox (""box_2d""""" ffl:xxxlyllx..xx" "x2.,y2,,"lbbel"".. "".."""<""..""" bounding box format and give the final answer with "Final answer:".
The size of the image is (image.width) (image.height). [original_question]
Open-R1 Data
(original question)
Give step by step reasoning before you This requires engaging in comprehen- sive cycle of analysis. summarizing, exploration, reassessment, reflection, backtracing. and iteration to develop well-considered thinking process. You need to use <think> </think> to wrap your reasoning process and answer the final answer enclosed in La- TeX's 's \boxed tag.
Data Annotation
Cold-start Model
You must locate and focus on the major objects that significantly contribute to solving the question. Prioritize the output of bounding boxes for larger and more significant areas, minimizing the inclusion of smaller, less relevant regions. Output the bound- ing box coordinates of these key objects JSON format. you reason step by step, ensure each step includes detailed considerations such as amalyzing the ques- tion, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. During this process, emphasize larger and more important areas using the bounding box format <sot><"bbox ("bo="bbxxdddxxx_,,__. [x 1,2yy,2y,,,y1 "label":"... <]ett to reference visual details infor- mation. Reference the area before mentioning its content. Finally, answer the question with "Final Answer: xxx". For example:
="""""><ott>>
To answer the question [state the question here]. first, we need to identify [describe what needs to be identified], let me focus on this significant region <sot>(""bos "dd=_ddd=='" [d":[x1.y1 1,x2222,,221 "label""" "K<ts>. You need to replace xl. yl with the actual pixel coordinates. In this region, observe [describe what you see in the region, such the letter xxx]. This observation indicates [explain the significance of what you saw]. Based this analysis, we can conclude that [continue with reasoning process]. Therefore, the answer [state the answer]. Final Answer: [answers]
Now, I will provide you with a Question. Please output the answer with the bounding box incorporated into the reasoning described above, focusing larger and key areas, minimizing small or irrelevant boxes." (original question)
Annotation Model
Think step by step and answer the following question, you need to reference the key area with <sot> [""box__20 (bbox__d":[xx 1f,.xxxll,x,xxxxx.. "x2.y21."laael"". "label":"... """".."<<oot""" "eooot> bounding-box format and give the final answer with 'Final answer:". " The size the image is (image, width) (image.height].
[original question)
Give step by step reasoning before you answer. This requires engaging in comprchen- sive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing. and iteration to develop a well well-considered thinking process. You need to use <think and </think> to wrap your reasoning process start and end. During reasoning, reference only at the thinking process. Do not include box information in the final answer. Ensure the final answer appears only once and contains only the solution or conclusion.
"lahel":".."" the key area with ( ("bbox_24 [x"::xx..y1... "l,.a
Reasoning Model Training
VGR-SFT Data
Think step by step and answer the following question, you need reference the key area with (xx22x 1,yyyx. 1.y2]eot>>" bounding- "Final answer:".
format and give the final answer with
(original question)
23


Reviews:
[{"review_id": "fEo49Cyvbw", "reviewer": "Reviewer_joFM", "summary": "This paper introduces VGR (Visual Grounded Reasoning), a framework that aims to enhance multimodal reasoning in vision-language models by incorporating a visual memory replay mechanism. The key idea is to allow the model to explicitly reference relevant visual regions during the reasoning process through a controlled replay signal (<sot>[x1,y1,x2,y2]<eot>). This design reduces the number of visual tokens, improves efficiency, and enhances interpretability. The authors conduct extensive experiments on multiple benchmarks (e.g., AI2D, ChartQA, MMStar), showing consistent improvements over LLaVA-NeXT baselines with fewer visual tokens. The paper also provides detailed ablation studies and efficiency analyses. While the empirical results are strong, the conceptual novelty is somewhat limited, and certain aspects of the method and presentation could be improved.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper includes a large number of experiments across diverse benchmarks with fair comparison to strong baselines, along with clear ablations and efficiency metrics.\n\n2. The proposed replay mechanism and token-efficient design are well implemented and practically useful for multimodal reasoning systems.", "weaknesses": "1. The work is technically solid; however, the contribution in terms of novelty could be clarified further. The concept of visual CoT (chain-of-thought grounded in image regions) has been explored in prior studies (e.g., VisualCoT[1], SketchPad[2], Refocus[3]), which are currently only briefly discussed. Including a more detailed comparison could help position the paper within the existing literature. Among these, approaches most similar to “visual CoT” could be highlighted more explicitly, both conceptually and empirically.\n2. The paper adopts LLaVA’s any-resolution encoding to handle arbitrary image sizes. It would be helpful to provide additional discussion on why this design choice is preferred over an alternative strategy such as cropping the image into regions, which may be more architecture-agnostic and computationally straightforward. Clarifying this could strengthen the reader’s understanding of the method’s design trade-offs.\n3. The title emphasizes “Visual Grounded Reasoning.” It may be useful to clarify the scope of reasoning in this context. Prior studies (e.g., DeepSeek-R1[2]) suggest that reasoning ability in large language models often arises from reinforcement learning or dedicated reasoning-phase training, whereas supervised fine-tuning primarily encourages pattern learning or imitation. Explicitly situating the current work within this context could help manage expectations regarding the type of reasoning capability demonstrated.\n4. The proposed replay mechanism assumes that accurately predicting relevant visual regions contributes to improved reasoning. Currently, the paper does not examine how often the replay regions are correctly predicted, nor how reasoning performance is affected when the replay predictions are inaccurate. A quantitative analysis, such as tracking replay prediction accuracy during training and correlating it with task performance, could provide stronger evidence for the causal role of replay. This would also help determine whether the model genuinely relies on visual replay or mainly benefits from additional supervision signals.\n5. This paper only conducts experiments on the LLaVA-Next backbone. It would be better to conduct experiments on other backbones, such as Qwen-VL or InternVL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oQ9HCK52Op", "reviewer": "Reviewer_CQgV", "summary": "The paper introduces the VGR framework, designed to support visual reasoning through visual memory replay during the thinking process. The main contribution of VGR lies in its visual memory replay module, which retrieves and replays visual information from memory. During inference, the model can request projections of image patches from memory, enabling it to revisit relevant visual regions and perform more focused reasoning. This approach allows the model to start with a smaller set of snapshot embeddings (around 30% of the original LLaVa) without sacrificing quality.\n\nTo train VGR, the authors first use an existing model to “cold-start” a dataset: given an image and a question from existing benchmarks (e.g., AI2D), the model generates a reasoning chain and answer. A rejection strategy is then applied to filter out incorrect samples using formal verification, correctness checks, and visual grounding validation. A smaller model is trained on the verified cold-start data and later used as an annotator to create a large-scale dataset. This larger dataset is then used with SFT methods to train the final VGR model.\n\nThe evaluation section is well structured, consisting of two main parts: (1) main results demonstrating the strong performance of the VGR framework across a range of datasets — both those included in the cold-start data and new, out-of-domain benchmarks such as MMStar and TableVQA — and (2) an extensive series of ablation studies exploring different components of VGR, including its reasoning process, memory module, and loss function.", "questions": "Q1) Why did the authors choose not to formulate the memory access as a tool call, rather than introducing special tokens? To me, that seems like a potentially more straightforward approach.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper is clearly written and methodologically solid. The assumptions are well stated, and the authors do an excellent job of detailing their experimental setup and reasoning process. The evaluation section is one of the strongest aspects of the work, especially the ablation study, which provides valuable insight and helps position the paper as a meaningful contribution to the field.", "weaknesses": "There are a few points that could use more clarification.\nFirst, the paper doesn’t clearly separate in-domain and out-of-domain benchmarks in Table 2. I tried to cross-check the datasets listed in Table 1 myself to identify which ones were new, but I would still like the authors to explicitly confirm whether the evaluations indeed include out-of-domain data. Without this distinction, it’s hard to fully assess the framework’s generalization ability.\n\nSecond, the main comparisons focus on relatively smaller VLMs. It would be interesting to see how VGR performs when integrated with larger models. If there are challenges or limitations in doing so, it would be helpful for the authors to discuss them. Otherwise, including such results would strengthen the paper’s claims about scalability.\n\nFinally, as shown in Table 10, VGR’s main drawback appears to be its computational cost (e.g., 1s runtime for LLaVa-NeXT vs. 7s for VGR). Apart from a brief note in Appendix A.2, the paper doesn’t explore this issue in depth. A more thorough discussion of runtime and hardware costs would make the evaluation more balanced and practical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XzScUnmDGs", "reviewer": "Reviewer_R29m", "summary": "This paper proposes VGR (Visual Grounded Reasoning), a novel multimodal large language model (MLLM) designed to address language bias and limited visual detail understanding in existing multimodal chain-of-thought (CoT) reasoning approaches. VGR introduces a dynamic visual memory replay mechanism that enables the model to actively detect and retrieve visual tokens from key image regions during reasoning, mimicking human visual cognition. The authors also construct a large-scale supervised fine-tuning dataset (VGR-SFT) through a three-stage pipeline (cold-start generation, reject sampling, and annotation model scaling) to teach the model visual grounding and reasoning integration. Extensive experiments on benchmarks like MMStar, ChartQA, and AI2D show that VGR outperforms the LLaVA-NeXT-7B baseline with only 30% of the image tokens, validating its efficiency and effectiveness in visual-linguistic reasoning.", "questions": "*To facilitate discussions during the Rebuttal phase, authors are advised to respond point-by-point (indicating the question number).*\n\n(1) Could you provide a comparative analysis of different cold-start models (e.g., 13B, 34B, 72B MLLMs) to justify why the 72B model was chosen for initial data generation? Please include metrics such as data quality (e.g., grounding accuracy), generation speed, and reject rate.\n\n(2) What are the specific parameter settings (e.g., temperature, top-p, max response length) and verification criteria of Doubao1.5-VL used in the reject sampling pipeline? Could you provide the exact prompts used for format verification, correctness verification, and visual grounding verification?\n\n(3) Why is β set to 2 in the detection loss? Could you conduct ablation experiments with different β values (e.g., 1, 3, 4) and present the results to confirm that 2 is the optimal choice?\n\n(4) Could you extend the comparison with ZoomEye to more benchmarks (e.g., TextVQA, InfoQA, DocVQA) and provide detailed metrics including accuracy, reasoning time, and visual token usage? Also, please analyze the fundamental differences in reasoning mechanisms between VGR and ZoomEye.\n\n(5) Have you tested more advanced visual encoders (e.g., InternViT-6B, EVA-CLIP-18B) or LLMs (e.g., LLaMA 3 8B/70B, Mistral 8X7B) with the VGR framework? If so, please provide the experimental results; if not, please explain the reasons and discuss the potential impact on performance.\n\n(6) What is the rationale for setting the maximum number of cropped images to 64 during test-time token scaling? Could you present performance curves with varying numbers of crops (e.g., 20, 40, 64, 80) to demonstrate the trade-off between token count and performance?\n\n(7) Could you analyze the contribution of each subset in the VGR-SFT dataset (e.g., AI2D, GQA, ChartQA) to the model's performance on different benchmarks? Please present ablation results showing the model's performance when trained on individual subsets.\n\n(8) How does the model determine when to generate the replay signal during inference? Please explain the trigger mechanism in detail, including whether it relies on semantic cues, statistical thresholds, or other factors, and provide illustrative examples.\n\n(9) Could you provide complete training hyperparameters, including batch size, number of training epochs, weight decay, learning rate scheduling (e.g., warm-up steps, decay rate), and optimizer type (e.g., AdamW, SGD) for both pre-training and fine-tuning stages?\n\n(10) Have you evaluated the model's performance on low-resolution images (e.g., 224×224, 112×112)? Please present the results and discuss how resolution affects the model's grounding and reasoning capabilities.\n\n(11) Could you provide a comparison with commercial state-of-the-art MLLMs (e.g., GPT-4V, Gemini Pro Vision, Claude 3 Opus) on the same benchmarks? If direct comparison is not feasible, please explain the constraints and provide indirect evidence (e.g., relative performance to open-source baselines vs. commercial models).\n\n(12) How do you quantify the reduction of language bias in VGR? Please define specific metrics (e.g., bias score, fairness gap) and present comparative results between VGR and the LLaVA-NeXT baseline on these metrics.\n\n(13) Please detail the high-resolution cropping strategy in visual memory pool construction, including parameters such as crop size, overlapping ratio, number of crops per image, and selection criteria for crop positions. Could you also explain how this strategy balances detail preservation and computational efficiency?\n\n(14) What is the annotation error rate of the VGR-SFT dataset? How did you detect and handle these errors during data curation? Please provide statistics on error types (e.g., incorrect bounding boxes, misaligned labels) and their distribution across datasets.\n\n(15) Could you report the model's inference speed (e.g., tokens per second, questions per minute) on different hardware platforms (e.g., A100, RTX 3090, RTX 4090, CPU)? Please also include memory consumption (e.g., GPU VRAM usage) for different model sizes (7B, 13B).\n\n(16) Have you tested the model's performance in multi-turn reasoning scenarios? Please design a set of multi-turn tasks (e.g., sequential visual reasoning, follow-up questions) and present the results, including accuracy and consistency across turns.\n\n(17) What are the specific thresholds for format verification and correctness verification in reject sampling? For example, what ANLS threshold is used for closed-ended tasks, and what semantic alignment score threshold is used for open-ended tasks? Please justify these threshold choices.\n\n(18) Please provide the detailed structure of the MLP in the detection head, including the number of layers, hidden size, activation function, input/output dimensions, and initialization method. Could you also explain how this structure is optimized for bounding box regression?\n\n(19) Have you evaluated the model's generalization ability on cross-domain images (e.g., medical images from ChestX-ray14, remote sensing images from NWPU-RESISC45)? Please present the results and discuss the challenges of adapting VGR to specialized domains.\n\n(20) What is the basis for selecting the MLLM used in data rewriting? Could you compare the rewriting effectiveness of different models (e.g., LLaMA 2 7B, Mistral 7B, Qwen2 7B) using metrics such as reasoning coherence, format compliance, and ground-truth alignment?\n\n(21) Why are 2×2 and 4×4 pooling strategies chosen for visual token compression? Could you conduct ablation experiments with other pooling sizes (e.g., 3×3, 5×5) and present results on performance, computational cost, and information preservation?\n\n(22) How does the model's parameter size (7B vs. 13B) affect the trade-off between performance and computational cost? Please provide detailed metrics including training time, inference speed, memory usage, and accuracy across benchmarks for both sizes.\n\n(23) Is the VGR-SFT dataset balanced across different task types? Please provide a detailed breakdown of data distribution (e.g., percentage of OCR, general VQA, science QA tasks) and, if imbalanced, explain how you addressed it (e.g., oversampling, weighted loss).\n\n(24) What metrics do you use to evaluate the accuracy of region selection in dynamic visual memory replay? Please define the metrics (e.g., IoU with ground-truth regions, precision/recall of key region selection) and present comparative results between VGR and the baseline.\n\n(25) How did you handle duplicate samples in the training data? Please explain the detection method (e.g., hash-based, semantic similarity) and processing strategy (e.g., removal, merging), and provide statistics on the number of duplicates found and processed.\n\n(26) Have you tested the model's performance in few-shot learning scenarios? Please design experiments with varying training data sizes (e.g., 1K, 10K, 50K samples) and present results on key benchmarks, comparing VGR with other few-shot multimodal models.\n\n(27) Could you provide a more in-depth comparison with Chain-of-Spot, including reasoning logic, computational complexity (e.g., number of forward passes), performance on spot-based visual search tasks, and adaptability to high-resolution images?\n\n(28) Does visual memory replay introduce additional latency during inference? If so, could you quantify the latency overhead (e.g., percentage increase in reasoning time) and discuss potential optimizations (e.g., precomputing visual tokens, parallel processing)?\n\n(29) Besides region annotation, could you provide other interpretability analyses (e.g., attention heatmaps, reasoning chain decomposition, visual-linguistic alignment scores) to explain how the model integrates visual and linguistic information for reasoning?\n\n(30) Could you provide detailed license information for all datasets used in the study (e.g., LLaVA-558K, VGR-SFT subsets)? Please specify any restrictions on use, redistribution, or commercial application to address copyright concerns.\n\n(31) How are the pre-training data (LLaVA-558K) and fine-tuning data (LLaVA-NeXT-770K + VGR-SFT) fused during training? Please explain whether the data is concatenated, mixed in batches, or assigned different weights, and justify the chosen method.\n\n(32) Have you evaluated the model's robustness on complex scenarios (e.g., occluded images, blurred images, low-light images)? Please use standard robustness benchmarks (e.g., ImageNet-C, VQA-C) and present results on accuracy degradation compared to clean images.\n\n(33) Please explain the normalization method of bounding box coordinates in the GIoU loss calculation. Are coordinates normalized to [0,1] based on the original image size, cropped patch size, or other standards? Please provide a mathematical formulation if applicable.\n\n(34) How do you evaluate the relevance between visual cues and language reasoning in the VGR-SFT dataset? Please define a relevance metric (e.g., semantic similarity between region labels and reasoning steps) and present statistics on the dataset's relevance distribution.\n\n(35) Could you report the detailed computational resource consumption for training VGR, including the number of GPUs used (e.g., 8×A100), total GPU hours, power consumption, and training time for pre-training and fine-tuning stages separately?\n\n(36) Have you explored combining VGR with reinforcement learning (RL) methods like GRPO? Please provide preliminary results (if any) on whether RL can further improve the model's reasoning accuracy, grounding precision, or efficiency. If not, please discuss the technical challenges and potential benefits.\n\n(37) How does the length of the reasoning chain affect the model's performance? Could you conduct ablation experiments with varying reasoning chain lengths (e.g., short, medium, long) and present results on accuracy, efficiency, and language bias reduction?\n\n(38) How do you evaluate the generalization ability of the 14B annotation model during dataset scaling? Please present metrics such as grounding accuracy, reasoning coherence, and reject rate on unseen data types compared to the cold-start 72B model.\n\n(39) Does reducing visual tokens by 70% lead to information loss in complex visual scenes (e.g., dense objects, complex layouts)? Please design experiments with such scenes and present comparative results between VGR and the baseline (using full tokens) on detail-rich benchmarks.\n\n(40) Have you tested the model's performance in multilingual scenarios? Please evaluate VGR on non-English benchmarks (e.g., VQA in Chinese, German OCR tasks) and present results on accuracy, grounding precision, and language bias compared to English-only performance.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The proposed dynamic visual memory replay mechanism is innovative, as it enables on-demand retrieval of key visual regions during reasoning, effectively mitigating language bias and enhancing fine-grained visual understanding.\n\n(2) The VGR-SFT dataset construction pipeline is rigorous and scalable, avoiding manual annotation bias through model-generated grounding areas and ensuring data quality via reject sampling and rewriting.\n\n(3) The experimental design is comprehensive, including extensive comparisons with state-of-the-art models, ablation studies on core components (e.g., detection loss, visual memory replay), and efficiency analysis, providing solid evidence for the method's superiority.", "weaknesses": "(1) The choice of the 72B MLLM for cold-start data generation lacks comparative analysis; the authors do not explain why smaller models (e.g., 13B or 34B) were not considered, nor do they demonstrate the advantages of the 72B model in generating initial visual grounding data.\n\n(2) The use of Doubao1.5-VL's API for reject sampling lacks detailed parameter settings (e.g., temperature, top-p, response timeout) and verification criteria, making it difficult for readers to reproduce the data filtering process.\n\n(3) The setting of β=2 in the detection loss (combining L1 and GIoU loss) is not supported by ablation experiments; the authors fail to test other β values (e.g., 1, 3, 4) to verify whether 2 is the optimal choice.\n\n(4) The comparison with ZoomEye is incomplete, as it only reports results on V* Bench and HR-Bench 8K, lacking performance data on other key benchmarks (e.g., TextVQA, InfoQA) and detailed analysis of reasoning logic differences between the two methods.\n\n(5) The combination experiments of visual encoders and LLMs are limited; the authors only test a few combinations (e.g., Qwen2.5+SigLIP, Vicuna+CLIP) and do not explore more advanced encoders (e.g., InternViT-6B, EVA-CLIP) or LLMs (e.g., LLaMA 3), limiting the demonstration of the framework's generalizability.\n\n(6) The rationale for setting the maximum number of cropped images to 64 during test-time token scaling is unclear; the authors do not explain whether this threshold is determined by empirical testing or theoretical analysis, nor do they show performance changes when exceeding this threshold.\n\n(7) The contribution of each subset in the VGR-SFT dataset (e.g., AI2D, GQA, ChartQA) to the model's overall performance is not analyzed; it is impossible to determine which data types drive the performance improvement, hindering the understanding of dataset design rationality.\n\n(8) The trigger mechanism for replay signal generation during inference is not detailed; the authors do not clarify how the model decides when to retrieve visual memory (e.g., based on semantic cues in the reasoning chain or statistical thresholds), leading to ambiguity about the core logic of dynamic replay.\n\n(9) Key training hyperparameters are missing, such as batch size, number of training epochs, weight decay, and learning rate scheduling strategy for both pre-training and fine-tuning stages, which is critical for reproducibility.\n\n(10) The model's performance on low-resolution images (e.g., below 336×336) is not evaluated; given that real-world images often have varying resolutions, this omission limits the assessment of the model's practical applicability.\n\n(11) There is no comparison with cutting-edge MLLMs like GPT-4V, Gemini Pro Vision, or Claude 3 Opus; the authors only compare with open-source models, failing to demonstrate VGR's competitiveness against commercial state-of-the-art systems.\n\n(12) The quantification of language bias reduction is insufficient; the authors claim to mitigate language bias but do not use specific metrics (e.g., bias score, fairness indicators) to measure the degree of reduction, making the claim lack objective support.\n\n(13) The details of high-resolution cropping in visual memory pool construction are vague; the authors do not specify key parameters such as crop overlapping ratio, number of crops per image, and selection criteria for crop positions, affecting the reproducibility of the visual memory module.\n\n(14) The annotation error rate of the VGR-SFT dataset is not reported; the authors do not explain how they identified and handled annotation errors during data curation, raising concerns about data quality.\n\n(15) Inference speed on different hardware platforms (e.g., A100, RTX 3090, RTX 4090) is not provided; the authors only report average reasoning time per question on specific benchmarks, failing to reflect the model's deployment feasibility on resource-constrained devices.\n\n(16) Performance in multi-turn reasoning scenarios is not tested; the authors only evaluate single-turn question answering, while real-world applications often require continuous reasoning based on previous interactions.\n\n(17) The specific thresholds for format verification and correctness verification in reject sampling are unclear; for example, the ANLS threshold for closed-ended tasks and the semantic alignment score threshold for open-ended tasks are not specified, making it difficult to replicate the data filtering process.\n\n(18) The detailed structure of the MLP in the detection head is missing; the authors do not mention the number of layers, hidden size, activation function, or output dimension, which is essential for understanding the region detection mechanism.\n\n(19) Generalization ability on cross-domain images (e.g., medical images, remote sensing images, satellite images) is not evaluated; the model is only tested on conventional VQA and OCR datasets, limiting the assessment of its applicability to specialized fields.\n\n(20) The basis for selecting the MLLM used in data rewriting is not explained; the authors do not compare different models (e.g., LLaMA 2, Mistral) for rewriting effectiveness, leading to uncertainty about the optimal choice for this task.\n\n(21) The selection of 2×2 and 4×4 pooling strategies for visual token compression lacks theoretical support; the authors do not explain why these pooling sizes are chosen over others (e.g., 3×3), nor do they provide ablation results on pooling strategies.\n\n(22) The impact of model parameter size (7B vs 13B) on performance is not analyzed in depth; the authors only report basic results but fail to discuss how parameter scaling affects the trade-off between performance and computational cost.\n\n(23) The balance of the VGR-SFT dataset across different task types is not addressed; the authors do not clarify whether the data distribution is balanced (e.g., proportion of OCR vs. general VQA tasks) or how imbalance is handled if present.\n\n(24) Metrics for evaluating the accuracy of region selection in dynamic visual memory replay are missing; the authors do not specify how they measure whether the model selects the correct key regions, making it difficult to assess the effectiveness of the grounding mechanism.\n\n(25) The handling of duplicate samples in the training data is not explained; the authors do not mention whether duplicate samples exist, how they were detected, or how they were processed (e.g., removal, merging), which may affect training stability.\n\n(26) Performance in few-shot learning scenarios is not tested; the authors only use full-scale training data, failing to demonstrate the model's ability to adapt to low-data regimes, which is important for real-world applications.\n\n(27) The comparison with Chain-of-Spot is insufficient; the authors do not discuss differences in reasoning logic, computational complexity, or performance on different task types, limiting the understanding of VGR's advantages over similar interactive reasoning methods.\n\n(28) Latency introduced by visual memory replay is not discussed; the authors do not quantify the additional time cost of retrieving and processing visual tokens, which is critical for real-time applications.\n\n(29) Interpretability analysis is limited; beyond region annotation, the authors do not provide other interpretability methods (e.g., attention visualization, reasoning chain decomposition) to explain how the model integrates visual and linguistic information.\n\n(30) Detailed license information for datasets is missing; the authors only state that datasets are publicly available but do not specify license types (e.g., MIT, CC BY-SA) or any restrictions on use, raising potential copyright concerns.\n\n(31) The fusion method of pre-training data (LLaVA-558K) and fine-tuning data (LLaVA-NeXT-770K + VGR-SFT) is not clarified; the authors do not explain whether the data is concatenated, mixed in batches, or processed with different weights, affecting reproducibility.\n\n(32) Robustness on complex scenarios (e.g., occluded images, blurred images, low-light images) is not evaluated; the authors only test on standard datasets, failing to demonstrate the model's ability to handle real-world noise.\n\n(33) The normalization method of bounding box coordinates in GIoU loss is not explained; the authors do not specify whether coordinates are normalized to [0,1] based on image size or other standards, leading to ambiguity in loss calculation.\n\n(34) The relevance between visual cues and language reasoning in the dataset is not evaluated; the authors do not measure how well the annotated visual regions align with the reasoning chain, affecting the assessment of dataset quality.\n\n(35) Training time and computational resource consumption are not reported; the authors do not specify GPU hours, number of GPUs used, or total training time, making it difficult for researchers with limited resources to replicate the work.\n\n(36) The potential of combining VGR with RL methods (e.g., GRPO) is not explored; the authors only use supervised fine-tuning, failing to discuss whether RL can further enhance the model's reasoning and grounding capabilities.\n\n(37) The impact of reasoning chain length on performance is not analyzed; the authors do not test whether longer or shorter reasoning chains affect accuracy or efficiency, limiting the understanding of optimal reasoning chain design.\n\n(38) The generalization ability of the annotation model during dataset scaling is not evaluated; the authors do not measure how well the 14B annotation model performs on unseen data types, raising concerns about dataset quality during scaling.\n\n(39) The impact of reducing visual tokens by 70% on the model's ability to capture complex visual information is not discussed; the authors do not clarify whether the token reduction leads to information loss in complex scenes (e.g., dense objects, complex layouts).\n\n(40) Performance in multilingual scenarios is not tested; the authors only use English questions and images, failing to demonstrate the model's applicability to non-English languages, which is important for global use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9SVX6DYjCf", "reviewer": "Reviewer_ysjS", "summary": "The paper proposes VGR, a multimodal reasoning framework that lets an MLLM “replay” visual memory on demand during chain-of-thought. It depicts a mechanism when the model emits a special replay signal with a bounding box, VGR fetches the corresponding image tokens from a high-resolution feature map and appends them into the ongoing context for grounded reasoning. The authors also curate VGR-SFT, a supervised dataset mixing reasoning traces with explicit region grounding produced via a cold-start model, rejection sampling, and a smaller “annotator” model for scale. On LLaVA-NeXT-7B backbones, VGR reports strong gains on perception-heavy benchmarks while using ~30% of the visual tokens versus the baseline, with ablations isolating the contributions of replay and a detection loss for box accuracy.", "questions": "1. What fraction of generations emit at least one valid replay signal? What’s the average number of replay events per instance?\n\n2. How often are boxes invalid or out of bounds?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Simple, modular mechanism with clear intuition.**\nTurning visual inspection into an explicit token-level replay step is elegant and easy to graft onto LLaVA-style stacks. The control token and parser logic are well specified.\n\n2. **Efficiency/performance tradeoff.** \nThe expand-then-compress feature design + selective replay cuts visual tokens by ~70% yet improves perception tasks\n\n3. **Data contribution.**\n VGR-SFT explicitly couples reasoning with self-proposed RoIs, avoiding manual box bias and encouraging “ground-then-reason” behavior. The pipeline (cold start → reject sampling → annotator) is thoughtful.", "weaknesses": "1. **Replay supervision and parser brittleness.**\n\n The method depends on the model outputting valid coordinates via \\<sot\\>[x1,y1,x2,y2]\\<eot\\>; while a detection loss is added, the paper doesn’t quantify (i) the rate of invalid boxes and (ii) sensitivity to coordinate quantization / resolution scaling. Metrics for “replay success rate” and its effect on final accuracy would strengthen the claim.\n\n2. **Comparisons to other zoom-then-behavior baselines.**\n\nZoomeye/Chain-of-Spot and guided-search methods aim at similar zoom-then-reason behavior. While the appendix notes a comparison in cost and accuracy,  the main paper doesn’t present head-to-head accuracy/latency plots vs. these strategies under identical token budgets. A direct comparison table would help isolate the value of feature replay vs. re-encoding crops [1-3]\n\n3.  **Latency comparison.**\n\nThe paper provides a comparison on vision token cost, but as the method usually requires additional bounding box tokens and recall of the visual feature map, adding a inference time latency cost comparison compared with LLaVA-Next and other observe-then-reason baseline will help to understand the efficiency of the method.\n\n\n[1]. Chain-of-Focus: Adaptive Visual Search and Zooming for Multimodal Reasoning via RL\n\n[2]. Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models\n\n[3]. ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aaNfXLb9JB", "reviewer": "Reviewer_qwzK", "summary": "This paper introduces VGR, a multimodal large language model framework that augments chain-of-thought reasoning with selective visual grounding via a dynamic visual memory replay mechanism. VGR detects task-relevant image regions during reasoning, retrieves compressed high-resolution visual features from a maintained visual memory pool, and injects them into the LLM input to improve fine-grained visual-linguistic inference. The experiments on eight visual reasoning benchmarks demonstrate that VGR significantly outperforms existing vanilla, SoTA VLMs.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- VGR is a well-motivated and thoughtfully designed approach. The dynamic visual memory replay mechanism is a clever way to enhance the model's ability to focus on relevant image regions during reasoning, which is crucial for fine-grained visual-linguistic tasks.\n- The experiments are well-executed and comprehensive, covering a wide range of visual reasoning benchmarks and a series of ablation studies. The results convincingly demonstrate the effectiveness of VGR, with significant performance improvements over existing methods.", "weaknesses": "- The novelty of this paper is not clearly articulated, especially in relation to very similar prior work on CogCoM [1] and Chain-of-Focus [2]. Both CogCoM [1] and Chain-of-Focus [2] also focus on enhancing visual reasoning by incorporating dynamic, bounding box-based image retrieval mechanisms. The authors should clearly differentiate VGR from these existing approaches and highlight the unique contributions of their method. The reviewer recognizes L060-061 as an attempt to do so by saying \"our dataset empowers models to autonomously attend to arbitrary visual regions during reasoning.\", but the same can be said about CogCoM and Chain-of-Focus.\n- The experiment section could be strengthened by providing a bit more qualitative analysis into why VGR outperforms other methods. For example, sample a handful of examples (e.g., 10-20) where VGR succeeds but other methods fail, and analyze what aspects of VGR's design contribute to its success in those cases. This would provide deeper insights into the strengths of the proposed approach.\n\n[1] CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning. https://arxiv.org/abs/2402.04236\n[2] Chain-of-Focus: Adaptive Visual Search and Zooming for Multimodal Reasoning via RL. https://arxiv.org/abs/2505.15436", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)