Title: VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Chain-of-Thought; Multimodal Large Language Model; Region Recognition;
Abstract: Recently, reasoning-based MLLMs have achieved a degree of success in generating long-form textual reasoning chains. However, they still struggle with complex tasks that necessitate dynamic and iterative focusing on and revisiting of visual regions to achieve precise grounding of textual reasoning in visual evidence. We introduce VLM-R³  (Visual Language Model with Region Recognition, Reasoning, and Refinement ), a framework that equips an MLLM with the ability to (i) decide when additional visual evidence is needed, (ii) determine where to ground within the image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved chain-of-thought. The core of our method is \textbf{Region-Conditioned Reinforcement Policy Optimization (R-GRPO)}, a training paradigm that rewards the model for selecting informative regions, formulating appropriate transformations (e.g. crop, zoom), and integrating the resulting visual context into subsequent reasoning steps. To bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual Interleaved Rationale (VLIR) corpus that provides step-level supervision on region selection and textual justification. Extensive experiments on MathVista, ScienceQA, and other benchmarks show that VLM-R$^3$ sets a new state of the art in zero-shot and few-shot settings, with the largest gains appearing on questions demanding subtle spatial reasoning or fine-grained visual cue extraction.


Peer reviews:
comments: SUMMARY:
This paper proposes VLM-R3, a vision-language model framework equipped with an adaptive ROI (region-of-interest) cropping mechanism that enables autonomous selection of optimal visual input regions before actually answering questions. To develop this capability, the authors first constructed the VLIR dataset through multi-stage curation processes, which was subsequently used for supervised fine-tuning (SFT) followed by reinforcement learning (RL) optimization. Comprehensive evaluations, including both quantitative metrics and qualitative analyses, demonstrate the effectiveness of VLM-R3 in achieving accurate question answering while optimizing computational efficiency through intelligent region selection.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The motivation is clear and quite reasonable.
2. The overall paper is easy to follow.
3. The presentation is clear.

**Weaknesses**
1. Insufficient evaluations. VLM-R3 is mainly evaluated on LLM-centric benchmarks such as MathVista, MathVision, MMMU, and ScienceQA. These benchmarks could not actually reflect the ability of "grounding then answering", as vanilla CoT prompting also brings significant improvements on these benchmarks. Therefore, the authors should evaluate on *vision-centric* benchmarks like MMVP, CV-Bench, V*, HR-Bench, and MME-RealWorld. Furthermore, the vanilla text RL baseline should be incorporated.
2. The definition of the grounding accuracy illustrated in Figure 3 is not clear. A flowchart, pseudo-code, or equation is strongly encouraged.

QUESTIONS:
N/A

LIMITATIONS:
The authors **DID NOT** provide discussions on limitations.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main idea of the paper is to select the most relevant image regions to answer questions. Paper uses a region-conditioned reinforcement policy optimization training that rewards the model for selecting informative regions, formulating appropriate transformations (e.g.crop, zoom), and integrating the resulting visual context into subsequent reasoning steps.  The paper also introduces Visuo-Lingual Interleaved Rationale (VLIR), a dataset curated to support the development of MLLMs for interleaved text-image CoT reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n+ Paper addresses an important problem.\n+Automatically generation of bounding boxes for the reasoning steps is interesgting.\n\n- The introduction is too repetitive and their are many over-claims such as  \"pioneering dataset\", \"meticulously curated\" , \"a novel framework\", \"master this intricate reasoning\". some of these claims are not factual.\n- Some of the limitations mentioned in [42] has been addressed by many recent works.\n- Paper does not discuss more recent work on this topic.\n-The data filtering and rejection sampling processes rely on small VLMs and LLMs, whose outputs may not be reliable or verifiable.\n\nQUESTIONS:\nLine39-40: Can authors clarify why it is need for a visual model to look at the image after each reasoning step? Can't attention already takes this into account?\n\nIt is unclear whether the paper employs a specific cropping method. Modern vision-language systems, such as Qwen-VL, are capable of returning bounding boxes in response to specified prompts.\n\nLIMITATIONS:\nThere is not much novelty in this work. Section 3.3 and the rest of the method is simple application of existing methods.\n\nAs the VLIR is primarily constructed using automatic annotations, the size of the dataset is really small to count as a contribution. If the data is fully human annotated, the size of the dataset is acceptable. \n\nThere are no meaningful visualizations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
Introduces a new dataset, Visuo-Lingual Interleaved Rationale (VLIR), for supervised fine-tuning of VLMs. Extends GRPO to Region-GRPO and uses it to train a new model, VLM-R^3, based on Qwen2.5-VL 7B, which can automatically perform image grounding and zooming-in during reasoning. Results show that VLM-R^3 outperforms Qwen across seven different datasets, especially on ScienceQA.

STRENGTHS AND WEAKNESSES:
Strengths:

* The VIRL dataset will be useful for research in visual reasoning
* The performance of VLM-R3 is impressive
* Ablation studies demonstrate the effectiveness of the proposed method

Weaknesses:

* small issues related to formatting and visualizations

QUESTIONS:
1. How does R-GRPO prevent the model from cropping too many regions during reasoning?

2. According to the ablation in Figure 3, random cropping harms performance. As noted in Section 4.5.2, traditional MLLMs tend to lose visual context over long reasoning steps, leading to errors—I agree with this observation. However, to better demonstrate the effectiveness of cropping, it would be helpful to compare against a baseline where the input image is periodically re-inserted, e.g., by adding prompts like “Let’s look back to the input image \<image_token\>.”

3. The example in Figure 1 can be better. I can hardly recognize "HONGDONG" characters even in the zoomed image

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VLM-R3, a framework that enhances multimodal large language models with dynamic visual reasoning via region recognition and interleaved chain-of-thought. It leverages a curated VLIR dataset and a novel R-GRPO training strategy to enable adaptive region selection and integration during reasoning. VLM-R3 achieves state-of-the-art performance on benchmarks like MathVista and ScienceQA, especially in tasks requiring fine-grained visual grounding.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The writing is clear and easy to follow. The ideas are well explained and the structure makes sense.\n\n2. The motivation is solid and the design is thoughtful. It tackles a real limitation in existing models.\n\n3. The model performs well on key benchmarks, showing both practical value and strong impact.\n\nCons:\n1. The model’s success seems to hinge on domain-specific training using the carefully constructed VLIR dataset. Without it, the model fails to follow the correct formats or issue crop commands properly, even performing worse than the base Qwen model. This makes the method costly to adapt to new tasks and may limit its broader applicability.\n\nQUESTIONS:\nOne potential concern is that the proposed VLIR dataset is custom-built to favor region-level visual reasoning, which may amplify the advantages of VLM-R3. This raises the question of whether the observed gains would generalize to more natural or unbiased datasets with a broader range of reasoning types. Also, in real-world data, how many cases actually require such a technique is questionable. If the authors could evaluate on publicly available benchmarks without custom selection or filtering, the results would be more convincing.\n\nLIMITATIONS:\nA potential limitation is the scalability of the proposed approach. While the VLIR dataset is partially auto-generated, it still requires careful construction, prompt engineering, and task-specific filtering to ensure high-quality interleaved reasoning chains with region-level supervision. Extending this method to new domains may require significant adaptation effort, which could limit its practicality in low-resource or rapidly shifting scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}