Title: Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large multimodal model

grounded reasoning

reinforcement learning
Abstract: Recent advances in large language models have significantly improved textual reasoning through the effective use of Chain-of-Thought (CoT) and reinforcement learning. However, extending these successes to vision-language tasks remains challenging due to inherent limitations in text-only CoT, such as visual hallucinations and insufficient multimodal integration. In this paper, we introduce Point-RFT, a multimodal reasoning framework explicitly designed to leverage visually grounded CoT reasoning for visual document understanding. Our approach consists of two stages: First, we conduct format finetuning using a curated dataset of 71K diverse visual reasoning problems, each annotated with detailed, step-by-step rationales explicitly grounded to corresponding visual elements. Second, we employ reinforcement finetuning targeting visual document understanding. On ChartQA, our approach improves accuracy from 70.88% (format-finetuned baseline) to 90.04%, surpassing the 83.92% accuracy achieved by reinforcement finetuning relying solely on text-based CoT. The result shows that our grounded CoT is more effective for multimodal reasoning compared with the text-only CoT. Moreover, Point-RFT exhibits superior generalization capability across several out-of-domain visual document reasoning benchmarks, including CharXiv, PlotQA, IconQA, TabMWP, etc., and highlights its potential in complex real-world scenarios.


Peer reviews:
comments: SUMMARY:
This paper presents Point-RFT, a two-stage framework for enhancing multimodal large language models (MLLMs) with visually grounded chain-of-thought (CoT) reasoning for visual document understanding. In the first stage, the model is trained to generate visually grounded CoT rationales explicitly aligned with visual elements via point annotations. In the second stage, the model undergoes reinforcement finetuning with Group-wise Relative Policy Optimization (GRPO) to improve answer correctness and generalization. This paper introduces a large, cross-validated Point-CoT dataset and demonstrates substantial gains on ChartQA (in-domain) and several out-of-domain benchmarks, reporting improved accuracy and interpretability over strong baselines.

STRENGTHS AND WEAKNESSES:
[Strengths]
1. The idea of extending text-only CoT to visual-grounded CoT is novel and well-motivated.
2. The pipeline for constructing the Point-CoT with cross-model validation mechanism reduces errors caused from unimodal.
3. The analysis of each component in Robot-RFT is comprehensive.

[Weaknesses]
1. **Comparison with Prior SOTA**: The discussion and quantitative comparison with other recent methods for visual document understanding is limited.
2. **Dataset Overlap/Out-of-Domain Clarification**: There is confusion about what is considered out-of-domain. For instance, PlotQA is listed as both a source for training data (Tab. 1) and as out-of-domain in the evaluation tables (Tab. 2 and Tab. 3) .

QUESTIONS:
Can Point-RFT be applied to or benefit those general image QA understanding tasks, such as Seed-bench2 [1]?

[1] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal large language models." CVPR 2024.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Point-RFT, a reinforcement fine-tuning framework that enhances multimodal reasoning by integrating visual grounding into the chain-of-thought (CoT) process. The method proceeds in two stages: first, a supervised format fine-tuning stage using a newly curated dataset (Point-CoT) with step-by-step reasoning grounded to visual elements; second, a reinforcement fine-tuning stage using GRPO to optimize both answer correctness and reasoning format alignment. The paper also introduces a dataset, Point-CoT, a novel multimodal CoT dataset. Experimental results show that Point-RFT significantly improves in-domain accuracy on ChartQA and generalizes well to multiple out-of-domain visual reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper focuses on an interesting and meaningful problem. It presents a two-stage training pipeline combining supervised fine-tuning with reinforcement fine-tuning using GRPO. The empirical evaluation is thorough, covering both in-domain (ChartQA) and diverse out-of-domain datasets (e.g., PlotQA, IconQA), with consistent performance gains. The paper is clearly written, well-structured and easy to follow.\n\n**Weakness**\n\nWhile the use of GRPO in Point-RFT is effective, the paper does not clearly articulate what new insights or innovations it brings beyond prior applications of GRPO in other multimodal or reasoning tasks. A clearer justification of the novelty and insights would strengthen the contribution, and this is my major concern.\n\nThe paper shows strong results on visual document understanding benchmarks, but it remains unclear whether Point-RFT can generalize to broader vision-language tasks, such as those involving non-document images. This limitation weakens the overall persuasiveness of the claimed generalization ability of Point-RFT.\n\nOther weaknesses can be found in the Questions section.\n\nQUESTIONS:\n- The paper shows strong generalization on visual document understanding tasks, but it remains unclear whether the proposed Point-RFT framework can extend to broader vision-language reasoning domains. Could the authors evaluate the method on more general VQA-style benchmarks such as MMBench, MMStar, MMVet, or MathVista, which feature diverse visual contexts and question types? A positive result on such benchmarks would strengthen the claim of generalizability and showcase the method's applicability beyond charts and plots.\n\n- In Table 3 (Out-of-Domain), the base Qwen2-VL-7B model achieves a higher average performance than the Qwen2.5-VL series, yet the performance gain after applying Point-RFT appears smaller. Could the authors clarify why the improvement margin differs so much?\n\n- As shown in Table 1, the Point-CoT dataset already includes samples from PlotQA. However, in Table 2 and Table 3, PlotQA is listed as an out-of-domain (OOD) test set. Could the authors clarify this inconsistency? \n\n- The paper includes ablations on format syntax and grounding, but does not isolate the effects of GRPO vs. more standard RL approaches like PPO. Would the benefits of Point-RFT still hold with simpler optimization strategies?\n\nIf your response can address my concerns, I will consider increasing my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes Point-RFT, a novel post-training method designed to enhance multimodal reasoning performance. Point-RFT first constructs the Point-CoT dataset using a general-purpose reasoning model, such as GPT-4o, and a grounding expert model, like Molmo-7B. The Point-CoT dataset consists of reasoning trajectories that include coordinates within images and explanations of those regions. Then, a target model is supervised fine-tuned on the constructed dataset. Finally, the SFT-tuned model is further improved through reinforcement learning. The experiments demonstrate that Point-RFT is effective for document understanding tasks, and the results reveal that the point grounding technique is the primary factor behind this effectiveness.

STRENGTHS AND WEAKNESSES:
**Strrengths**
1. This paper is clear and well-organized overall.
2. The proposed idea is novel and closely mirrors the way humans reason.
3. The experiments demonstrate that the proposed method is effective on in-domain and out-of-domain document understanding tasks.

**Weaknesses**
1. Table 1 shows that Point-SFT already outperforms the Baseline and Base-RFT by a large margin. A deeper analysis of why the format training leads to such improvements would further strengthen the paper’s contribution.
2. How does the proposed method perform when applied to standard VQA tasks? If it shows limited effectiveness or results in performance degradation, what are the underlying reasons?

QUESTIONS:
1. Why does the format training lead to such improvements?
2. How does the proposed method perform when applied to standard VQA tasks? If it shows limited effectiveness or results in performance degradation, what are the underlying reasons?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multimodal reasoning framework that combines with visually\ngrounded chain-of-thought (CoT) supervision. It includes two-stages: first\nconstructing a 71K-sample dataset with step-by-step visual grounding, and then\napplying GRPO-based RFT to optimize for both reasoning accuracy and format\nadherence. Experiments on different datasets demonsrate the effectiveness of the\nproposed method.\n\nSTRENGTHS AND WEAKNESSES:\n#Strengths\n\n* The studied problem is interesting. The motivation of this paper is clear.\n\n* The empirical performance of the proposed framework is good.\n\n#Weaknesses\n\n* While the proposed framework shows clear engineering effort, the core\ncontribution is mainly in constructing a new dataset by distilling\nGPT-4o-generated reasoning traces with Molmo-7B-based visual grounding. The\nnovelty here lies more in implementation and data formatting rather than in new\nparadigms, and may be considered somewhat incremental.\n\n* The concept of visually grounded chain-of-thought (CoT) is not entirely new —\nprior works such as Visual CoT and Sketchpad have explored similar ideas in\ndifferent forms.\n\n* The empirical evaluation focuses almost exclusively on chart-based visual\nreasoning (e.g., ChartQA, PlotQA, CharXiv), with only a small-scale experiment\non Counting. However, multimodal reasoning is a broad area that spans various\nformats such as natural images, science documents, and embodied AI related\ntasks. Without testing the method on more diverse benchmarks on different\ndomains, it’s difficult to evalaute whether the proposed approach generalizes\nbeyond chart-based data.\n\n* All results are based solely on Qwen-VL models. There is no evaluation\non other model families (e.g., InternVL3 and LLaMA 3.2 vision) or larger models.\nIt is unclear whether the method scales to stronger models or transfers across\ndifferent VLM architectures.\n\nHu et al., Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. NeurIPS 2024\n\nShao et al., Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning. NeurIPS 2024\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}