Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Authors: Minheng Ni
Zhengyuan Yang
Linjie Li
Chung-Ching Lin
Kevin Lin
Wangmeng Zuo
Lijuan Wang
Venue: NeurIPS 2025 poster ()
Keywords: Large multimodal model
grounded reasoning
reinforcement learning
Primary area: applications
TL;DR: We propose Point-RFT, a multimodal framework using visually grounded Chain-of-Thought reasoning with two-stage finetuning, which exhibits superior generalization capability and potentials in complex real-world scenarios.

Abstract: Recent advances in large language models have significantly improved textual reasoning through the effective use of Chain-of-Thought (CoT) and reinforcement learning. However, extending these successes to vision-language tasks remains challenging due to inherent limitations in text-only CoT, such as visual hallucinations and insufficient multimodal integration. In this paper, we introduce Point-RFT, a multimodal reasoning framework explicitly designed to leverage visually grounded CoT reasoning for visual document understanding. Our approach consists of two stages: First, we conduct format finetuning using a curated dataset of 71K diverse visual reasoning problems, each annotated with detailed, step-by-step rationales explicitly grounded to corresponding visual elements. Second, we employ reinforcement finetuning targeting visual document understanding. On ChartQA, our approach improves accuracy from 70.88% (format-finetuned baseline) to 90.04%, surpassing the 83.92% accuracy achieved by reinforcement finetuning relying solely on text-based CoT. The result shows that our grounded CoT is more effective for multimodal reasoning compared with the text-only CoT. Moreover, Point-RFT exhibits superior generalization capability across several out-of-domain visual document reasoning benchmarks, including CharXiv, PlotQA, IconQA, TabMWP, etc., and highlights its potential in complex real-world scenarios.

Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Minheng Nil.21, 1, Zhengyuan Yang Linjie Lis, Chung-Ching Lin Kevin Lin
Wangmeng Zuo Lijuan Wang S
Hong Kong Polytechnic University Harrbi Institute of Technology
3 Microsoft
(a) Question: What difference percentage between the years 1979 and 1986?
Qwen-VL
Point RFT
In 1973, the gross enroliment rañoo in sertary education is aporosinataly 5% to 1986, the gross encaliment app
eurolinccat
Therefore, the answer is 4%.
Ihe representing Gunca enro Immet rato
Point-RFF RFT
1 reprnsenting encolmenr
@ line
Gurne''s enrollment
In- nn-oomein
rabo reaches
Ine repr esenting
To find the difference, subtrett the value 1966 from the value in 1979 5% 72%
Point AFT
Guinca's enroliment rate
$5.16
The difference in perten between the yours 1979 and 1980 5%
1804
Qwen-Vi 35.94
Therefore, the answer 3.
(c) Point-RFT
Supervised Format Finetuning (SFT) with Point-CoT Dataset
Visually Grounded Reinforcement Finetuning (RFT) with GRPO
Out Domain
Point-RFT improves multimodal reinforcement finetuning with visually grounded COT. (1) We first construct a Point-CoT dataset for Supervised Format Finetuning (SFT). It allows the model to generate step-by-step reasoning traces explicitly linked to visual pointing, mitigating hallucinations and enhancing pnrception-reasonigg alignment. (2) Reinforcement Finetuning (RFT) with GRPO: Optimizes answer correctness and grounded rationale coherence by rewarding localized visual-textual reasoning paths. Our model, code, and dataset can be found at this link
Figure 1:
Abstract
Recent advances in large language models have significantly improved textual reasoning through the effective use of -hhouhh (CoT) and reinforcement learning. However, extending these successes to vision-language tasks remains chal- lenging due to inherent limitations in text-only CoT, such as visual hallucinations and insufficient multimodal integration. In this paper, we introduce Point-RFT, a multimodal reasoning framework explicitly designed to leverage visually grounded CoT reasoning for visual document understanding. Our approach consists of two stages: First, we conduct format finetuning using curated dataset of 71K diverse visual reasoning problems, each annotated with detailed, step-by-step rationales explicitly grounded to corresponding visual elements. Second, we employ rein- forcement finetuning targeting visual document understanding. On ChartQA, our approach improves accuracy from 70.88% (format- -ineatt-d baseline) to 90.04%, surpassing the 83.92% accuracy achieved by reinforcement finetuning relying solely on text-based COT. The result shows that our grounded CoT is more effective for multimodal reasoning compared with the text-only CoT. Moreover, Point-RFT exhibits superior generalization capability across several out-of-domain visual document reasoning benchmarks, including CharXiv, PlotQA, IconQA, TabMWP, etc., and highlights its potential in complex real-world scenarios.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large reasoning models (olb 2024 Guo et al. complex text-based problems, such as STEM reasoning (Lightman et al. 2023 Rein et al.) 2024)
2025 have shown impressive capabilities in solving
and coding tasks (Jain et al. 2024b Quan et al. 2025). Central to their success is the large language
model's s ability to improve responses by expanding the textual Chain-of- Thouuhh (CoT) (Wei et al.
2022), also known scaling inference-time compute (Brown et al. 2024; Snell et al. 2024).
Reinforcement learning (Mnih et al.) 2015 2016 [Shhlmann et al. 2017 Shao et al. 2024 further optimizes these models by maximizing the rewards associated with these generated longer, more comprehensive, and accurate responses. Despite these successes in textual reasoning, adapting them to vision-language tasks remains challenging, with unique open challenges such imperfect visual perception (Fu et al. (024) as well as integrating reasoning across both visual and text modalities (Hao
al. 2025
Inspired by the introduction of visual-centric tokens Aasion-language models YYnng et al. 2022a Jain et al., 2024a Dong et al.) [203] Yang et al.) 2022b Wang et al. (022b) to improve visual perception, we present FooittRFF. with the insight that the effective CoT for multimodal reasoning should contain grounded visual tokens. To achieve this, first perform format finetuning across diverse problem sets to enable Point-RFT to generate grounded CoT rationales. Subsequently, the format-finetuned model undergoes reinforcement finetuning with outcome reward specialized visual document datasets, allowing it to explore the effective usage of grounded CoT in solving visual document questions.
To curate our format finetuning dataset, we establish a pipeline that integrates step-by-step text rationales generated by a large multimodal model (Hurst et al., 2024) with a grounding model (Deitke) et al., 2024) that links relevant reasoning steps explicitly to visual points. The resulting dataset comprises 71K images covering wide range of question types (Guo al., 2024). This diversity in the dataset design ensures that the model generalizes its learned formatting strategies broadly, allowing it to explore effective solutions for various problems during subsequent reinforcement finetuning. After the format finetuning for grounded CoT, we conducted reinforcement finetuning (Shao et al. 2024) specifically the ChartQA dataset (Masry et al. 2022). Experimental results demonstrate notable performance improvements: accuracy increased from 87.21% to 90.04% compared to the Supervised finetuning (SFT) baseline and from 83.92% 90.04% compared to reinforcement learning using text-only COT. These results show the effectiveness of our Point-RFT and the grounded Co'T approach for enhancing multimodal reasoning capabilities.
Beyond the performance improvements on ChartQA, we note that Point-RFT demonstrates several compelling properties for multimodal reasoning: (1) Improved generalization capabilities compared to supervised finetuning and reinforcement learning with text-only COT, with superior performance on various
tasks including ChartXiv (Wang al. 2024). PlotQA (Methani et al., 2020), IconQA (Lu et al. 2021). TabMWP (Lu et al., 2022b), etc. (2) Enhanced pointing accuracy not only contributes directly figher-quality final answers but also provides increased interpretability. This capability allows the disentanglement of perception errors from reasoning errors, facilitating more precise diagnostics and targeted improvements.
Our contributions are summarized as follows.
We present, Point-RFT, the first multimodal reasoning framework explicitly designed for visually grounded reinforcement fine-tuning. We empirically show that "visually grounded CoT" is more effective for multimodal reasoning compared with text-only thoughts.
We curate a 71K-example dataset where every reasoning step is aligned with point-level visual references, enabling supervised format finetuning that teaches the model to "think while pointing." " We then perform RFT with grounded COT.
On ChartQA, Point-RFT boosts accuracy from 70.88 % to 90.04 %. It also achieves the best average score over five out-of-domain benchmarks, demonstrating superior generalization ability and interpretability.

---

Point-CoT
Question: Compone the levels of sotisfaction with demoerasy coross difforent countries and regions. Discuso how outtural, politcall and wamnomic sontacts influence these levila lasiarlaction.
Step1 1
Step 2
Step
Stop 4
Stop
MEDHO Tome
oegiment
Bist
sarey The addcss ve snall petes
wving wedked Lowi cnnetaa incie
Mepeeae leeel decutteeet the
ad
the
hece anltetie can shoue
Step 6
Step? 7
Stop 8
Stop 9
Stop 10
The de chat the rean
the
hadd paieed
Caked when
and ump
aarrocio Feaurders acpeeninng goet har
vith
catkual and sace
a0d40% bang mouaaal uniled (cuag
uade and ynl bdora rosss dreats souate Conunnys seet budense
sew cilcci auhe laa tapikes
dersoney and inasadions arying
sarae and "oomever untiter
GT Answer: Sarisfaction whth de demooseey con vary sigh foardd scross diforennt countrie and regione, infueeodd by a mutthuue cutural, political and economio fectors, For Insoa ice, counties with a long Nistory of stoble denocratic govemanoe and strong Insth instt tend to have Nigher levess of sadisacction controst, regions exparter cing poliical freedoons Adiiionall, coono mi. prosprity and equitacle woath distibution can coolteely impoot satiffcciion with damocrooy while aconomio inoquality and poverty can lead to caelus enmam. Understanding these varaatoms can provide insights into how improve democratic processen and entance citie stuction globurly.
Figure 2: Visualization of Point-CoT dataset. Point-CoT dataset integrates the reasoning process of answering questions with point grounding, creating a novel form of multimodal CoT.
2
Related Work
Large Reasoning Language Models. Recent studies lb 2024| Guo et al. 2025) have demonstrated
that reinforcement learning (Mnih et al., 2015 2016: Schulman et al.] 2017 Shao al.
2024 can
significantly enhance the reasoning capabilities of language models (Lightman et al. 2023
Rein et al.
2024, Jain al. 2024b| Quan al. 2025). The approach involves the use of chain-of of-tho (CoT) reasoning to systematically generate high-quality responses, coupled with outcome- outcome-based reward optimization, enabling models explore and identify effective solutions. However, a purely textual CoT may be sub-optimal for vision-centric (Fu et al. 20244] Rahmanzadehgervi et al. 2024 Hua
jet al. 2024) Tong et al.) (202) and vision- language tasks Yuu et al. [023] Hao et al. 2025). In
this work, we propose and investigate grounded CoT approach to improve multimodal reasoning.
Vision-Language Models. Advancements in vision-language modeling (VLM) (Radford al.
202T; Yuan et al.) [02T] Wang et al.] 2022c a Yu et al., 2022: Li et al., 2024 Hu et 2022) have sigmificantly enhanced the joint understanding of visual and textual inputs, benefiting various downstream tasks such as image captioning (Chen et al. (2015) and visual question answering (Goya
2017). Recently, contemporary VLMs (Alayrac et al. 2022; OpenAl 2023a: Yang al. ct al.
2023
Liu et al. 2023: Wang et al. 2023; Chen et al. 2024, Bai et al. 2023 Team 2023) have
integrated multimodal modeling techniques with large language models (LLMs) (OpenAl| 2023b)
Grattafiori et al.) 2024 Yang et al., 2024). resulting in enhanced capabilities including instruction- following, in-context learning, and improved zero-shot generalization. Nevertheless, the application of reinforcement learning to enhance multimodal reasoning (Xiyao et 2024 /Zhou et al. 2024 Yu jt al. [2044a] (Sun et al. 2023] Wang et al. (025a) remains relatively underexplored due to the unique challenges associated with extending outcome reward-based reinforcement learning approaches from purely textual to multimodal.
Multimodal Chain-of- Chain-of-Thought. Multimodal -ftho-hf- reasoning reasoning Wang et al. (2025b) has recently attracted intense attention. Early works focus on text-only thoughts for mul- timodal problems Lu et al.) (2022)); Zhang et al. (2024); Xu et al. (2024), showing that explicit step-by-step reasoning boosts visual- qww.lll-uuuioooo-n -anworrnnnwwnnnng accuracy. More recent studies extend this idea to true multimodal thoughts Rose et al. (2023); Wu et al. (2024): Hu et (2024) ;; Fuu et al. (2025); Ma et al. (2024), including having models directly generating visualizations, as well as calling vision tools that generate crops and other augmented views. Despite this progress, all existing multimodal chain-of-thought approaches remain predominantly prompt-driven, rather than end-to-end RFT, as explored in this study.
3

---

3 Method
Our core hypothesis that reinforcement finetuning with grounded chain-of-thougg (CoT) reasoning enhances visual-language reasoning capabilities. To investigate this, we propose a two-stage training framework: (1) supervised fine-tuning (SFT) with format-finetuning dataset, to support grounded CoT, followed by (2) GRPO-based RI. optimization with outcome reward signals. Section 3.1 introduces the format fintuning data collection pipeline, and Section 3.2 details the model training approach.
Table I: Statistics of dataset. The final dataset comprises 71K images with diverse question type.
This mixture ensures coverage of both free-form and structured visual reasoning scenarios.
Source
Samples
Ratio
Turn
Points
ChartQA
29533
41.5%
3.97
2.02
DVQA
25269
35.5%
8.78
5.79
PlotQA
14055
19.8%
8.71
5.89
CLEVR (MathV360K,
1322
1.9%
4.61
3,43
TallyQA
931
1.3%
2.93
2.01
3.1 Point-CoT: Format-Fiin Fermat-Finetune Dataset
To enable models explicitly understand and interact with content in images during multimodal rea- soning, we propose the Point-CoT dataset. As shown in Figure 2 the Point-CoT dataset integrates the reasoning process of answering questions with point grounding, creating a novel form of multimodal chain-of-thought (CoT).
Our data generation pipeline addresses two critical challenges in visual-language reasoning: 1) synthesizing logically coherent reasoning chains with explicit visual grounding and 2) ensuring consistency between textual rationales and spatial referring. We design a hybrid pipeline that combines GPT-4o and Molmo-7B, leveraging their complementary strengths: GPT-4o excels at generating human-likk reasoning patterns, while Molmo-71 specializes fine-grained visual localization due to its pretraining on large-scaled geometric annotation tasks. Figure illlstrates the overall dataset generation pipeline.
Step 1: CoT Generation with GPT-4o. For each datasets, GPT-4o generates comprehensive, step-by-step reasoning traces with text tokens only. Each step may explicitly reference visual elements (e.g.. chart components) crucial for deriv- ing the correct and convincing answer. For the referenced visual elements, the model will be required to generate their descriptions, i.e., ...itt....///.tt>> and the number of objects, ppoint_result>.... http...///ooit_esiuiill
eequestioo-answer. triplet from Mammoth- VL
Step 2: Points Grounding Generation with Molmo-7B. We then use Molmo-7B to annotate coordinates of points {zo, lo. I1, V1, I2, 3/..... for visual elements mentioned in each reasoning step, Coordinates follow raw image resolution rather than normalized scales, and points are formatted <points x0=" After point generation, cross- validataio on counts of points between GPT-4o rationales, i.e., number in spoint_resultt ... </point_resultt result> and Molmo-7B annotations will be conducted, and will discard points with mismatches after eight retries.
y0="..
pesscription//pooii
preceding each reasoning step.
The validation process improves the integrity and accuracy of the data, enhancing the quality of the Chain-of-Thought (CoT) reasoning from two perspectives. First, by using Molmo-7B for visual grounding, hallucinated elements in GPT-4o's reasoning are filtered out because they will fail the grounding process. Second, if Molmo-78 fails to ground or provides incorrect annotations, the number of visual elements mentioned in GPT-4o's text (e.g.. "three bars in the chart") will not match the number instances grounded by Molmo-7B. Thus, a model's 's visual perception is accurate, the resulting CoT reasoning will also be accurate. This rigorous process reduces annotation errors compared to single-model approaches.
The final dataset comprises 71K images with diverse question types (counting, comparison, arithmetic) from the subset of Mammoth- VI.. This mixture ensures coverage of both free-form and structured visual reasoning scenarios.

---

Quary:
agas peimary secontary
idtenity
Point-CoT
agdint vasutriti spoint, reaatt
rpeinreh make the paints
vtints on yourth in 70ur veition ceetmieee moarttie the courtien lnted Seadeiii giiniaaeiin migeette
chrnwayg Dypres
Germany Couch fepul
rgalitt resuttrii- neaitt
qu ens empoevetie spanwey wi sooe eps eantie oedheeaeee
tp parernaeeniing zoeer aucenday educatinn agaripaiee agdint renutei--jpooet, reaute
rpdirs ot wor.gyyyyggg........ 6 23.8 you -0.2 23err 38,6 yon yo- S4--0 yeangg bare represarn lover escondary aducetion antrancc age sugaastnn Thais aicbars
The paraa eck cater uxciesio ges ahiiiis Creata, Gerrarre Cyptus piimay secendary educetos.
Couch Fazublic are indicssed consbraet of the
and Couch the
hiight blue
bar stoep
apos
Creak Hepurtie ar
plimay
adacation vitacs
county agus
The patten prvaeee in oourtie accanday aducrton.
rot prower the dert on of bruns agoo primary and lover vesonday ecticatinn acros these counter laae de- lan cmmuace agm Tie bookica wouict There ane woration
The danibuun of entrenee sse ptimeyy and lover seccnday edkisaton strop the reaamer have the cine esttait lgen bilt livel whlotiie Tanv are wolrdoww
Step 1 CoT Generation with GPT-40
Step 2 Paints Grounding Generation with Molma-7B
Figure 3: Overall dataset generation pipeline. The whole construction process combining LLM reasoning (GPT-40) and geometric grounding (Molmo-7B). The pipeline ensures spatial-textual consistency through cross-validation, producing our Point-Co'T dataset.
3.2 Model Training
Supervised Format Finetuning (SFT) with Point-CoT. We fine-tune Qwen2.5-VV VL the base model using our format finetuning dataset. Each training instance adheres to the following format:
<think>
<points ...>... /....<//oints.....
<points ................................ ........//oontttt </points>...
</think>
<answer>... </answer>
The template design serves three purposes: 1) explicitly separating visual grounding from textual reasoning, 2) enabling deterministic parsing of coordinate references during evaluation, and 3) forcing the model to attend to spatial features before generating answers. Our ablation studies indicate that interleaving point annotations with textual content significantly enhance answer accuracy, suggesting that the model benefits from the alignment between spatial referring and thinking in grounded CoT.
Let 0 be model's parameter and T' be the length of the text. The loss function is:
[())= := (R..)) Drius log P x (eee; 0);
(I)
where (I,y) y) is the query and target response with thinking in dataset Droin--ooT
Reinforcement Finetuning (RFT) with GRPO. Despite obtaining point-grood grounded reasoning capa- bility, initial SFT may cause performance drops in OOD domains, suggesting overfitting to synthetic reasoning patterns. To address this, we apply Reinforcement Learning (RL) to further optimize it.
We use the Group-wise Relative Policy Optimization (GRPO) algorithm (Shao al. 2024) as the optimization method, which extends PPO by introducing task-specific advantage normalization. We implement GRPO with dual rewards to tackle the tension between structural and semantic correctness:
</think> <answer> </answer> template via regex matching. It employs the graemar-level regex parser that checks 1) thinking process integrity and 2) whether the answer can be extracted from the answer tag.
Format Reward (Rt) measures structural adherence to the <think
5

---

Accuracy Reward (Ra) computes answer correctness against ground-trute answer using
Rs = I - y).
The combined reward R = R2 + Ra is optimized on 15K ChartQA vanilla training data by:
[(0) : E(z.y)>
To R B DKL. (ToI|[TSFT)
where 3 is a haper-paramann parameter of KL. divergence, and πиr, Tæ, and Tiy the model after SFT, the optimized model and the model before GRPO optimization.
4
Experiment
4.1
Implementation Details
All models use A optimizer with 5 x 10-6 learning rate, and 512 batch size for SFT and 56 batch size for RL. Training converges in 500 steps for SFT and 100 RL steps with B = 0.00. We use soft matching for numeric answers (tolerating +5% relative error) and exact matching for other responses GRPO. We implement our two-stage training pipeline using PyTorch with 8x A10 GPUs based on Easy-RI (Zheng et al. 2025)
4.2 Experimental Setup
We evaluate approach on six multimodal reasoning benchmark datasets spanning diverse domains:
ChartQA (Masry et al.
2022 y A chart understanding task requiring extraction and reason-
ing over chart elements. We use the official test split for evaluation.
2024) Scientific charts from academic papers with complex visual encodings. We use the official validation split for evaluation.
CharXiv (Wang et al.
PlotQA (Methani et al.
2020): Statistical plots testing trend analysis and numerical reason-
ing. We randomly sampled 2,000 examples as the test split.
IconQA (Lu et al.
2021): Iconographic reasoning requiring symbolic interpretation. We
randomly sampled 2,000 examples as the test split.
TabMWP (Lu
2022b): task involving tabular math problems that combine table parsing and arithmetic reasoning. We use the official 1,000-sample mini-test split.
Counting (Li et al. 2023): Based on the SuperCLEVR dataset, we sampled 200 examples for evaluation.
We compare the following approaches: (1) Base-RFT: A baseline model trained with supervised finetuning Point-CoT without explicit point annotations, followed by reinforcement learning finetuning with GRPO on CharlQA. This serves as our primary baseline. (2) Point-SFT: A model trained with supervised finetuning on Point-CoT with explicit point annotations. (3) Point-RFT: A model based on Point-SFT that undergoes reinforcement learning finetuning with GRPO on ChartQA. (4) Base:
The backbone model without any finetuning is also included in the tables reference.
For the base model and Basc-RFT, we only require the model output the reasoning process and the final answer. For Point-SFT and Roint-RFT, we additionally require the model to output reasoning process with explicit point annotations, along with the final answer.
To evaluate all models, we extract the content from the answer tag in their output and compare to the ground truth to calculate the answer accuracy (Overall) as the primary evaluation metric. In ablation studies, we further analyze the format compliance (Format) and the accuracy of answers within compliant formats (Inner). Note that all models are evaluated with COT.
4.3
Quantitative Results
4.3.1
Main Results
Table 3
presents our main findings across in-domain and out-of-domain test sets. Our Point-RFT achieves state-of-the-an performance the primary ChartQA benchmark, significantly outperform-

---

Table 2: Overall results among different datasets. Comparison of different training variants in-domain (ChartQA) and out-of-domain test sets. Point-RFT achieves state-of-the-ard in-domain per- formance (90.04%) and demonstrates strongest OOD generalization (53.16% average), significantly outperforming baselines. The improvement is statistically significant with p < 0.01 under t-test.
Setting Point
In-Domain CharQA
Out-of-Domain
Method
Backhone
RL
CharXiv
PlotQA
IoonQA
TabMWP
Counting
Avg.
Base
Qwen2.5-VL-7B
70.88
26.50
17.80
53,40
61.00
21.00
35.94
Base-RFT
83.92
25.20
14.85
48.55
61.40
19.00
33.80
Point-SFI
Owen2.5-VI.77
87.21
28.00
55,25
60.30
70.00
46.78
Point-RFF RFT
90.04
36.20
20.40
59.80
70.90
78.50
53,16
Table Overall results among different models. The improvement statistically significant with
p < 0.01 under t-test.
Setting CoT
In-Domain ChartQ
Out-of-Domain
Method
Backhone
Point
CharXiv
PlocQA
IconQA
TabMWP
Counting
Avg.
Base Point-RFT
Owen2-VL-7B
/
55.40
25,20
10.50
49.15
61.00
68.00
42.77
86.16
28.30
18.80
46.90
61.60
68.50
44.82
Point-RFT
Qwen2.5- VL-3B
V
V
v
86,24
27.50
28.80
10.80
17.90
51.00
44.75
61.00
62.20
60.50
44,08
Base
Qwen2.5-VLL7B
70.88
26.50
17.80
53.40
61.00
21 21.00
35.94
Point-RFT
90.04
36.20
20.40
59.80
70.90
78.50
53.16
Question: How many bar has same value?
VL
cat has value
Therefore, the unswer
X
Yeuegh beeokoee shouthud
Point-RFT
for representig Reband volue
@ The ba repessentie ting Humgary duo has
lue of fion tommo lemorns
Sinze both finland and Hunqery have the same le
aprettating
klomenns. there
Therefore, the anower 2c
Figure 4: Cases of In-domain Chart. These cases highlight the limitations of pure text reasoning in analyzing complex visual elements. Point-RFT successfully reasons by integrating the visual content.
ing the Point- SFT model and text-only reinforcement learning variants, Base-RFT, More importantly, we observe substantial improvements in of-domain generalization.
In-Domain Performance. On ChartQA, Point-RFT achieves accuracy of 90.04%, showing significant improvement over Point-SFT (87.21%), which demonstrates the importance of RI. in helping the model learn point-based reasoning. Compared with the text-only reinforcement learning variant (83.92 (83.92%), the absolute improvement validates that visual grounding significantly enhances reasoning capability through better perception-reasoning alignment. Our method also outperforms the base instruction- tundd Qwen model (70.88%). Since the base model does not perform CoT reasoning or visual localization, Point-RFT can learn advanced visual reasoning patterns.
Out-of-Domain Generalization. Point-RFT achieves an average accuracy of 53.16% across five out- of-domain benchmarks, surpassing the text-only reinforeement learning variant by 19.36% absolute. Notably, the gains on PlotQA (+6.55%) and IconQA (+1 1.25%) suggest that explicit visual references facilitate the transfer of reasoning patterns to novel visual layouts. Furthermore, similar to the in-domain results, Point-RFT significantly outperforms Point-SFT, demonstrating the critical role of RL in acquiring this unique capability. Additionally, we observe that the OOD performance is comparable to the backbone model that undergoes large-scale multi-task instruction finetuning, further validating the generalization capability of Point-RIT.
For extra comparisons, please refer to the Supplementary Material.
4.3.2 Ablation Studies
Pointing Format. Table compares the XMI. and ISON syntax formats and the impact of point indexing schemes on performance. XML syntax achieves an accuracy of 56.72% while JSON
7

---

Table 4: Ablation studies of pointing format. Differ- Table 5: Ablation studies of point ground- ing, Removing visual grounding reduces per-
ent coordinate formats and indexing schemes highly influence performance.
formance.
Method
Format
Type
Point
Overall
ChartQA Inner
Format
Method
Point
Training SIT
RL
Overall
ChartQA
Inner
Formal
Base
79.28
82.17
96.48
Base
79.28
82.17
96.48
Base-FFT Point-SFT
v 27.60
75.82
36.40
ISON XML XML
49.36
71.88
69.56
56.72
76.20
74.44
Point-SFT
Random
31.32
75.82
39.72
Base RFT
v
77.16
79.61
96.92
56.72
76.20
74.44
Point-RFTT
99.68
Question: What the difforence percomage betweon the years 1979 and 1886?
Point-RFT
Question: Whac wikic pratu
RFT
aluved sale
revoliment rato couts 5%
reroliment rato mather 7hiin
diferercs aubtrect wikus 1986 from valae 1972 3%
The diflerence cantigg betwoon the year 1575 and 1986 3%.
Thersfore, the annwer
Questio:: chac hightt os wrfic pratt
Pokce 8FT
Question: What & the color of the bar that reprements Iel?
Point: RFT
Char
e for representing buly colord
repessening Singapore
DAY HEpteninng
colored blus.
The represeriig taly
Tharefore, the answer red.
Case of OOD Chart. Point-RFT success- fully transfers coordinate referencing skills learned from bar charts.
Figure 5:
Cases of different RL steps. As the steps increased, accurate localization led to correct answers.
Figure
achieves 49.36%. This suggests that for more complex reasoning, the XML format better aids the model in generating accurate outputs. Surprisingly, the indexing scheme (0-based vs. I-based) shows significant difference (56.72% vs. 31.32%). Interestingly, Qwen's default learned indexing starts from 1. We hypothesize that introducing appropriate variation helps the model better learn the ability to perform pointing during reasoning.
Explicit Grounding. We find that removing visual pointing (Table results in severe performance degradation: a 28.5% absolute drop in accuracy for SFT (from 56.72% 27. 27.60%) and 3.88% drop for RL (from 81.04% to 77.16%). This validates our hypothesis that explicit visual references are crucial for both initial format learning and subsequent reinforcement training. The 98.32% format compliance rate indicates that structured pointing stabilizes RL optimization, reducing hallucinated reasoning steps. This is critical for real-world applications, where predictable output formats facilitate downstream integration.
For more ablation studies, please refer to the Supplementary Material.
4.4
Qualitative Results
4.4.1
Chart Reasoning
In-domain Chart To investigate how visually grounded reasoning impacts the model's inference, we selected two groups of cases to observe the differences between our reasoning and our baseline model Base-RFT in Figure 4 We observe that our model successfully acquires the ability to perform step-by-step reasoning and locate corresponding visual elements, enabling Point-RFT to arrive at the correct answers these two high-inffommaiion-densii examples. This demonstrates the strengths of RFint-RFT, particularly its systematic exploration of visual evidence.

---

Question: at Aipoot How can' gre to Dongshorkou?
Qwen-Vi VI
Question: On April 21, weather
Qwen-
mus zarting from Housee
Apii
At Ahhaarggng Stitee Change
Since the laage only poovidss information madter for Apoll froms the saage aloce
16
Ae Deaptartie Stakeer
Point-RFT
Point-RFT
Tueeda:
10
syst pirtt Apel
Howeee
T'e dhatnna war the cardes
reproeneet by
2 Nuchice
15
queike
Guatte To Ind the vasther for apt Caurinn tanard
Folos
Trarate Gaargshuu failway lie whicis yallos
18'
Contiea: Desplaches dening
12'
Ascediag image Agti 21, party loo.
Seadle
Sunday
10
Cases of generalization. Point-RFT can perform reasoning on more complex graphics, showcasing its generalization capability.
Figure 7:
Oft-of-domain Chart Figure illustrates how CoT-based explicit grounding facilitates OOD adap- tation. When faced with chart types lacking explicit visual elements (e.g., line charts), Point-RFT successfully transfers coordinate referencing skills learned from bar charts. Despite consistent descrip- tions of visual elements across steps, the model correctly identified different locations. Meanwhile, the model was able to accurately locate bars containing special shapes and text. This suggests that explicit grounding helps abstract visual reasoning from comain-specifie details.
4.4.2 Training Steps
Figure 6 tracks the reasoning quality across different reinforcement learning iterations. In the early stages (10 steps), identified points gradually deviated from their correct positions in the chart, leading to reasoning errors. By the final model (100 steps), the correct positions were successfully located, demonstrating concise and evidence-based reasoning chains. Additionally, we observed that more steps led to more diverse but equally correct reasoning steps in textual form. This evolution underscores the critical role reinforcement learning in extracting effective grounded reasoning strategies from the exploration space.
4.4.3 Generalization
Not only can Point-RFT operate effectively in abstract charts, we observe its equal efficacy in real- world scenarios. As illustrated in Figure 7 through two representative examples: In the first case, point-grounded reasoning significantly enhances the model's 's analytical capabilities when processing complex traffic diagrams, enabling it to recognize fine-grained visual elements and perform logical reasoning. In the second example, the pointing mechanism facilitates explicit expression and reasoning about latent content, allowing accurate interpretation of scenarios that would otherwise be incomprehensible to the original model.
For further analysis on dataset, please refer to the Supplementary Material.
5 Conclusion
We have presented Point-RFT, a visually grounded Chain-of-Though framework designed to en- hance multimodal document reasoning. Our results demonstrate that structured visual grounding significantly improves accuracy and interpretability in multimodal reasoning, improving the ChartQA accuracy from 70.88% to 90.04%. Among various compared baselines, Point-RFT shows superior generalization capabilitiess across diverse, out-of- benchmarks such as CharXiv, PlotQA, and TabMWP. Additionally, provide visually grounded CoT dataset containing 7 71K samples, facili- tating easy format finetuning for future studies on multimodal reasoning. Our findings highlighted the potential disentanglement of perception errors from reasoning errors, facilitating more precise diagnostics and targeted improvements for large multimodal models.
For statements of broader impact and limitations, please refer to the Supplementary Material.

---

Acknowledgment
This work was supported by National Key R&D Program of China under Grant No. 2022YFA1004100.
References
Learning to reason with llms. htttps//opeaal.ccmmmmmmee//rrrrp tht.:///w......eeeoooooooooooeaa
2024.
Jean-Bay Ctan-Baptiste Alayrae, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12200 2023.
Bradley Brown, Jordan Juravsky, Ryan Fhrlich, Ronald Clark, Quoe Le, Christopher Ré, and Azalia Mithoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint 2rXiv:2407.21787 2024.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. Ri-v Reinforcing super generalization ability in vision language models with less than $3. released Feb 2025.
2025. Version u0.2
Xinlci Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint urXiv:1504.000 2015.
The Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou 7hu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28185-24119, 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. arXiv preprint arXiv:2409. /7146, 2024.
Yihe Deng, Hritik Bansal. Fan Yin, Nanyun Peng, Wei Wang, and Kai- Wei Chang. Openvithinker: An early exploration complex vision-language reasoning via iterative self-improvement, 2025.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.1 11499. 2023.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148 166. Springer, 2024.
Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan La, Jianwei Yang, Dan Roth, Dinci Florencio, and Cha Zhang. Refocus: Visual editing as a chain of thought for structured image understanding. arXiv preprint arXiv:2501.0544 2025.
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The lama herd of models. arXiv preprint 2rXiv:2407.2/788 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in lims via reinforcement lcarning. arXiv preprint 250. 2025.
Jarvis Guo, Tuney Zheng. Yuelin Bai, Bo Li, Yubo Wang. King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024.
10

---

Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li. Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.0544 054444444044. 2025.
Xiaowei Hu, Zhe Gan. Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up lision-language pre-training for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 17980-17989, 2022.
Yushi Hu. Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as visual chain of thought for multimodal language models. arXiv preprint arXiv:2406. 09403, 2024.
Hang Hua, Yunlong Tang, Ziyun Zeng, Liangliang Cao, Zhengyuan Yang. Hangfeng He, Chenliang Xu, and Jiebo Luo. Mmcomposition: Revisiting the compositionality of pre-trained vision- languale models. arXiv preprint arXiv:2410.09733, 2024.
Wenxuan Huang, Bohan Jia. Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models, 2025.
Aaron Hurst, Adam Leter, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Jitesh Jain, Zhengyuan Yang, Humphrey Shi, Jianfeng Gao, and Jianwei Yang. Ola-vlm: Elevating visual perception in multimodal Ilms with auxiliary embedding distillation. arXiv preprint arXiv:2412. 09585, 2024a.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezane Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation large language models for code. arXiv preprint arXiv:2403.0790 007407774. 2024b.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general purpose assistants. Foundations and Trends in Computer Graphics and Vision, 16(1-2):1 1214,,):2211, 2024.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14963-14973, 2023.
Hunter Lighitnn.. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee. Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20051 2023.
Haotian Liu. Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08455 0004000485..28 2023.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu. Xiaodan Liang, and Song- Chunn Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint 1rXiv:2110.132144 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wa Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems (NeurIPS). 2022a.
Pan Lu, Liang Qiu, Kei-Wei Chang. Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 12611 2022b.
Zixian Ma. Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, et al. Taco: Learning multi-mode action models with synthetic th-nn----tthougt-aaa anctiin arXiv preprint arXiv:2412.05479, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, arXiv preprint arx: 10244, 2022.
Fanqing Meng, Lingxiao Du Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule- based large-scale reinforeement learning arXiv preprint ar Xiv: 2533::503.07366 2025.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2020.
11

---

Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-53,, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver. and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PmLR, 2016.
OpenAT. Gpt-Av(ision) system card. 2023a.
OpenAL. Gpt-4 technical report, 2023b.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07538 2025.
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng. et al. Codeelo: Benchmarking competition-leve code generation of Ilms with human comparable elo ratings. arXiv preprint 2xXi:2500.00257, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint (xxx:2103300022: 2021.
Pooyan Rahmanzadchgervi, Logan Bolton, Mohammad Reza Tacsiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pages 18-34, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-leve google-prod q&a benchmark. In First Conference on Language Modeling, 2024.
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chimmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: bridging logical gaps with multimodal infillings, arXiv preprint or xrvi: 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:100700337, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 03300, 2024.
Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 03314, 2024.
Zhiqing Sun, Sheng Shen, Shengcao Cao. Haotian Liu, Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xionp Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Google Gemini Team. Gemini: A family of highly capable multimodal models. 2023.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37:87310
2024.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.1 14100, 2022a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- -oo-uuec-----sueeeee learning framework. In International Conference Machine Learning, pages 13-3434. PMLR, 2022b.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:231 1.03079, 2023.
Xiyao Wang, Zhengyuan Yang, Chao Feng. Hongjin Lu, Linjie Li. Chung Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts lots-guided sample selection for fata-efficient visual reasoning self- self-improeement. arXiv preprint aiv::22004.0077 07934, 2025a.
12

---

Yaoting Wang, Shengqiong Wu, Yuecheng Zhang. Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multi- modal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503. 12605,, 2025b.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022c.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances Neural Information Processing Systems, 37:1 13569 113697, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 11933, 2022.
Wenshan Wu, Shaoguang Mao, Yadong Zhang. Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind's eye of Ilms: Visualization-ot offthouttoon-t-tuuuuht elicits spatial reasoning large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Wang Xiyao, Yang Zhengyuan, Li Linjie, Lu Hongjin, Xu Yuancheng, Lin Chung-Ching Lin, Lin Kevin, Huang Furong, and Wang Lijuan. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704, 2024.
Guowei Xu, Peng Jin, Hao Li, Yibing Song. Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step- arXiv preprint arXiv:2411. 10440, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1xXi::2112.15115. 2024.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box ouiputs for grounded vision-language modeling. In European Conference on Computer Vision, pages 221-339.55 Springer, 2022a.
Zhengyuan Yang. Jianfeng Wang. Zhe Gan, Linjie Li. Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng. al. Reco: Region-controlled text-to-image generation. arXiv preprint arXiv:2211.15518.
2022b.
Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang. Chung Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.11 17421, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0m 220.00177.. 2022.
Tianyu Yu, Yuan Yao. Haoye Zhang, Taiwen He. Yifeng Han. Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 13807-13816, 2024a.
Weihao Yu, Zhengyuan Yang. Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. 2023.
Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li. Jianfeng Wang. Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408 00765, 2024b.
Lu Yuan, Dongdong Chen. Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li. Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2/11.1 1432, 2021.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. Trans. Machine Learning Research, 2024.
Yaowei Zheng, Junting Lu, Shenzhi Wang. Zhangchi Feng. Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality training framework. httpss//gittuub.com hatp://hyyoooaaaaaaa Baauga 2025.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference line-tuning, arXiv preprint arXiv:2402.11411. 2024.
13

---

A Quality of Point-CoT
To verify the quality of Point-CoT, we invited 10 volunteers and GPT-4o itself validate 1,000 randomly selected samples, calculating the proportion of high-quality samples. We define high- quality samples those that do not contain reasoning errors and have essentially correct points. As shown in Table 6 we found that the proportion of nigh-quality samples in Point-CoTl approaches 95% demonstrating the effectiveness of our Point-CoT.
Table 6: Statisties of dataset.
Evaluation
Samples
Ratio
Human Machine
1000
94.5%
1000
96.8%
B Comparison of Models
Table 7 extends the main paper's Table by comparing two additional baselines, Molmo-7B and Qwen2.5-VL--7 evaluated without the CoT prompting. Superficially, Qwen2.5-VI.-77 appears to regress when verbose CoT reasoning is enabled. This drop, however, stems primarily from the model's in-domain short-answer post-training, not from intrinsic weakness of fong-form reasoning phenomenon echoed in other RFT studies Deng et al. (2055): (aang et al. (2025a): Huang et al. (2025); Meng et al. (2025; y; |eng et al. (2025); Chen et al. (2025) Zheng et al. (2025). Nonetheless, our Point-RFT, which couples grounded point-level reasoning with Co',, consistently outperforms both well as Molmo-7B. These results confirm that integrating point-based reasoning substantially boosts multimodal reasoning performance, validating the design of both the Point-CoT dataset and the Point-RFT model.
versions of Qwen2.5-VL-7B
Table 7: Overall results among different datasets.
Setting CoT
In-Domain ChariQA
Diioff-Dmmain
Method
CharXiv
PlotQA
IconQA
TahMWP
Counting
Avg.
Owen2.5-VL-7B
87.30
34.40
22.30
59.10
56.40
15.00
39.06
Qwen2.5-VL-77B
70.88
26.50
17.80
53.40
61.00
21.00
35.94
Molmo-7B
20.48
7.60
3.95
16.50
20.70
0.00
9.75
Point-RIT
90.04
36.20
20.40
59.80
70.90
78.50
53.16
C Training Steps
As shown in Table 8 SFT requires careful balancing of training steps, reaching peak accuracy at 500 steps (56.72%). Training beyond this point leads to overfitting and performance degradation. For RL, performance monotonically improves with increasing steps, peaking 100 steps (81 104%). This indicates that extended reward shaping continually refines the reasoning patterns grounded in visual pointing.
D Training Dataset
Table that mixing SFT datasets harms performance (81. 11.04% drops to 75.20%). This suggests that targeted format learning on high-quality grounded CoT data is more effective than training on broader but noisier datasets. Notably, Point-RFT even outperforms models trained on pure ChanQA data for both SFT and RL, demonstrating its ability generalize.
E Broader Impact
present Point-RFT, multimodal reasoning framework that bridges textual and visual reasoning through visually grounded Chain-of-Thoughh (CoT). By explicitly anchoring rationales to visual

---

elements, our approach mitigates hallucinations and enhances multimodal integration, advancing the reliability of AI systems real-world document analysis. This innovation holds significant potential for human-Al collaboration. Point-RFT paves the way for trustworthy Al assistants capable of seamless multimodal reasoning across diverse domains.
Table 8: Ablation studies of training steps.
Steps
ChartQA Inner
Method
SFT
RL
Overall
Format
Base
79.28
82.17
96.48
300
50.00
71.88
69.56
500
56.72
76.20
74.44
Point-SFT
600
45.56
75.48
60.36
700
37.00
74.71
49.52
1000
20.44
75.45
25.68
500
80.92
82.14
98.52
500
10
83.36
83.66
99.64
Point-RFT
500
20
84.72
84.96
99.72
500
50
85.48
85.58
99.88
500
100
86.24
86.52
99.68
F
Limitation
Despite the promising results, our approach has several limitations. First, the two-stage training pipeline (format finetuning and reinforcement finetuning) requires substantial computational re- sources, particularly when scaling to larger datasets. Moreover, like most autoregressive models, the sequential reasoning process in Point-RFT introduces significant inference latency and resource consumption, especially for complex visual documents. These challenges highlight the need for more efficient training paradigms and inference acceleration techniques. In future work, we will explore lightweight training strategies and parallelizable decoding mechanisms to address these limitations while maintaining reasoning accuracy.
Table 9: Ablation studies of training dataset.
Dataset
ChartQA Inner
Method
SFT
RI.
Overall
Format
Base
79.28
82.
96.48
Base-SFT
ChartQA ChartQA
3.88
74.62
5.20
Base-RIF RFT
ChartQA
80.92
81.51
99.28
Point-RFT
Point-CoT (w/o ChartQA)
ChartQA
81.04
82.42
98.32
Point-RFF RFT
Point-CoT
ChartQA
86.24
86.52
99.68
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We create separate "Limitations" section in paper to discuss the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results and proposed dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We will release all code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwticccguie//eeieeeeeeii CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources needed to repro- duce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly mentioned and properly respected licenses of existing assets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced the paper well documented and is the documentation provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwpariipp..ccoooeeeeeeeeeea /t/CCooffeeencss//2222 for what should or should be described.
22


Reviews:
[{"review_id": "yYDKewJd6B", "reviewer": "Reviewer_EaVN", "summary": "This paper proposes a multimodal reasoning framework that combines with visually\ngrounded chain-of-thought (CoT) supervision. It includes two-stages: first\nconstructing a 71K-sample dataset with step-by-step visual grounding, and then\napplying GRPO-based RFT to optimize for both reasoning accuracy and format\nadherence. Experiments on different datasets demonsrate the effectiveness of the\nproposed method.", "questions": "See Weaknesses", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "#Strengths\n\n* The studied problem is interesting. The motivation of this paper is clear.\n\n* The empirical performance of the proposed framework is good.\n\n#Weaknesses\n\n* While the proposed framework shows clear engineering effort, the core\ncontribution is mainly in constructing a new dataset by distilling\nGPT-4o-generated reasoning traces with Molmo-7B-based visual grounding. The\nnovelty here lies more in implementation and data formatting rather than in new\nparadigms, and may be considered somewhat incremental.\n\n* The concept of visually grounded chain-of-thought (CoT) is not entirely new —\nprior works such as Visual CoT and Sketchpad have explored similar ideas in\ndifferent forms.\n\n* The empirical evaluation focuses almost exclusively on chart-based visual\nreasoning (e.g., ChartQA, PlotQA, CharXiv), with only a small-scale experiment\non Counting. However, multimodal reasoning is a broad area that spans various\nformats such as natural images, science documents, and embodied AI related\ntasks. Without testing the method on more diverse benchmarks on different\ndomains, it’s difficult to evalaute whether the proposed approach generalizes\nbeyond chart-based data.\n\n* All results are based solely on Qwen-VL models. There is no evaluation\non other model families (e.g., InternVL3 and LLaMA 3.2 vision) or larger models.\nIt is unclear whether the method scales to stronger models or transfers across\ndifferent VLM architectures.\n\nHu et al., Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. NeurIPS 2024\n\nShao et al., Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning. NeurIPS 2024", "was_revised": true, "final_justification": "See official comment.", "rebuttal": "Thanks for your invaluable advice!\n\n## Contribution and Novelty Beyond New Dataset\n\n**As stated in L61, our core contribution lies in being the first to propose a multimodal language model (VLM) that is outcome reward trained through visually grounded thought.** We demonstrate the advantages of visually grounded chain-of-thought (CoT) reasoning, including: \n- Better scalability compared to pure text-based reasoning in L, shown in Tables 2, 3, and 5; \n- Improved generalization ability, shown in Tables 2 and 3;\n- Better interoperability, shown in Section 4.4.3.\n\nThe datasets and engineering implementations are means to achieve these goals, but they are not the core of our aforementioned conceptual contributions.\n\nThe core finding in our paper is that, unlike in the text domain, after RL fine-tuning on specific tasks, VLM models exhibit significant in-domain performance gains but often suffer large performance drops on unseen out-of-domain (OOD) tasks, indicating poor generalization. Point-RFT demonstrates that the RL process for VLMs must effectively align reasoning with visual evidence to achieve OOD capabilities. To this end, we train the model to generate “point-and-think” trajectories via SFT and then apply RL, thereby enabling chain-of-thought generalization in RL comparable to LLMs in pure text domains. \n\nAs a result, we not only propose an effective dataset and model but also introduce a VLM RL paradigm with cross-task generalization. The real-world examples presented in Section 4.4.3 validate that our method successfully transfers in-domain reasoning ability to real-world domains, effectively addressing the generalization problem of RL in VLMs.\n\n## Difference with Visual-CoT and Sketchpad\n\nOur approach relies on **outcome incentivization** rather than **process design or supervision**. This enables broader generalization to diverse tasks, beyond problems with specific handcrafted designs. Specifically, Visual-CoT is limited to one bounding box detected in a narrow domain, and Sketchpad requires external tool, making them difficult to generalize.\n\n## Diverse Benchmarks on Different Domains\n\nIconQA and PlotQA are related to abstract images and scientific documents instead of standard chart tasks, which demonstrates generalization to document-like content and abstract visual inputs.\n\nWe additionally introduced the MMVet dataset as a benchmark, as it is a standardized task suite covering a wide range of multimodal reasoning tasks, further validating our method’s generalization to natural images.\n\n\n|              | Model                | Rec      | Ocr      | Know     | Gen        | Spat     | Math      | Overall   |\n|--------------|----------------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | Qwen2.5-VL-7B        | 34.4     | 49.2     | 14.9     | 12.1       | 52.3     | 59.2      | 40.0      |\n| Point-RFT    | Qwen2.5-VL-7B        | **37.1** | **61.5** | **17.6** | **14.0** | **65.3** | **73.1** | **47.6** |\n\nThe results show that our method significantly improves performance across various scenarios, strongly supporting our findings. Although we are unable to include images, we observed that pointing-based thought generalizes natural images well. We will include more examples in the revision.\n\n## Evaluation on Other Model Families\n\nWe choose Qwen2.5-VL as it is a widely used practice for VLM learning. To further validate the effectiveness of our method, we also conducted experiments on LLaMA 3.2 Vision, which also has different model scales.\n|              | Model                | ChartQA  | CharXiv  | PlotQA   | IconQA     | TabMWP   | Counting  | OOD Avg   |\n|--------------|----------------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | LLaMA-3.2-Vision-11B | 42.1     | 23.1     | 8.5      | 30.0       | 55.0     | 6.0       | 26.5      |\n| Point-RFT    | LLaMA-3.2-Vision-11B | **86.2** | **26.5** | **14.4** | **49.0** | **58.4** | **45.0** | **38.7** |\n\nThe results demonstrate that our method adapts well to different architectures."}, {"review_id": "MFUf6h4NXt", "reviewer": "Reviewer_TsjN", "summary": "This paper proposes Point-RFT, a novel post-training method designed to enhance multimodal reasoning performance. Point-RFT first constructs the Point-CoT dataset using a general-purpose reasoning model, such as GPT-4o, and a grounding expert model, like Molmo-7B. The Point-CoT dataset consists of reasoning trajectories that include coordinates within images and explanations of those regions. Then, a target model is supervised fine-tuned on the constructed dataset. Finally, the SFT-tuned model is further improved through reinforcement learning. The experiments demonstrate that Point-RFT is effective for document understanding tasks, and the results reveal that the point grounding technique is the primary factor behind this effectiveness.", "questions": "1. Why does the format training lead to such improvements?\n2. How does the proposed method perform when applied to standard VQA tasks? If it shows limited effectiveness or results in performance degradation, what are the underlying reasons?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strrengths**\n1. This paper is clear and well-organized overall.\n2. The proposed idea is novel and closely mirrors the way humans reason.\n3. The experiments demonstrate that the proposed method is effective on in-domain and out-of-domain document understanding tasks.\n\n**Weaknesses**\n1. Table 1 shows that Point-SFT already outperforms the Baseline and Base-RFT by a large margin. A deeper analysis of why the format training leads to such improvements would further strengthen the paper’s contribution.\n2. How does the proposed method perform when applied to standard VQA tasks? If it shows limited effectiveness or results in performance degradation, what are the underlying reasons?", "was_revised": true, "final_justification": "While I still find this paper somewhat limited in advancing our understanding of the underlying mechanisms of MLLMs, the motivation is sound, and the proposed method demonstrates meaningful improvements. The additional results provided in the response are interesting, though I believe both the number of samples and the range of VQA benchmarks should be expanded to strengthen the findings. Overall, I believe the paper meets the acceptance bar for NeurIPS, and I will keep my score as Accept.", "rebuttal": "Thank you for your valuable feedback and suggestions!\n\n## Improvements by Format Training\n\nThis is because even in out-of-domain scenarios, using Point-SFT alone can significantly reduce perception and alignment errors, thereby lowering the overall error rate. However, standalone SFT remains constrained by data quality. In contrast, our Point-RFT method based on outcome reward modeling further enhances the quality of visual grounding, leading to improved performance.\n\nTo further explore why format training leads to such improvements, we conducted additional experiments where volunteers manually categorized errors on 200 samples that were predicted incorrectly by the base model.\n\nWe simply categorized the errors into four types: perception errors, reasoning errors, alignment errors, and other errors. Perception errors represent mistakes in visual perception, such as failing to correctly identify objects or details in an image. Reasoning errors refer to mistakes in logical reasoning or understanding of the text. Alignment errors indicate incorrect referencing of visual information that has been correctly perceived during the reasoning process. Other errors encompass various other types of mistakes.\n\n\n| Error Type               | Base                          | Point-SFT                         |\n|--------------------------|-------------------------------|-----------------------------------|\n| Perception               | 73                            | 29                                |\n| Reasoning                | 61                            | 59                                |\n| Alignment                | 44                            | 13                                |\n| Other                    | 22                            | 20                                |\n\nWe found that through format training, the model generates \"point-and-think\" trajectories during reasoning, significantly improving the alignment between visual information and text reasoning. This leads to a substantial reduction in perception errors and alignment errors, thereby decreasing the overall error rate. This also explains why base-RFT without format training is ineffective.\n\n## Results on Standard VQA Tasks\n\nWe additionally introduced the MMVet dataset as a benchmark, as it is a standardized task suite covering a wide range of multimodal reasoning tasks, further validating our method’s generalization to natural images.\n\n|              | Model                | Rec      | Ocr      | Know     | Gen        | Spat     | Math      | Overall   |\n|--------------|----------------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | Qwen2.5-VL-7B        | 34.4     | 49.2     | 14.9     | 12.1       | 52.3     | 59.2      | 40.0      |\n| Point-RFT    | Qwen2.5-VL-7B        | **37.1** | **61.5** | **17.6** | **14.0** | **65.3** | **73.1** | **47.6** |\n\nThe results show that our method significantly improves performance across various scenarios, strongly supporting our findings. Although we are unable to include images, we observed that pointing-based thought generalizes natural images well. We will include more examples in the revision."}, {"review_id": "sGCCa418bj", "reviewer": "Reviewer_dht9", "summary": "This paper proposes Point-RFT, a reinforcement fine-tuning framework that enhances multimodal reasoning by integrating visual grounding into the chain-of-thought (CoT) process. The method proceeds in two stages: first, a supervised format fine-tuning stage using a newly curated dataset (Point-CoT) with step-by-step reasoning grounded to visual elements; second, a reinforcement fine-tuning stage using GRPO to optimize both answer correctness and reasoning format alignment. The paper also introduces a dataset, Point-CoT, a novel multimodal CoT dataset. Experimental results show that Point-RFT significantly improves in-domain accuracy on ChartQA and generalizes well to multiple out-of-domain visual reasoning benchmarks.", "questions": "- The paper shows strong generalization on visual document understanding tasks, but it remains unclear whether the proposed Point-RFT framework can extend to broader vision-language reasoning domains. Could the authors evaluate the method on more general VQA-style benchmarks such as MMBench, MMStar, MMVet, or MathVista, which feature diverse visual contexts and question types? A positive result on such benchmarks would strengthen the claim of generalizability and showcase the method's applicability beyond charts and plots.\n\n- In Table 3 (Out-of-Domain), the base Qwen2-VL-7B model achieves a higher average performance than the Qwen2.5-VL series, yet the performance gain after applying Point-RFT appears smaller. Could the authors clarify why the improvement margin differs so much?\n\n- As shown in Table 1, the Point-CoT dataset already includes samples from PlotQA. However, in Table 2 and Table 3, PlotQA is listed as an out-of-domain (OOD) test set. Could the authors clarify this inconsistency? \n\n- The paper includes ablations on format syntax and grounding, but does not isolate the effects of GRPO vs. more standard RL approaches like PPO. Would the benefits of Point-RFT still hold with simpler optimization strategies?\n\nIf your response can address my concerns, I will consider increasing my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper focuses on an interesting and meaningful problem. It presents a two-stage training pipeline combining supervised fine-tuning with reinforcement fine-tuning using GRPO. The empirical evaluation is thorough, covering both in-domain (ChartQA) and diverse out-of-domain datasets (e.g., PlotQA, IconQA), with consistent performance gains. The paper is clearly written, well-structured and easy to follow.\n\n**Weakness**\n\nWhile the use of GRPO in Point-RFT is effective, the paper does not clearly articulate what new insights or innovations it brings beyond prior applications of GRPO in other multimodal or reasoning tasks. A clearer justification of the novelty and insights would strengthen the contribution, and this is my major concern.\n\nThe paper shows strong results on visual document understanding benchmarks, but it remains unclear whether Point-RFT can generalize to broader vision-language tasks, such as those involving non-document images. This limitation weakens the overall persuasiveness of the claimed generalization ability of Point-RFT.\n\nOther weaknesses can be found in the Questions section.", "was_revised": true, "final_justification": "The authors’ rebuttal has largely resolved my concerns, and I will increase my score by one point.", "rebuttal": "We appreciate your constructive advice!\n\n## New Insights or Innovations\n\n**As stated in L61, our core contribution lies in being the first to propose a multimodal language model (VLM) that is outcome reward trained through visually grounded thought.** We demonstrate the advantages of visually grounded chain-of-thought (CoT) reasoning, including: \n- Better scalability compared to pure text-based reasoning in L, shown in Tables 2, 3, and 5; \n- Improved generalization ability, shown in Tables 2 and 3;\n- Better interoperability, shown in Section 4.4.3.\n\nThe datasets and engineering implementations are means to achieve these goals, but they are not the core of our aforementioned conceptual contributions.\n\nThe core finding in our paper is that, unlike in the text domain, after RL fine-tuning on specific tasks, VLM models exhibit significant in-domain performance gains but often suffer large performance drops on unseen out-of-domain (OOD) tasks, indicating poor generalization. Point-RFT demonstrates that the RL process for VLMs must effectively align reasoning with visual evidence to achieve OOD capabilities. To this end, we train the model to generate “point-and-think” trajectories via SFT and then apply RL, thereby enabling chain-of-thought generalization in RL comparable to LLMs in pure text domains. \n\nAs a result, we not only propose an effective dataset and model but also introduce a VLM RL paradigm with cross-task generalization. The real-world examples presented in Section 4.4.3 validate that our method successfully transfers in-domain reasoning ability to real-world domains, effectively addressing the generalization problem of RL in VLMs.\n\n## Additional Benchmarks on Different Domains\n\nIconQA and PlotQA are related to abstract images and scientific documents instead of standard chart tasks, which demonstrates generalization to document-like content and abstract visual inputs.\n\nWe additionally introduced the MMVet dataset as a benchmark, as it is a standardized task suite covering a wide range of multimodal reasoning tasks, further validating our method’s generalization to natural images.\n\n|              | Model                | Rec      | Ocr      | Know     | Gen        | Spat     | Math      | Overall   |\n|--------------|----------------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | Qwen2.5-VL-7B        | 34.4     | 49.2     | 14.9     | 12.1       | 52.3     | 59.2      | 40.0      |\n| Point-RFT    | Qwen2.5-VL-7B        | **37.1** | **61.5** | **17.6** | **14.0** | **65.3** | **73.1** | **47.6** |\n\nThe results show that our method significantly improves performance across various scenarios, strongly supporting our findings. Although we are unable to include images, we observed that pointing-based thought generalizes natural images well. We will include more examples in the revision.\n\n## Clarification on Improvement Margins\n\nThe difference in average scores between the two models (42.77 vs. 35.94) is primarily driven by the large performance gap on the counting task (68 vs. 21). The average scores on the other datasets are (36.46 to 38.90) and (39.68 to 46.80), respectively, and show consistent performance.\n\nSo, why does the counting task exhibit such an outlier value? This is because Qwen2-VL-7B tends to give direct and correct answers, while Qwen2.5-VL often generates chain-of-thought (CoT) responses that introduce hallucinations. Since the counting task is highly sensitive to hallucinations, Qwen2.5-VL performs worse. However, after applying Point-RFT, Qwen2.5-VL-7B shows reduced hallucinations and its stronger reasoning abilities become more effective, resulting in greater performance gains.\n\nThis also demonstrates the advantage of our Point-RFT: through grounded CoT, our method reduces hallucinations and improves reasoning ability during RL.\n\n## Clarification on OOD Evaluation\n\nSorry for the confusion. Some of the evaluation data categorized as out-of-domain (OOD) also appear in the SFT data. This is because our definition of OOD is based on whether the data is used in the RL phase. Since directly applying reinforcement learning to vision-language models often leads to performance degradation on out-of-domain tasks, using RL data as the criterion allows us to better assess whether our method maintains robust performance beyond the scope of RL-trained data (e.g., ChartQA). We include a broader mix of data types in the SFT stage to train a more generalizable capability, thereby enhancing the model’s usability in real-world scenarios. We will clarify this more explicitly in the revised version.\n\n## Evaluation on Other RL Algorithms\n\nWe have tried the PPO algorithm on Qwen2.5-VL-7B with Point-RFT.\n\n|              | Model                | RL Algorithm | ChartQA  | CharXiv  | PlotQA   | IconQA     | TabMWP   | Counting  | OOD Avg   |\n|--------------|----------------------|--------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | Qwen2.5-VL-7B        |              | 70.9     | 26.5     | 17.8     | 53.4       | 61.0     | 21.0      | 35.9      |\n| Point-RFT    | Qwen2.5-VL-7B        | PPO          | **88.3** | **34.9** | **19.0** | **56.7** | **65.1** | **74.5** | **50.0** |\n| Point-RFT    | Qwen2.5-VL-7B        | GRPO         | **90.0** | **36.2** | **20.4** | **59.8** | **70.9** | **78.5** | **53.2** |\n\n\nWe can find that PPO obtained similar results, showing that the key factor in achieving strong model performance is not the specific RL algorithm itself, but rather our design of the “point-and-think” trajectory and the corresponding outcome reward."}, {"review_id": "eIXVuA4nsZ", "reviewer": "Reviewer_TioY", "summary": "This paper presents Point-RFT, a two-stage framework for enhancing multimodal large language models (MLLMs) with visually grounded chain-of-thought (CoT) reasoning for visual document understanding. In the first stage, the model is trained to generate visually grounded CoT rationales explicitly aligned with visual elements via point annotations. In the second stage, the model undergoes reinforcement finetuning with Group-wise Relative Policy Optimization (GRPO) to improve answer correctness and generalization. This paper introduces a large, cross-validated Point-CoT dataset and demonstrates substantial gains on ChartQA (in-domain) and several out-of-domain benchmarks, reporting improved accuracy and interpretability over strong baselines.", "questions": "Can Point-RFT be applied to or benefit those general image QA understanding tasks, such as Seed-bench2 [1]?\n\n[1] Li, Bohao, et al. \"Seed-bench: Benchmarking multimodal large language models.\" CVPR 2024.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. The idea of extending text-only CoT to visual-grounded CoT is novel and well-motivated.\n2. The pipeline for constructing the Point-CoT with cross-model validation mechanism reduces errors caused from unimodal.\n3. The analysis of each component in Robot-RFT is comprehensive.\n\n[Weaknesses]\n1. **Comparison with Prior SOTA**: The discussion and quantitative comparison with other recent methods for visual document understanding is limited.\n2. **Dataset Overlap/Out-of-Domain Clarification**: There is confusion about what is considered out-of-domain. For instance, PlotQA is listed as both a source for training data (Tab. 1) and as out-of-domain in the evaluation tables (Tab. 2 and Tab. 3) .", "was_revised": true, "final_justification": "The raised concerns are solved by the authors during rebuttal. Thus, I maintain my positive rating.", "rebuttal": "Thank you for your valuable advice and comments!\n\n## Comparison with Prior SOTA\n\nWe have added comparisons with LLaMA Vision 3.2 and Molmo. Specifically, we also trained our Point-RFT using LLaMA Vision 3.2 as the backbone.\n\n|              | Model                | ChartQA  | CharXiv  | PlotQA   | IconQA     | TabMWP   | Counting  | OOD Avg   |\n|--------------|----------------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | Molmo-7B             | 20.5     | 7.6      | 4.0      | 16.5       | 20.7     | 0.0       | 9.8       |\n| Base         | LLaMA-3.2-Vision-11B | 42.1     | 23.1     | 8.5      | 30.0       | 55.0     | 6.0       | 26.5      |\n| Base         | Qwen2.5-VL-7B        | 70.9     | 26.5     | 17.8     | 53.4       | 61.0     | 21.0      | 35.9      |\n| Base-RFT     | Qwen2.5-VL-7B        | 83.9     | 25.2     | 14.9     | 48.6       | 61.4     | 19.0      | 33.8      |\n| Point-SFT    | Qwen2.5-VL-7B        | 87.2     | 28.0     | 20.3     | 55.3       | 60.3     | 70.0      | 46.8      |\n| Point-RFT    | LLaMA-3.2-Vision-11B | **86.2** | **26.5** | **14.4** | **49.0** | **58.4** | **45.0** | **38.7** |\n| Point-RFT    | Qwen2.5-VL-7B        | **90.0** | **36.2** | **20.4** | **59.8** | **70.9** | **78.5** | **53.2** |\n\nThe experiments demonstrate the effectiveness of our method across different models. In the revised version, we will add more model comparisons and qualitative discussions.\n\n## General Image QA Understanding Tasks\n\nWe additionally introduced the MMVet dataset as a benchmark, as it is a standardized task suite covering a wide range of multimodal reasoning tasks, further validating our method’s generalization to general images.\n\n|              | Model                | Rec      | Ocr      | Know     | Gen        | Spat     | Math      | Overall   |\n|--------------|----------------------|----------|----------|----------|------------|----------|-----------|-----------|\n| Base         | Qwen2.5-VL-7B        | 34.4     | 49.2     | 14.9     | 12.1       | 52.3     | 59.2      | 40.0      |\n| Point-RFT    | Qwen2.5-VL-7B        | **37.1** | **61.5** | **17.6** | **14.0** | **65.3** | **73.1** | **47.6** |\n\nThe results show that our method significantly improves performance across various scenarios, strongly supporting our findings. Although we are unable to include images, we observed that pointing-based thought generalizes natural images well. We will include more examples in the revision.\n\n## Out-of-Domain Clarification\n\nSorry for the confusion. Some of the evaluation data categorized as out-of-domain (OOD) also appear in the SFT data. This is because our definition of OOD is based on whether the data is used in the RL phase. Since directly applying reinforcement learning to vision-language models often leads to performance degradation on out-of-domain tasks, using RL data as the criterion allows us to better assess whether our method maintains robust performance beyond the scope of RL-trained data (e.g., ChartQA). We include a broader mix of data types in the SFT stage to train a more generalizable capability, thereby enhancing the model’s usability in real-world scenarios. We will clarify this more explicitly in the revised version."}]

Author rebuttal:

Meta-review: This paper proposes a post-training enhancement "Point-RFT" for multimodal reasoning and a dataset "Point-CoT" that contains reasoning trajectories with image coordinates and explanations.  The proposed approach is fine-tuned on this dataset.  Experiments show improvements in chart understanding and other visual reasoning benchmarks.

The reviews were leaning positive after the rebuttal and identified strengths of the work such as sound motivation, thorough evaluation (including the results provided during the rebuttal) and analysis.  While major reviewer concerns were addressed, some concerns remain such as expanding the number of samples and the range of VQA benchmarks and OOD VQA benchmarks and different baselines/backbones -- these should be incorporated.
Decision: Accept (poster)