Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
Authors: Honghao Chen
Xingzhou Lou
Xiaokun Feng
Kaiqi Huang
Xinlong Wang
Venue: NeurIPS 2025 poster ()
Keywords: VLM
Reasoning
PRM
Primary area: deep_learning

Abstract: Chain of thought reasoning has demonstrated remarkable success in large language models, yet its adaptation to vision-language reasoning remains an open challenge with unclear best practices. Existing attempts typically employ reasoning chains at a coarse-grained level, which struggles to perform fine-grained structured reasoning and, more importantly, are difficult to evaluate the reward and quality of intermediate reasoning. In this work, we delve into chain of step reasoning for vision-language models, enabling assessing reasoning step quality accurately and leading to effective reinforcement learning and inference-time scaling with fine-grained rewards. We present a simple, effective, and fully transparent framework, including the step-level reasoning data, process reward model (PRM), and reinforcement learning training. With the proposed approaches, our models set strong baselines with consistent improvements on challenging vision-language benchmarks. More importantly, we conduct a thorough empirical analysis and ablation study, unveiling the impact of each component and several intriguing properties of inference-time scaling. We believe this paper serves as a baseline for vision-language models and offers insights into more complex multimodal reasoning. Our dataset, PRM, and code at https://github.com/baaivision/CoS.

Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
Honghao Chen 12233 Xingzhou Lou Xiaokun Fengl.2+ Kaiqi Huang 1.2f Xinlong Wang
"Istitute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University Chinese Academy of Sciences Beijing Academy of Artificial Intelligence
Abstract
Chain of thought reasoning has demonstrated remarkable success in large language models, yet its adaptation to vision- saaionla reasoning remains an open challenge with unelear best practices. Existing attempts typically employ reasoning chains at a coarse coarse-grained level, which struggles to perform fine-grained structured rea- soning and, more importantly, intermediate reasoning. In this work, we delve into chain of step reasoning for laigunn models, enabling assessing reasoning step quality accurately and leading to effective reinforcement learning and inference-time scaling with fine- grained rewards. We present simple, effective, and fully transparent framework, including the step-level reasoning data, process reward model (PRM), and rein- forcement learning training. With the proposed approaches, our models set strong baselines with consistent improvements on challenging Aision-language bench- marks. More importantly, we conduct thorough empirical analysis and ablation study, unveiling the impact of each component and several intriguing properties of inference-time scaling. We believe this paper serves as a baseline for vision- language models and offers insights into more complex multimodal reasoning. Our dataset, PRM, and code
difficult to evaluate the reward and quality of
wwcccluu..ooooeeeo httpbaaavvsioo/CSSS
Introduction
The complex reasoning capabilities of large language models (LLMs) have recently garnered widespread attention (57
47 []. By generating detailed chain of thought (CoT) process before producing the final answer. LLMS enable effective test-time scaling and significantly enhancing the reasoning and logical capabilities. The integration of CoT reasoning with large-scale reinforcement learning 44 has further led to revolutionary advancements, exemplified by groundbreaking models such as OpenAl-ol [17] and DeepSeck-RL [12]. Inspired by the success of NLP, many studies focus on exploring complex reasoning methods in vision- languuge models (VLMs), achieving various advances 66
29 Compared to L.LMs, the presence of visual information in VLMs introduces greater complexity and plays more critical role, emphasizing the necessity of leveraging visual cues for precise step-by-step reasoning and problem-solving.
58
10 55
Despite the advances, enabling VLMs to perform human-level reasoning remains key challenge 661 fundamental difference is that humans typically engage in systematic and structured thinking (48 whereas the CoT outputs of current VLMs are constrained to coarse-grained reasoning, making it difficult to execute systematic and structured inference [58]. Specifically, current VLMs typically output lengthy thought without unified structure or clear step-by-step progression. This form
Equual contribution. Work done during Honghao's internship at BAAL t Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of reasoning tends to become disorganized or verbose and, more importanty, makes it difficult to evaluate the quality of reasoning. As a result, selecting high-quality reasoning trajectories for efficient training (e.g., reinforcement learning) or inference becomes challenging. It is natural to ask whether step-by-step fine-grained reasoning can facilitate better training or inference. However, answering this is non-trivial, as it involves two major challenges: i) defining step, i.e., how to decompose the overall reasoning chain into logically coherent and progressive steps; ii) evaluating step, i.c. how to provide fine-grained reward signals for step-wise reasoning to facilitate training and inference.
To address these challenges and answer the above question, we propose Chain-of- Step reasoning and introduce corresponding supervised fine-tuning (SFT) and reinforcement learning strategies to enhance VLMs' reasoning capabilities. We start by defining step: the entire reasoning chain is decomposed into consecutive steps, with each step consisting of three components: Name, Thought, and Reflection, separated by special tags. The Name component provides a concise summary of the step, the Thought component contains detailed reasoning specific to that step, and the Reflection component establishes connections between the current reasoning step, the visual information, and previous steps. Our design boasts three essential advantages: i) structuring reasoning into discrete steps to ensure clarity; ii) maintaining a concise and well-organized structure; iil) incorporating visual content along with reflective reasoning. We use GPT-4o [16] to construct ShareGPT-Step-300 dataset of 300K structured step-wise reasoning samples following the designed template. Our dataset encompasses diverse range of tasks, utilizing 17 datasets that demand various reasoning skills, such as scientific reasoning [8 31, mathematical reasoning [5] 64] and world knowledge [41.
Through SFT on ShareGPT-Stee-3000 VLMS can output step-by-step reasoning chains. However, reinforcement learning more crucial for enhancing the model's reasoning capabilities [66]. and providing accurate rewards is of vital importance [34. Therefore, we delve into the precise evaluation of reasoning steps. We collect process-annotated data using two mainstream methods: Monte Carlo estimation and LLM-as-Judge [67]. Specifically, we employ Math-Shepherd [52) and GPT-4o [66] to annotate the step-wise reasoning trajectories, which we then use to train a process reward model (PRM). Based on PRM, we can assess the quality of intermediate reasoning steps with high accuracy, thereby providing fine- ine-grained rewards.
Building upon the above framework, we can effectively choose reasoning paths to conduct iter- ative Direct Preference Optimization (DPO) 7 and inference-time scaling. Experiments across multiple benchmarks demonstrate that our approach significantly enhances reasoning capabilities, outperforming strong baselines with consistent improvements. Our empirical analysis and ablations indicate that incorporating step-level evaluation is more reasonable and effective than relying solely on answer-level evaluation. Interestingly, we find that longer reasoning is not necessarily better for VLMs, and reasoning steps' quality is much more important than reasoning length. Different from prior arts, our work highlights the importance of introducing fine-grained reasoning chains in complex multimodal reasoning and leveraging fine- gme-inrieed reward- based reinforcement learning to enhance VLMs. We hope our findings provide new insights for the community.
2 Related Work
VLM Reasoning, As the application of VLMs in complex tasks such as mathematics 30 51 and
grows, the ability to reason through complex problems becomes increasingly important. For instance, in solving mathematical diagram problems, Mavis (65] constructs corre- sponding chain-of-thought (COT) (57 reasoning dataset and optimized the model through instruction fine-tuning 40). Visual-CoT
science [61 T8
42 focuses on enhancing the model's ''s ability to correspond visual questions with key areas using CoT reasoning. Insight-V V [10] designs a multi-agent system consisting of reasoning agent dedicated to performing long-chain reasoning and a summary agent trained to judge and summarize reasoning results. The success of CoT in improving VLMs' complex visual reasoning reveals the importance of enabling the model to analyze intermediate reasoning steps in detail [66]. Unlike most existing methods [21 42 9 that use reasoning chains at a coarse-graie graiiee level, our work introduces a grained structured reasoning approach, offering greater potential for enhancing the reasoning capabilities of VLMs.
Inference-Time Scaling. Inference time scaling 45 refers to the process wherein model utilizes additional reasoning time to solve problems more effectively. This System-2 style thinking ability 39 is recognized as an effective way to enhance the complex reasoning capabilities of LLMs and

---

Traditional inference time scaling methods include majority voting [14), best-N search 56 and sentence-level beam search [49]. Despite their widespread application, majority voting is typically only for problems with standard answers, whereas best-N search and sentence-level beam search present challenges terms of accuracy assessment
has recently garnered significant attention 69
12
and the quality of individual responses [58]. Recently, OpenAl has developed verifiers to supervise and select reasoning paths during the inference process [25]. while Math-Sheppp Shehh-S [52] evaluates intermediate reasoning steps based their likelihood of leading correct answers. Inspired by the
success of these LLMs, VLaVA-CoT [58] and URSA 34] have started exploring the inference time scaling capabilities of VLMs. Procedural reasoning datasets have been constructed and used for the supervised fine-tun tie-ttning of the models. Different from those prior arts, we utilize PRM to provide granular rewards and facilitate step-level beam search to enhance the inference time scaling of VLMs.
Method
In this section, we present the pipeline of
Base Model
SET Model
CoS illustrated in Fig. | In Section 3.1
we introduce the design of our structured reasoning template. In Section 3.2
ShareGPT- Step-300K
we de-
scribe the composition of the SFT dataset ShareGPT-Ster Stok-300K. Then explain the acquisition process annotation data in Sec- tion tion3.3 3.3 Last, in Section [34] elaborate the details of the RI. training pipeline with fine-grained rewards.
Stage Supervised Fine- line-Tuning
PRM
Dataset
Stage 2: PRM Training (Optional)
3.1 Structured Reasoning
We structure the entire thought into multi- ple consecutive cognitive steps, ensuring that these steps logically flow natural and co- herent manner towards the final answer. Each step consists of the following three compo- nents:
Sampled Reasoning
Score
)
Filter
Step-Level
Preference Data
Stage Iterative Preference Learning with PRM
Name. summary and overview of the current step, providing high-level expla- nation. This outlines the key aspects the step to be executed.
Figure 1: Overview of our framework. We con- duct SFT to enable the model generate step-by-step reasoning paths. Then we train a PRM to provide fine-grained RL reward, and enhance the complex rea- soning ability through step-level preference learning. Note that PRM only needs to trained once for dif- ferent models.
Thought. detailed description of the thought process in this step, explaining what is being observed, inferred, or an- alyzed.
Reflection. Establishing connections be- tween the visual content and previous steps. Since VLMs are prone to generating content that contradicts the image, the reflection introduced to create links with the visual content and prior steps, thus alleviating hallucination.
We use special tokens to establish this reasoning format, rather than simply relying on additional prompts external prompt engineering, because depending prompt-based output format is unstable and would require more data filtering and cleaning. In contrast, ensuring stable output of this format is more critical for splitting and evaluating steps.
CoS begins with the special token |ressoning Laaring_sttrrttt >. We take reasoning step as the basic unit for structured and fine- tine-griieed reasoning. Steps are connected by < The reasoning stops after outputting the token < [reasoning_ent
proceed| >. >, then the model derives final answer from previous reasoning. The model generates reasoning steps and answers in an auto- regressive manner, where the number of reasoning steps and their specific content are self-generated without any restrictions. This structured output and its length autonomously adapted and adjusted
3

---

Heatr yuew FTL -...
Reusening alopee
Shyp 4ifettonntrr the bengih afthee
Thargh lang (-33) 1 I-S
< <Treggi Wi shd by wrggg the we les non your Ril, (1.40 wil
O andit:
4i thatggae detemnad
dested
form inavtiin passrattroed PCL
spuatanee wellie y-
with dipe 2 spaudie
Scoaeatae the from
willey
Thengic seottte
the verioal dira fre P1L4)) theceal Saca divesly soone tas coss the bagint tu daapy daig
ciaflcttinn io This dap une up the dustaato deg lell docse
ecuakios.
cotep 2 Suuplit Lake Squaaiane
cadaeciaan beaalikku taclogge cingh are clialiio an disatly
hart
dis s0e
Slouali the
Quostion
una cea aaoaiy 22- -i
Step de Coleniiiee the lnce atThtangte dron
cieellction
by or glonp un colttaaton
parco whae e lnso intervirr ent crade for cs baveaed
te uuetttu beain andtergg gra c. -n 332 a de
cotie the doad
Seep 3: Thd Interecciie with the sadee
Step Vilakdae the Consinency
dech
earee the cacita cludee saage.
las y 2i sey Quni i-2 pous aa o0
Ql leggitrrus F-6 leuan 1/2 xi pin.. lqpiaatyy
catir
Answvor
svvezulle oearr uy vrang caloaas to deseusaas.
cwnest
Aecoonning algase
Saep Scädianca Tareard the Sardayg Regin
Sopp Mesall Starting Polats
avoutiga
Thought wi len for thar thrre mork wallor on is welledy
peniie patk allare
Rotfoction Th the eite cor the pridkes lrunttyy sorn
1n inger hvrr to
Soe 2 Anabyoe the Pick and Dorsners
Threghh - 1-l gite murees flooe prsecn firm in tnsitii or
dianee tha to Cnl dennir
Belleciion Arrono tralowe cari to Boorslite snotur reverrddaaa conritattr
sperh
dn fice 3.
Step 75Tand Dotanes calcalatios
Sm on seepthendd pert od deckedt fibra Whet thanocepeeddg
Questio:
Botbetteen The
Threght enbler engf the leweeds fuo and aaind for boreew
the we
Pinde 2. V) to ii two villpple do to ip 2h 16
dhyy
Step Delorreee
Sopp Parther Progres
Strtksgfee 2. ifee
mave in the core ii
Thought voleer it for contrnee to The patk hove aibrince
Wy suuingg anditrall wrnm han haral bi to d Che thess
Alswer:
Betfectee tilla dreckes Eorropends vet with the tragecheatter
Beflectann whharapecneeee dirame partes te prehirmmeeceeeeee mecoremmann 20
precie
Figure 2: Qualitative examples. We depict the reasoning process on cases that need to understand both visual content and mathematics. Our CoS has accurate understandings of both aspects, and more importantly, combine them to derive multimodal reasoning. Note that special tokens are omitted.
by the model, thereby enhancing its ability to perform complex reasoning. We depict two qualitative examples in Fig. and the details of special tokens can be found in Appendix A
3.2 ShareGPT-Step-300K
Given that most VQA, multi-modal math, and reasoning datasets only provide short answer an- notations, we design method to leverage these short annotations to generate long, step-by-step reasoning data. Specifically, we provide the question and its ground-truth answers as references, and prompts GPT-4o to generate step-by-step reasoning. This can be seen as reasoning from the results. The reference answer can greatly reduce the difficulty of the question, making it easier to generate high-quality thinking steps.
To ensure diversity, our data is collected from various multimodal datasets, encompassing 17 tasks that require diverse multimodal reasoning capabilities. These tasks can be categorized into four major types including mathematical reasoning [51 64 3 26 24 scientific reasoning [18, 23 5). chart&do analysis [35 32 37 36), and world knowledge
summary of the
dataset sources can be found in Appendix(D [
After generating the data, we conducted meticulous data cleaning and filtering remove a large amount of data that did not meet the required formatting standards. Ultimately, we obtained 300K high-quality step-level data instances, which we named as ShareGPT-Step-3000 We plan release this dataset to advance research in the community on fine-grained step-level reasoning.
3.3 Step-Level Annotation
Currently, mainstream methods for evaluating reasoning steps can be categorized into three types [67] i) Human, which relies on human annotators to produce high-quality data but incurs substantial costs; ii) Monte-Carla (MC) Estimation, which infers the correctness of steps from the outcomes of vast number of sampled solutions; iii) LLM-as-Judge, which leverages LLM as critic by directly prompting the LLM to evaluate the correctness of steps. In this work, for the sake of efficiency and scalability, we employ MC Estimation and LLM-as-Judhh
annotate leee- data.
For MC Estimation, we use math-shepherd problem with golden answer and a step-by-step solution, to annotate the correctness of a specific step, we utilize a fine-tuned VL.M to decode N subsequent reasoning paths from this step. We further validate whether the decoded final answer matches with the golden answer. If a reasoning step can
52 to annotate steps automatically. Specifically, given a

---

deduce more correct answers than another, it would be assigned higher correctness score. Here we use N=16 by default.
For LLM--a-Judge, we utilize GPT-4o [6] to perform the annotations. Fach step is evaluated with one of three possible ratings: Good: the step is logical, directly relevant the question, and most importantly correct; ii) Neutral: the step is relevant but contains minor flaws or gaps; iii) Bad: the step flawed, wrong, or shows misunderstanding of the question or context. For solutions where the final answer is correct, both Good and Neutral ratings lead to a correct label. Conversely, for solutions with incorrect final answer. only steps annotated as Good are ultimately marked as correct. The prompt is provided in Appendix B
We both math aath-shepherd and GPT-4o to generate 100K step-annotater data, respectively. These step-annotated data are then utilized to train a PRM, which can provide fine-grained reward signals for reinforcement learning and inference-time scaling.
3.4 RL with Fine-grained Rewards
As illustrated in Fig. our training pipeline is designed to be straightforward and efficient. Starting from well- cell-pretrained VLM, we first perform supervised fine-tuning on the ChareGPT-Step-300 dataset to enable the model generate step-by-step reasoning paths. Subsequently, we train a PRM to provide fine- grained rewards. Leveraging the PRM, we efficiently select reasoning paths for iterative Direct Preference Optimization, thereby further enhancing the model's reasoning capabilities.
Process reward model training. In RLHF, the reward models are typically trained with binary preferences [2) In reasoning tasks where step-annotation of solutions is accessible, reward models can be trained under the supervision of such ground-truth correctness. Given a question process reward model (PRM) estimates the correctness of individual reasoning steps. The PRM Pprocess is trained with the following objective:
LRRM -p [Loss Pprocess (q.p()) (()))
(1)
where y is the label for the partial solution p(*) and Loss is the loss function. Here we adopt binary cross entropy loss by default. We train a Intern 21-.V.---2-5--3666 (55) on the step-annotated data as introduced in Section 3.3 to get our PRM. We ablate on the choice of PRM in Section 4.3.4
Iterative direct preference optimization. Preference learning has gained increasing focus in VLMs [10]. To further enhance the quality of reasoning chains, we apply RL using the iterative DPO 7 algorithm better align the model's reasoning process towards high-quality reasoning paths, We use SFT model as the policy model Ty. generating 16 reasoning paths for each question. Each reasoning path is evaluated with the trained PRM to assess its quality. We calculate the average score of reasoning steps and the score of final answer respectively, and take the weighted sum of them as the final score. The score is normalized to [0. 1]. Then we select positive-negative pairs depending on the scores. Note that the score of the positive path must exceed that of the negative path by threshold I to ensure sufficient distinction between them, thereby avoiding cases where the differentiation minimal.
Formally, we denote the dataset as DDDo = (x,, U+. y-), where is the question, y+ and y- are the positive and negative responses. The DPO objective is defined below:
CDPO(Ta; weet) -B 1wD log ar (ol Teet Blo no =o(y-lz)
(v(++
(2) where To is the policy model be optimized and Tief is the reference model and they are both initialized with SFT model. is the logistic function and / is set to 0.1. After training, we use
(v
the trained policy model my to generate next-round data ((2) DPo while keeping Wref fixed (i.e., the SFT model). We repeat this procedure for multiple rounds to progressively enhance the the model's reasoning.
4 Experiment
In this section, we first introduce the setting and implementation details in Section 4.1 Then we present a comprehensive comparison with state-of-the-art VLMs across multiple reasoning benchmarks Section 4.2 In Section 4.3] we conduct extensive ablation studies and analyses to validate the effectiveness of our approach and explore several interesting properties.
5

---

Table I: Comparison with open-source VLMs. We evaluate method on six benchmarks covering both general and task-specific reasoning capacities. Our CoS has consistently strong performances these benchmarks, surpassing different baselines and other state-of-the-ar VLMs by large margins. The items in bold and underlined respectively represent the first or second highest scores.
Method
Vista
MMStar
MMMU
M3CTT
A12D
ChariQA
Average
LLaVA-1.5
13B
27.6
32.8
39.5
IXC-2.5
42.9
Ovisl.5-
88
63.0
57.3
48.3
76.4
LLaVA-CoT 58
11B
54,8
57.6
78.7
LlamaV-ol [50
11B
54.4
59,5
81.2
Vision-RI-Llamay>
15
1 11B
62.7
61.4
83.9
RI-VL 62
7B
63.5
60.0
83,9
RI-Omevision
MiniCPM-V-2-6
57.5
56.0
64.2
LLaVA--Oneiiiee Vision
63.2
61.7
48.8
52.3
81.4
80.0
64.6
Qwen2-VL LL [
7B
58.2
60.7
53.7
57.8
83.0
77.4
65.1
Insight-V 10
7B
59.9
61.5
50.2
61.5
79.8
81.5
65.7
InternVT.22.5
62.2
59.6
54.1
62.4
84.5
84,9
68.0
LLaVA-NeXT-LLaMA 28
8B
45.9
43.1
36.9
45.6
71.5
69.4
52.1
SFT
8B
51.4
54.7
39.6
67.4
76.1
75.7
60.8
Itteraivve DPO (CoS-TLaaA)
54.7
58.9
41.8
71.7
79.2
79.1
64.2
IntemVL2..5MMP 55
8B
65.0
60.7
53.8
67.5
84.2
85,0
69.4
SFT
8B
65.9
61.0
53.7
75.7
81.6
88.3
71.0
Iterative DPO (CoS)
67.8
63.5
55.5
81.0
84.9
87.4
73.4
4.1 Experimental Setup
Implementation details. To verify the effectiveness and generalizability our approach, we employ LLaVA-NeXt (28) and IntemVL-2..5MPO [55] as base VLMs, respectively. For each base VLM, Then, based on the SFT model, we apply three rounds of iterative DPO obtain the final model. For each round, compile approximately 20K preference pairs as training data for DPO. The training recipes can be found in Appendix C For PRM, we select Intern VL-2.5-MPO-388 trained it for two epochs on 200K process process-annotated data, using a learning rate of 1c-6 and a batch size of 128 to get the PRM. For the training time, SFT takes about hours on single A800 node, and three rounds of iterative DPO takes about 6 hours in total on a single A800 node.
we first conduct one epoch supervised fine-tuning on ShareGPT-Step-3000
(55) the base model. We
Benchmarks. We evaluate method several representative benchmarks, spreading wide range of tasks that demand the model's complex reasoning abilities. Math Vista 300 is a well-known dataset for multi-modal reasoning, focusing mathematics skills such plane geometry, function and puzzle. MMMU [61] evaluates the model's multimodal reasoning capability across various university-level disciplines. ChartQA 35 aims at assessing logical reasoning with charts. MMstar 4 focuses on various multi-modal problems that must rely on visual contents. M3CoT [5] evaluates the model's s chain-of of-thought reasoning abilities. AI2D [8] is designed test scientific knowledge and reasoning.
4.2
Main Results
We conducted a comprehensive comparison between our method and leading open- VLMs on the aforementioned benchmarks. As shown in Table our CoS achieves substantial improvement over different baselines. Specifically, our method yields an average increase of 4.9% and 3.2% when applied to LLa VA-NNAT and IntemnV-22.5-MPP demonstrating the effectiveness and generalizability of our framework.
For LLaVA-NeXt, both SFT and DPO lead to significant performance gains, indicating that for relatively weaker VLMs, both SFT and reinforcement learning (RL) play crucial roles. contrast, for Intern -L2.5-MPO, the performance improvement from SFT alone is relatively limited. However, after applying iterative DPO, the model can achieve a notable performance boost, reaching state- of-the-art. This indicates that for stronger VLMs, the benefits of further improvement through SFT are marginal, while RL more essential and effective for enhancing strong model's reasoning capabilities.

---

4.3 Analysis and Ablation Study
4.3.1 Reasoning Step Matters
Firstly, we dive into the impact of intermediate reasoning steps and final answers on evaluating reasoning quality. To be specific, we use LLaVA-NeXi (2 fine-tuned on the ShareGPT-Stepp300K as base model and sample reasoning paths and corresponding answers for each question M3CoT. We then utilize a trained PRM to score these reasoning steps and answers. We use the weighted sum of the step score answer score as the selection criterion to report best-of- 16 accuracies.
As shown in Fig- 3 we progres- sively increase the weight of step scores from 0% (i.c., relying solely on the final answer for evalua- tion) to 100% (i.e., relying exclu- sively on reasoning steps). The model's best-of-Ie 16 accuracy ex- hibits trend of initially increas- ing and then decreasing. This indi- cates that relying solely on either the final answer or the intermedi- ate reasoning steps is not optimal; instead, balanced evaluation that considers both preferable. This conclusion also holds for reinforce- ment learning training as shown in Section 4.3.3 Unless mentioned, we set the weight of step scores to 20% by default in our experiments.
LLaVA-NeXI-SFT
Accuracy (%)
80
78
76
74
30
50
100
Step Weight (%)
Figure 3: Ablation step weight. We progressively increase the weight of step score from 0% to 100% and report the best-of- 16 accuracy M3CoТ. Considering both step's ss and answer's quality leads to optimal performances.
4.3.2 Inference-Timee Scaling
Step-level beam search. order to further enhance the reasoning ability during inference, we design step-level beam search by making full use of the structured output CoS. The specific implementation steps are as follows:
Sample responses for the first step.
Use PRM score the N first steps.
Continue to sample the next step based on the best step.
Repeat the above steps until the final answer is generated.
worth noting that step-level beam search's 's cost and sample efficiency exactly the same as that of Best-of-N (i.e., independently sample N responses, score with PRM, and select the best). An illustration is shown Fig.
Next, we explore the inference-time scaling properties of CoS reasoning. We compare four methods: @, Self-Consie Seff-Consistency, PRM, and PRM with step-level beam search. Pass@N refers to the probability that at least one out of N sampled solutions is correct, which can be regarded as the upper bound of inference-time performance. Self-Consistency refers to selecting the most frequent answer among N sampled solutions. PRM refers to selecting the Best-of-N solution based on the scores provided by the PRM. PRM-BS refers to selecting the Best-of-N solution obtained through step-level beam search. We incrementally increase the number of samples N from I to 64 to evaluate the performance of these methods. As shown in Fig. [5] the answer selected by PRM has a significant advantage Self Consistency. Moreover, PRM-B. BSS has achieved consistent improvement PRM. Considering that the sampling efficiency of PRM-BS and PRM is completely the same, this further illustrates the advantages of our method in the inference-time.
4.3.3 Ablation on RL training
proceed investigate the impact of rewards on the RL. training. Specifically, we utilize LLaVA- ScX our base model and employ three methodologies offline select training data for
7

---

Question
Question
Question
0.5
0.9
0.3
0.7
0.6
0.0
0.5
0.9
0.7
0.8
0.7
0.6
0.8
0.8
(a) Consisteeyy
(b) Best-of-N with PRM (ours)
(c) Step Beam Search with PRM (ours)
Verification
Chosen Step
Rejected Step
Figure 4: Illustration of inference-time scaling. Our structured and fine-grained designs enable us to execute (b) Best-of-N with PRM and (c) Step Beam Search with PRM. Both them outperforms Self-Consiste Seff-Consistency by significant margins.
100
100
PaaSSN
PaugN
PRM (vens
97.7
06.5
PRS (nes))
M3CoT Accuracy (%)
PRM4-85
MMStar Accuracy (%6
PHD--B (00-BSSae
K1.1
87.9
118
11.1
629
55.3
79.2
65.4
027
583
62.
70.7
115
71.
-23
70
68
53.8
57.4
50
500
16
15
Sample Num
Sample Num
Figure 5: @ass@N and Best-of-N comparison on M3CoT and MMStar. Here SC refers to Self- Consistency, PRM-BS refers to step-leve beam search. We increase N from 1 to 64, and our method demonstrates appealing inference-ting tetennee-tmeeeeeee property.
iterative DPO: i) Outcome. This approach uses the ground-truth labels of the questions as the gold criterion for judging the quality of responses. Responses with correct answers selected positive samples, while those with incorrect answers are negative ones; ii) PRM scores for answers. This can be seen as lower bound on performance compared the outcome method, as the accuracy of PRM's evaluation of answers is inherently lower than that of direct evaluation using ground-trute labels; iii) PRM scores for both reasoning steps and answers. This method comprehensively considers the quality of step and answer to selects positive and negative pairs.
As shown in Table 2 solely relying answer scoring for selection underperforms the outcome-baat based method, which is an anticipated result. The strategy that takes into account both the intermediate steps and the answer quality achieves the best performances, surpassing outcome-based method with clear gains. This underscores the significance of fine-grained rewards in assessing reasoning quality, thereby enhancing the efficacy of RL training.
Table 2: Ablation on the rewards for RL. Taking both the steps and answer quality into account leads to the best performances.
Method
MathVista
MMStar
M3C.T
LeXVA-NeXt-SFT
51.4
54.7
67.4
Answer (PRM)
53.1
57.3
69.7
Outcome
53.5
58.1
70.0
Answer (PRM)
54.7
58.9
71.7

---

Table 3: Ablation on the PRM. Stronger PRM leads more accurate evaluation for reasoning steps.
Base Model
Question
Step Acc
Answer Acc
LLaVA-NeXI-SB
Seen
85.1
90.5
LLaVA-NeXI-BB
Unseen
83.7
84.7
IntemVL-22.5888
Unseen
85.7
91.1
Intern L--2.-8-
Unseen
87.3
88.7
4.3.4 Ablation on PRM
To evaluate the accuracy of the PRM, we reserved 10K process-annotated reasoning data for assess- ment. Among these, 5K questions were from the PRM training dataset, while the other 5K were unseen during PRM's training, allowing us to evaluate PRM's ability to assess in-domain data and generalization capability unseen questions. We employed three base models: LLaVA-NeXt-8D Intern VL2.5-8B, and Intern VL2.5-38B. As demonstrated in Table the stronger the base model, the more accurate its judgment of intermediate steps for the PRM. Given the critical importance of step evaluation in our framework, we select Intern VL2.5-38B as the base model for our PRM.
4.3.5 Reasoning Length
It has been observed that the length of reasoning tends to increase pro- gressively when LLMs are trained with RL on mathematical prob- lems, and their performance im- proves accordingly [12]. However, things become different for visual complex reasoning. Existing vi- sual complex reasoning problems do not require lengthy intermedi- processes pure mathematics does, but rather rely on the utiliza- tion of visual information and the connection of relevant knowledge trigger reasoning (e.g., geomet- ric shapes). Therefore, we investi- gate how the length of VLM rea- soning steps changes with training. the
6.55
6.45
6,46
6.38
6.37
Reason Step Length
6.4
6.29
5.6
5.48
5.2
5.01
4,66
4,99
5.1
Outzome (Srpe))
(SFT)
DPO Iteration
Figure 6: Reasoning step length curve. Increasing reasoning length is not necessarily better. For visual complex reasoning, ensuring high-quality steps is of vital importance.
We take LLaVA-NeXt-SS base model and calculate the aver- age reasoning step after each round of outcome DPO and PRM DPO, respectively. We depict the change curve of reason step length in Fig.
For outcome DPO, select reasoning paths solely based the correctness of the final results. can be observed that as the training iteration progresses, the reasoning step length progressively increases. This indicates that the model tends to keep extending the reasoning length if don't consider the quality of intermediate steps. Interestingly, the situation is entirely the opposite for PRM DPO. The model tends initially reduce the length of reasoning improve the quality of reasoning. and only after stabilizing does it slowly increase the length of reasoning. Considering that PRM DPO yields better results than outcome DPO as shown in T this phenomenon indicates that for visual complex reasoning, increasing reasoning length is not necessarily better. It is more crucial to first ensure that the model can output quglituaitt reasoning steps.
4.4 Ablation on Reasoning Pattern
In order to demonstrate our reasoning design's advantages, we add direct quantitative comparison between the proposed method and simpler prompting strategies in Table below. Specifically, No Reason denotes setting where the model performs direct question answering without any explicit reasoning; Direct Prompt means directly prompting VLMMs to reason step by step (as in [19]; CoS corresponds to our proposed method using step-level structured reasoning with fine ineignaieed rewards. All methods are trained using the same SFT and RL datasets to ensure fair comparison. For No

---

Table 4: Ablation the reasoning pattern. No Reason denotes setting where the model performs direct question answering without any explicit reasoning; Direct Prompt means directly prompting VLMs reason step by step. The corresponding SFT dataset's format has been adjusted accordingly to make a fair comparison.
Method
Reason
Reward
Math Vista
MMStar
M3CoT
Average
-LaVA-NeXt
45.9
43.1
45.6
44,9
No Reason SFT
49.1
53.9
62.1
55.0
No Reason RL
outcome
51.5
56.4
63.4
57.1 (42.1)
Direct Prompt SFT
51.2
54.9
66.3
57.5
Direct Prompt RL
outcome
69.3
60.2 (+2.7)
CoS SFT
51.4
54.7
67.4
57.8
CoS RL
PRM
54.7
58.9
71.7
61.8 (+4.0)
Table Ablation for GRPO experiments. Our Cos has consistently better performance than the outcome-reward counterpart.
Method
Reward
Math Vista
MMStar
M3CoT
Average
51.4
54.7
67.4
57.8
outcome GRPO
outcome
54.3
57.9
71.4
61.2
CoS GRPO
PRM
56.3
59.1
73.7
63.0
Reason, we use only the question-anssen answer pairs (without intermediate reasoning thought) from the ShareGiTT-tep-3000 dataset for SFT. For Direct Prompt, we concatenate all steps of reasoning (i.e., "thoughts") from each instance into a single reasoning sequence. Since both No Reason and Direct Prompt can not output strict step-level thought, we cannot assess their intermediate reasoning quality, so we apply the outcome reward (i.e., correctness of the final answer) during RL training.
There are two key observations: i) Models trained with aasooing-aaggmente SFT data (i.e., Direct Prompt and Cos) clearly outperform models trained with QA-only data (No Reason) on reasoning benchmarks; ii) Our Cos yields much greater improvements during RL compared to outcome- reward settings. This ablation study directly supports our central motivation for fine- ane-grained reasoning and strengthens the empirical foundation of our conclusions.
4.5 GRPO-based ablation
Recently, GRPO 43 has become the mainstream algorithm in current reinforcement learning and has demonstrated advantages across multiple tasks. Therefore, we further conducted an ablation experiments based the GRPO algorithm validate the effectiveness of proposed method. For GRPO experiments, we deploy the 38B PRM as the reward server. We set batchsize to 128, rollout.n 5, Ir to le-5 and run 100 steps starting from the SFT model. As shown in Table our fine-grained reward RL outperforms outcome reward RL consistently using GRPO, further strengthening our method and analysis.
Conclusion
In this work, introduce Chain of Step (CoS) reasoning for vision-language models, enabling assessing reasoning step quality accurately and leading to effective reinforcement learning and inference-time scaling with fine-grained rewards. Experimental results across multiple benchmarks demonstrate the effectiveness of CoS. More importantly, we conduct extensive empirical analysis and ablations to unveil CoS's appealing properties. We hope this paper offers insights into complex multi-modal reasoning.
Acknowledgments
This work is supported in part by the National Science and Technology Major Project, Grant No.2022ZD01 16403.
10

---

References
[1] Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Vian-Guang Lou. Skill-based sew-shot selection for in-context learning, arXiv preprint arXiv:2305. 14210, 2023.
[2] Ralph Allan Bradlcy and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 345, 1952.
[3] Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings the 29th international conference on computational linguistics. pages 1511- 2022.
[4] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.203300 2024.
[5] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. MB cot: A novel benchmark for multi-domain multi-step multi-moda chain-of-thought. arXiv preprint arXiv:2405. /6473, 2024.
[6] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye. Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 77Xiv:2412.0527/ 2024.
[7] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint 2xXv:2441.001335, 2024.
[8] Zheng Chu. Jingchang Chen. Qianglong Chen. Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning: Advances, frontiers and future. corr, Abs/2309.15400 2023. doi: 10.48550. arXiv preprint ARX/V22309 15402, 2023.
[9] Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, et al. R-cot: Reverse bfain-of-thought problem generation for geometric reasoning in large multimodal models. arXiv preprint arXiv:2410.17885 2024.
[10] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang. Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint 14X1v:2411.14432,
2024.
[11] Robert Geirhos, Jörn-Henri Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-67 673, 2020.
[12] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforvement learning. arXiv preprint arXiv: /2948, 2025.
[13] Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. Sok: Memorization general-purpose large language models. arXiv preprint arXiv:2310. 2023.
[14] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210. 16610, 2022.
[15] Wenxuan Huang. Bohan Jia. Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXi::2003.06745 2025.
[16] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.2222 2024.
[17] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishkyy Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Camey, et Openai system card. arXiv preprint arXiv:2412.16720, 2024.
[18] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
11

---

[19] Takeshi Kojima, Shixiang Shane Gu. Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 25:22199-22211 2022.
[20] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406. 18629, 2024.
[21] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang. Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 2024.
[22] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal lims with generative comprchension. arXiv preprint 2.X1v:2307.161225 2023.
[23] Lei Li, Yuqi Wang. Runxin Xu, Peiyi Wang. Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision- laiion-lngguag models. arXiv preprint arXiv:2403. 002311, 2024.
[24] Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. Scemqa: scientific college entrance level multimodal question answering benchmark. arXiv preprint arXiv:2402.05138. 2024.
[25] Hunter Lightman. Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cohhe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[26] Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language. visual and mathematical reasoning, arXiv preprint arXiv:2208.05355, 2022.
[27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tniigg In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[28] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee, Llavanext: Improved reasoning, ocr, and world knowledge, 2024,
[29] Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu., Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403. /2966, 2024.
[30] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint 2r10:23310.02255. 2023.
[31] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances Neural Information Prucessing Systems, 35:2507-2521 2022.
[32] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209, 14610, 2022.
[33] Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye, Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.2077 20797, 2024.
[34] Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying thain-of-thought reasoning in multimodal mathematics. arXiv preprint arXiv:2501.0445 2025.
[35] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.15 10244, 2022.
[36] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706. 2022.
[37] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200 2209, 2021.
12

---

[38] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin. Bin Wang. Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advancey in Neural Information Processing Systems, 36:25081-25094 2023.
[39] Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192 192-25204, 2021.
[40] Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, and Le Sun. Learning or self- coff-aligning rethinking instruction fine-tuning, arXiv preprint 182::2402..12233 2024.
[41] Dustin Schwenk. Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[42] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang. Yu Liu, and Hongsheng Visual cot: Advancing multi--odal language models with a comprehensive dataset and benchmark for chain-of thouugt reasoning. Advances in Neural Information Processing Systems, 37:8612-8642 2025.
[43] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.08 03300, 2024.
[44] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlci Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[45] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033/4 2024.
[46] Till Speicher, Bishwamittra Ghosh, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Krishna Gummadi, and Evimaria Terzi. Rethinking memorization llms: On learning by rote vs, with understanding.
[47] Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly math and symbolic reasoning. arXiv preprint arXiv:2409. 2et.22222922//. /2/83, 2024.
[48] Winnie Street, John Oliver Siy, Gcoff Keeling, Adrien Baranes, Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Robin IM Dunbar, et al. Lims achieve adult human performance on higher-order theory mind tasks, arXiv preprint arXiv:2405. 18870, 2024.
[49] Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[50] Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint 25010:2500066/600 2025.
[51] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjic Zhan, and Hongsheng Li. Mcasuring multimodal mathematical reasoning with math-vision datasct. arXiv preprint arXiv:2402.14880 2024.
[52] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math sheh-ede Verify and reinforce Ilms byep-by-step without human annotations. arXiv preprint arXiv:2312.00m 08935. 2023.
[53] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution, arXiv preprint arXiv:2409.1219)) 2024.
[54] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40110):4433-2477, 2017.
[55] Weiyun Wang. Zhe Chen, Wenhai Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint 2xX::2411111042.. 2024.
13

---

[56] Xiaofei Wang, Jinhua Li, and Yifan Zhang. Improved value alignment in large language models using variational best-of-n techniques. 2024.
[57] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems, 01224824-24837 2022.
[58] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:2411 2024.
[59] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.1060 2025.
[60] Yuan Yao. Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0180 2024.
[61] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang. Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciplin multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9556 9567, 2024.
[62] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. /29377 2025.
[63] Pan Zhang, Xiaoyi Dong, Yuhang Zang. Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo. Haodong Duan, Bin Wang, Linke Ouyang, et al. 2.oeplmmxxoommserr2..555 A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320 2024.
[64] Renrui Zhang, Dongzhi Jiang. Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[65] Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.08739 2024.
[66] Ruohong Zhang, Bowen Zhang. Yanghao Li, Haotian Zhang, Zhiqing Sun, 7he Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410. 141.11988. 2024.
[67] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint 2xXiv:2501.07300 2025.
[68] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.0349.. 2022.
[69] Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai ol: Opportunities and challenges of agi. arXiv preprint 2409 18486, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the introduction and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We have made the code and dataset publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Section
4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::://///ip...eeeee http:////ccccclleccccccccceeeeea
Answer: [Yes]
Justification: The research is conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper currently does release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

reasoniing_saarfy
reasoning_step_stavee
<
rressoning_ster
Siee__ame_star//Naaeeeee starti/aaaeel///easoonnngggnnn http://assniinnnnteaaaaaeeeeeeee name name_end|>
ooooghnssstttt///oogggh/e/oooooo step. thought_ends
@reasoning_ste__ree
preasoningsstep_enee
<
jreasoning_procceddp
greasoning_step_staree
<|reasoning_stee
N stwpooonnnnntttaaamaaaaeee nam.nameeeee name__nd|>
greasoninggssep_thooee
< reasoningg htapiingggteeeeeee_eeeeee
< begtoninggsstp_endeee
praceed
< |reaseling_proceel>
reasoning <nd|>
answer_start|):Th answer A<|answer_endd:
Figure 7: Illustration of structured reasoning. We take reasoning step (special tokens marked in blue) as the basic unit to conduct structured and fine- fine-grained reasoning.
Special Tokens
Our CoS uses special tokens realize structured step-by-step reasoning. We list all 11 special tokens and their corresponding definitions in Table For the initialization of these special tokens, we use the average value of the token embeddings of its corresponding definition. The reasoning structure illustrated in Fig.
B Prompts
In this section, we provide a demonstration of the prompts used for SFT and PRM data construction. The prompt used create ShareGPT-Step-3000 illustrated Fig. The prompt used to utilize GP to label process-level annotation is depicted in Fig.
C Training Recipes
In this section, we provide the specific hyper- ppper-paramtter settings for the three stages in Table All SFT and iterative DPO experiments are conducted on NVIDIA-A800 GPUs by default. For PRM, it is trained on 16 x NVIDIA-AS00 GPUs. Additionally, we provide important parameters used data construction. During the generation of positive and negative example pairs and the Math-Shepherd (52) Sampling period, we set the temperature to 1.0, return sequences to 16, and top p to 0.95.
Table 6: Special tokens we utilized and their definitions.
Special token
Definition
(reasoning_startl>>
Start token of the reasoning process
End token of the reasoning process Start token of a single reasoning step End token of single reasoning step Start token of the reasoning step name End token of the reasoning step
/reasoniing_steesstartt//
Ireasoninggsteppeedd)
rreasniigg.sttep_aaa__ttarrr
Treasmiinggsttepprmmee_.ddd
reasomingg.ste..toouuttstttrrt
Start token of the reasoning thought content End token of the reasoning thought content Start token of the reasoning reflection content End token of the reasoning reflection content Delimiter token between reasoning steps
reasomiigg.sttep.thoogtttteeddd
ressonigg.teq..eellcciooooaatrr >>
rraaoningg.ttep_refeeectoo....
treasoning_proceed]
22

---

Table 7: Hyperparameter setting and training recipes.
SFT
PRM Training
DPO
Learning Rate
5c-6
1c-6
Epoch
2
3 (iteration)
Warm-up Ratio
0,03
0,03
0,03
Weight Decay
0.05
0.05
0.05
Batch Size
128
128
256
Drop-path Rate
0.1
0.4
0.1
Update Parts
Projector, LLM 300K
Projector, LLM
ViT, Projector, LLM 20K
Data Size
200K
D Dataset Sources
To ensure diversity, our data is collected from various multimodal datasets, encompassing 17 tasks that require diverse multimodal reasoning capabilities. As shown in Table 8 these tasks can be
categorized into four major types including mathematical reasoning [51] 64 3 26 24], scientific
reasoning [[8 31 23 5), chart&document analysis [35 32 37 36), and world knowledge [4] 54
22
Table 8: SFT dataset sources. To ensure diversity, we collect our dataset from a wide range of tasks that demands diverse reasoning capabilities.
Task Type
Dataset
GeoQA+
3]
Vision
Math Vision [TT] SceMQA 24
Mathematics
Math Verse
664] ClatVER-Math 26
Chart
ChariQA
TabMWP 32
ArxivQA
M3CoT S ScienceQA
Science
A12D 18
Document
Voc 37
InfoVQA
36
A-OKVQA
41
FVQA
54
TextVQA 44
Knowledge
SEED
22
E
Attempt for Step-wise DPO
Due to our structured and fine-grained design, we can improve the reasoning ability of the model by conducting fine-grained step level beam search during reasoning. Naturally, wonder whether could construct similar fine-graines preference data during RL training. Specifically, we use PRM to find the optimal step and bad one each step, and then build set of pair data. Then sample the next step based on the optimal step, and cycle until we get the final answer. In this way, we build a pair of preference data at each step, which can provide step-wise fine-grained preference rewards. We term this method to be step-wise DPO. This is different from step-dpo 20 which only detect the first error step while we build pairs of every single step.
However, this method does not work as expected, but has a significant negative impact on model performances shown in Table D We believe that the failure of this method lies in that the next sampling based on a good step is probably also a good step. Therefore, the distinction between the constructed chosen and reject pairs is relatively small. As consequence, in order to avoid outputting negative steps during training, the model will also tend not to output chosen steps. This conjecture validated in the rewards: the rewards of chosen and reject both very small negative numbers. Therefore, we mainly conduct fine-grained step-by-step sampling in the inference time.
F Discussion of Table 3
We observed a counterintuitive phenomenon that the larger InternVL-225-- does not achieve higher Answer Accuracy compared to its smaller counterparts. Our speculation on this phenomenon is as follows:
23

---

Table 9: Ablation on per-step-wise DPO. The chosen and rejected steps too similar to form effective preference pair, so the model will refuse to output both, backfiring performances.
Chosen Rewards
Reject Rewards
Method
M3CoT
Math Vista
SFT
67.4
51.4
Outcome
70.0
53.5
-0.07
-0.16
Per-step-wise
64.8
50.7
-1.39
-3.86
Notably, evaluating intermediate reasoning steps is substantially more challenging than evaluating the final answer for three reasons: i) Higher semantic complexity: Step-level evaluation requires deeper and more holistic understanding of the question. The model must assess whether each step contributes constructively toward the final answer. Similarly, annotating process data is also more cognitively demanding for human annotators. Equation T PRM is trained to minimize the sum of cross-entropy losses over all intermediate steps and the final answer. complete reasoning trace contains multiple steps but only single answer, thus accurately assessing all the intermediate steps is much harder than only assessing the answer. iii Answer memorization. As highlighted in recent studies [L3] 46 LLMs often exhibit "answer hacking" -er--ppcnuungg correct answers via memorized question patterns rather than genuine understanding. In contrast, intermediate steps exhibit higher variance across instances and are less amenable to memorization.
More supervision signals. As shown in
As know, neural networks are inherently short-cut learners [[]] During training, smaller models (i.e., 8B PRM) tend to prioritize optimizing for final answer accuracy, as it offers a more accessible gradient path loss reduction. Improving step-level predictions poses greater optimization challenge. The 38B PRM, with its higher capacity, is better able to model and optimize intermediate reasoning quality. Consequently, achieves greater improvements in step-level accuracy, which more effectively reduces the total mass-resulting in the pattern observed in Table 3
G Limitations
We generated process-labelees data through Monte Carlo estimation and LLLM-as-Judge and verified the effectiveness of our method through extensive experiments. But it cannot be denied that the process data obtained by these two methods still cannot be guaranteed to be 100% correct. Even the process data labeled by humans cannot guarantee accuracy and quality [25] and we do not use human annotation due to the high cost and inability to scale. the future, if there is way to produce nigher-quality process labeled data, our method can hopefully shine further.
H Broader Impacts
This paper presents work whose goal is to advance the field of Vision-language Models and Machine Learning. There are many potential societal consequences of work, none of which we feel must be specifically highlighted here.
24

---

You are thought assistant, will provide you with an image, a question, and reference answer. Your responsibillt is devalop sequence intermediate thinking steps that would logically connect the question to answer.
stop must summarizzng
only consist the following three parts:
Step Thinking: in-depth description the thought process
of step.
explaining being possible.
analyzed.
Note
thinking reasoning
Reflection: reflection
connection prior stops. pointing any patential possible. evidence the that supports informs this step.
There are some guidelines you should follow:
1. Make sure the steps are dear, sequential, and detailed enough to make each step flows logically into the next.
2. The number of steps must be no less than and no more than 10.
Do not state an answer at the beginning. Explain the visuel clues that help to derive the answer.
Don't state that the reference answer correct or consistent to your finding. You are writing your own solution.
explanation can help you derive trinking process..
math probiems, you must give specific operation steps concrate intermediate sotution leading final answer.
sure incorporate
format strictly organized follows:
##tt <Step 1: X0X>
<Detailed Thinking:: xox
<Reflection>: XXX
##8 <Step 2: XXX>
<Detailed Thinking>: XXX
<Reflection>: XXx
<Step x:
<Detailed Thinkinga:
<Reflection>:
#M <Answer>
XXX
Figure Prompt used for generating step-by-step reasoning data.
You are judging assistant specializing in evaluating process reward, the following is an image, a question, reference answer and step-by-step intermediate answer process for the question. The reference answer correct while the intermediate answer steps are not necessarilyy correct.
review and critique each step the following ratings each step:
answer process on correctness. Assign one
Good: The step logical, directly relevant to the question, and most importantly, correct. This provides substantive
progress solve the question.
in reasoning, Or this step only includes some abstract practices rather than concrete solutions, which is not helpful for solving the question in effect. Bad: The step is flawed, wrong, shows misunderstanding of the question context. This step has negative effect on solving the question.
Neutral: The step somewhat relevant but contains minor flaws
For each step, provide brief explanation of your evaluation, identifying areas for improvement Neutral or Bad. Finally, return your output in the form of a step label list. 1 stands Good; stands for Neutral; -1 stands for Bad.
final step label format must strictly organized follows:
Step Label: [1,1,0,
Figure Prompt used for generating process-leve annotation.
25


Reviews:
[{"review_id": "2h7WbMnC1r", "reviewer": "Reviewer_S6Gd", "summary": "This paper introduces a straightforward pipeline designed to enhance the general-purpose capabilities of Multimodal Large Language Models (MLLMs). The proposed method involves three main stages: first, synthesizing step-level reasoning data; second, training a Process Reward Model (PRM) on this data; and finally, leveraging the PRM-generated data for training the MLLM using Direct Preference Optimization (DPO).", "questions": "Please see the weaknesses. I will adjust my score based on the rebuttal.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n- The paper is well-written, presenting a clear and logical progression of ideas, supported by effective illustrations.\n\n- The study is comprehensive, thoroughly addressing key aspects of the pipeline, including data construction and training strategy.\n\nCons:\n\n- Methodological Novelty: While the empirical contributions are appreciated, the methodological novelty appears to be a primary concern. The pipeline primarily integrates existing techniques such as structured Chain-of-Thought (CoT) data synthesis, PRM training, and DPO. The authors should more explicitly articulate the specific innovations their work introduces beyond the novel application and combination of these established methods. Highlighting the distinction between contribution (the overall pipeline) and innovation (the unique technical advancements) would strengthen the paper.\n\n- Analysis of Model Scaling: The results presented in Table 3 are counterintuitive. It is unclear why the larger InternVL-2.5-38B model does not achieve a higher Answer Accuracy compared to its smaller counterparts.\n\n- Inclusion of Stronger Baselines: The evaluation could be made more robust by including stronger, more relevant baselines. For instance, reinforcement learning methods like PPO or GRPO, when combined with a rule-based verifier/reward model, have demonstrated significant improvements in the reasoning abilities of MLLMs. A comparative analysis against such methods under fair settings would provide a clearer perspective on the performance and advantages of the proposed Iterative DPO approach.\n\n- Significance of Performance Gains: The performance improvement of Iterative DPO over the Supervised Fine-Tuning (SFT) baseline appears marginal. The paper would benefit from a more detailed discussion and analysis of these results to better substantiate the significance and practical benefits of the proposed DPO stage.", "was_revised": true, "final_justification": "Although the innovation of this paper is somewhat limited, it is technically solid and supported by sufficient experiments, which may provide some inspiration to the community. I tend to give it a borderline-accept score.", "rebuttal": "Dear reviewer S6Gd,\n\nThank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper. \n\n**Weakness 1** \n\n**For the technical novelty aspect**, our key innovation lies in the step-level structured design, and the consequent fine-grained RL training and test-time scaling with step-level rewards. Prior attempts on structured CoT reasoning, such as LLaVA-CoT[1], explicitly decompose the reasoning into components like *summary*, *caption*, *reason*, and *conclusion*. However, similar to traditional CoT, these methods rely solely on the correctness of the final answer to evaluate the intermediate reasoning quality, which is clearly suboptimal. In contrast, the core motivation of our work is to enable accurate assessment of intermediate reasoning steps, thereby facilitating more effective RL training and test-time scaling. For example, the DPO used in Insight-V[2] select positive and negative sample pairs based only on the answer, while we choose high-quality reason paths based on PRM scores for every single step and the answer, making the training more effective as shown in the experiments. This step-level design is simple and straightforward, but it serves as the foundation for fine-grained evaluation of reasoning quality and distinguishes our method from prior approaches from the technical perspective. \n\n**For the contribution aspect**, we propose a complete and transparent framework for studying the feasibility, empirical performance, and unique training/inference properties of fine-grained step-level reasoning chains. And we plan to release all components to the community to facilitate future research in fine-grained reasoning. In the revision, we will explicitly highlight the distinctions between technical novelty and contributions to help readers better appreciate the significance of our work.\n\n**Weakness 2**\n\nWhile the results in Table 3 may appear counterintuitive, they are in fact consistent with the underlying training dynamics.  \n\nNotably, **evaluating intermediate reasoning steps is substantially more challenging than evaluating the final answer** for three reasons: **i)** Higher semantic complexity: Step-level evaluation requires a deeper and more holistic understanding of the question. The model must assess whether each step contributes constructively toward the final answer. Similarly, annotating process data is also more cognitively demanding for human annotators[3]. **ii)** More supervision signals. As shown in equation 1 in the paper, PRM is trained to minimize the sum of cross-entropy losses over all intermediate steps and the final answer. A complete reasoning trace contains multiple steps but only a single answer, thus accurately assessing all the intermediate steps is much harder than only assessing the answer. **iii)** Answer memorization. As highlighted in recent studies[4,5], LLMs often exhibit \"answer hacking\"—producing correct answers via memorized question or patterns rather than genuine understanding. In contrast, intermediate steps exhibit higher variance across instances and are less amenable to memorization. \n\nAs we know, **neural networks are inherently short-cut learners**[6]. During training, smaller models (i.e., 8B PRM) tend to prioritize optimizing for final answer accuracy, as it offers a more accessible gradient path to loss reduction. Improving step-level predictions poses a greater optimization challenge. The 38B PRM, with its higher capacity, is better able to model and optimize intermediate reasoning quality. Consequently, it achieves greater improvements in step-level accuracy, which more effectively reduces the total loss—resulting in the performance pattern observed in Table 3. \n\nTo verify the consistency of our core finding in this paper, we add an additional experiment using the 8B-PRM. Compared with the  of 38B-PRM, it has a lower accuracy of steps and a higher accuracy of answers. Despite its limited capacity, the results support the same conclusion: jointly optimizing rewards for both intermediate reasoning and final answers yields superior performance.\n\n|                       | MathVista | MMStar | M3CoT | Average |\n| --------------------- | --------- | ------ | ----- | ------- |\n| LLaVA-NeXt-SFT        | 51.4      | 54.7   | 67.4  | 57.8    |\n| Answer (PRM-38B)      | 53.1      | 57.3   | 69.7  | 60.0    |\n| Answer (PRM-8B)       | 53.3      | 57.3   | 70.1  | 60.2    |\n| Outcome               | 53.5      | 58.1   | 70.0  | 60.5    |\n| Step&Answer (PRM-38B) | 54.7      | 58.9   | 71.7  | 61.8    |\n| Step&Answer (PRM-8B)  | 54.6      | 58.3   | 71.5  | 61.5    |\n\n**Weakness 3**\n\nThanks for your suggestion. The core focus of our work is to investigate the impact of fine-grained rewards on reasoning performance—specifically, comparing *step-level reasoning with fine-grained supervision* against the traditional CoT paradigm with final-answer outcomes. The optimization of the RL algorithm itself is not the primary focus of this work. However, we agree that a more relevant baseline can strengthen this paper, thus we add a new experiment comparing against more relevant baselines as suggested. \n\nSpecifically, No Reason denotes a setting where the model performs direct question answering without any explicit reasoning; Direct Prompt means directly prompting VLMs to reason step by step (as in [1]); Cos corresponds to our proposed method using step-level structured reasoning with fine-grained rewards. All methods are trained using the same SFT and RL datasets to ensure a fair comparison. For No Reason, we use only the question-answer pairs (without intermediate reasoning thought) from the ShareGPT-Step-300K dataset for SFT. For Direct Prompt, we concatenate all steps of reasoning (i.e., \"thoughts\") from each instance into a single reasoning sequence. Since both No Reason and Direct Prompt can not output strict step-level thought, we cannot assess their intermediate reasoning quality, so we apply the outcome reward (i.e., correctness of the final answer) during RL training. We use LLaVA-NeXt as the base model. There are two key observations: i) Models trained with reasoning-augmented SFT data (i.e., Direct Prompt and Cos) clearly outperform models trained with QA-only data (No Reason) on reasoning benchmarks; ii) Our Cos yields much greater improvements during RL compared to outcome-only reward settings. \n\n|                   | Reason | RL Reward | MathVista | MMStar | M3CoT | Average     |\n| ----------------- | ------ | --------- | --------- | ------ | ----- | ----------- |\n| LLaVA-NeXt        | -      | -         | 45.9      | 43.1   | 45.6  | 44.9        |\n| No Reason SFT     | ×      | -         | 49.1      | 53.9   | 62.1  | 55.0        |\n| No Reason RL      | ×      | outcome   | 51.5      | 56.4   | 63.4  | 57.1 (+2.1) |\n| Direct Prompt SFT | √      | -         | 51.2      | 54.9   | 66.3  | 57.5        |\n| Direct Prompt RL  | √      | outcome   | 53.1      | 58.2   | 69.3  | 60.2 (+2.7) |\n| CoS SFT           | √      | -         | 51.4      | 54.7   | 67.4  | 57.8        |\n| CoS RL            | √      | PRM       | 54.7      | 58.9   | 71.7  | 61.8 (+4.0) |\n\nThese additional results further support our central motivation: the step-level reasoning paradigm, combined with fine-grained rewards, consistently outperforms traditional CoT approaches that rely solely on final-answer supervision. Regarding RL algorithm, we agree that including experiments with GRPO or PPO would further strengthen the work. However, both GRPO and PPO require online deployment of the PRM, which is much more computationally expensive. In contrast, DPO offers a more resource-efficient alternative by enabling offline preference modeling using PRM. We plan to add a set of GRPO experiment in the final version to further validate our findings. Thanks for your understanding of the computational and time constraints.\n\n**Weakness 4**\n\nWe would like to clarify that our intention is not to demonstrate that iterative DPO is necessarily better than SFT, actually they are both effective in enhancing VLM's reasoning abilities. We conduct experiments on two base models, the weaker LLaVA-NeXt and the stronger InternVL2.5-MPO. For LLaVA-NeXt, both SFT and iterative DPO lead to performance gains and SFT gains are better than iterative DPO gains (+8.7 v.s. +3.4), indicating that for relatively weaker VLMs, SFT is more effective and direct in enhancing the model's performance. In contrast, for the stronger InternVL2.5-MPO, the performance improvement from SFT alone is relatively limited (+1.6). However, after iterative DPO, the model can still achieve a notable performance boost (+2.4), reaching state-of-the-art. This indicates that for stronger VLMs, the benefits of further improvement through SFT are marginal, while RL is more essential and effective for enhancing strong model’s reasoning capabilities. **From the perspective of significance, improving over a strong baseline is more difficult and meaningful than improving over a weak baseline.** For the strong baseline, iterative DPO's gains exceed SFT's gains (+2.4 v.s. +1.6), demonstrating the effectiveness of fine-grained reward RL in improving the upper limit of reasoning ability.\n\n[1] LLaVA-CoT: Let Vision Language Models Reason Step-by-Step.  \n[2] Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models.  \n[3] Let's Verify Step by Step.  \n[4] SoK: Memorization in General-Purpose Large Language Models.  \n[5] Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding.   \n[6] Shortcut Learning in Deep Neural Networks."}, {"review_id": "u0ACaSUTUQ", "reviewer": "Reviewer_gk2j", "summary": "This paper introduces Chain-of-Step (CoS) reasoning, a framework designed to enhance the fine-grained reasoning capabilities of vision-language models (VLMs). The approach decomposes reasoning into structured steps with three components (Name, Thought, Reflection) and leverages a 300K dataset (ShareGPT-Step-300K) for supervised fine-tuning (SFT). It further proposes a Process Reward Model (PRM) trained on step-annotated data to provide fine-grained rewards, enabling effective reinforcement learning via iterative Direct Preference Optimization (DPO). Experiments across multiple benchmarks (e.g., MathVista, MMStar) demonstrate consistent improvements over baselines.", "questions": "1. It appears that the authors have re-implemented supervised fine-tuning (SFT) and Direct Preference Optimization (DPO)-based reinforcement learning from human feedback (RLHF) on vision-language models (VLMs). From a novelty perspective, what are the essential differences between this work and the application of SFT and DPO in large language models (LLMs)?\n2. The baselines only include models size smaller than 8B such as Qwen2-VL-7B, without comparing against the results of the latest Qwen2.5-VL (7B, 72B).\n3. The base model used is only an 8B model. It would be interesting to explore how the proposed approach would perform with larger models, such as Qwen2.5-VL 72B.\n4. After defining the step name, what is the basis for splitting a task into specific steps when completing it?\n5. It would be valuable to see the performance of the proposed approach across additional benchmarks, including MathVerse[1], MathVisoin[2], MMMU-Pro[3], EMMA[4], MegaBench[5].\n\n\n- [1] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186, 2024\n- [2] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024a.\n- [3] Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025.\n- [4] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024b.\n- [5] Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over real-world tasks. arXiv preprint arXiv:2410.10563, 2024a.", "limitations": "See in Quesions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Novel Structured Reasoning Design: The decomposition of reasoning into steps with Name, Thought, and Reflection components, enforced by special tokens, addresses the lack of structure in existing VLM reasoning chains, improving clarity and alignment with visual information.\n\n- Comprehensive Framework: The integration of SFT, PRM, and iterative DPO provides a holistic solution for enhancing VLM reasoning, with rigorous data construction (ShareGPT-Step-300K) and annotation methods (Monte Carlo estimation, LLM-as-Judge) ensuring robustness.\n\nWeaknesses:\nSee in Quesions.", "was_revised": true, "final_justification": "The reviewer answered all my review comments and the questions raised by the rebuttal, and used experiments to prove the effectiveness of their method, so I increased the score accordingly.", "rebuttal": "Dear reviewer gk2j,\n\nThank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper. \n\n**Question 1**\n\nWe would like to highlight two key differences between our framework and prior reasoning methods in LLMs: **i)** Step-level reasoning and fine-grained rewards. Existing LLM (and VLM) approaches typically generate a complete reasoning chain and evaluate its quality based solely on the correctness of the final answer. For example, in LLM-based DPO, preference pairs are constructed based on final-answer's correctness. In contrast, our approach leverages a PRM to assign scores to intermediate reasoning steps and the final answer, enabling the construction of preference pairs that reflects both intermediate reasoning quality and outcome correctness. This fine-grained supervision provides a more informative and nuanced learning signal. **ii)** Visual information. A key distinction between VLMs and LLMs lies in the incorporation of visual information. In VLMs, the reasoning process can reference concrete visual evidence from the image. To support this, we introduce a *visual reflection* mechanism into the design of reasoning steps, allowing the model to explicitly incorporate and verify visual context during reasoning. This capability is absent in standard LLM settings, where no visual modality is available as an external referent.\n\n**Question 2**\n\nFollowing your suggestion, we have added the latest Qwen2.5-VL as a baseline as shown in the table below. Although we build from a weaker base model (InternVL-2.5-MPO), our CoS-InternVL still have competitive performances against Qwen2.5-VL. **Despite the competitive results, we would like to emphasize that achieving state-of-the-art performance is not the primary goal of this work.** Instead, our focus is on exploring and validating the effectiveness of fine-grained reasoning and reward mechanisms, and on empirically characterizing their unique properties. Our core contribution lies in establishing a complete and transparent framework that we believe can serve as a foundational tool for the VL community to further investigate fine-grained reasoning chains. And we will release all the components to the public, facilitating future work on step-level reasoning and fine-grained rewards.\n\n|                 | MathVista | MMStar | MMMU | M3CoT | AI2D | ChartQA | Average |\n| --------------- | --------- | ------ | ---- | ----- | ---- | ------- | ------- |\n| LLaVA-OneVision | 63.2      | 61.7   | 48.8 | 52.3  | 81.4 | 80.0    | 64.6    |\n| Qwen2-VL        | 58.2      | 60.7   | 53.7 | 57.8  | 83.0 | 77.4    | 65.1    |\n| Insight-V       | 59.9      | 61.5   | 50.2 | 61.5  | 79.8 | 81.5    | 65.7    |\n| Qwen2.5-VL      | 68.2      | 63.9   | 58.6 | 77.2  | 83.9 | 87.3    | 73.2    |\n| CoS (ours)      | 67.8      | 63.5   | 55.5 | 81.0  | 84.9 | 87.4    | 73.4    |\n\n**Question 3**\n\nOur experiments are conducted using a medium-scale 8B model, and we compare against baselines with similar parameter sizes (7B, 8B, and 11B). We appreciate the suggestion to explore larger-scale models, but training a 72B model is prohibitively expensive for a school lab. For example, even initiating SFT on a 72B model requires at least 32 A100-80G GPUs, not to mention the cost of iterative DPO and extensive empirical studies. More importantly, we would like to **highlight that 8B is a standard setting for developing and validating new techniques and are commonly adopted in the VLM and reasoning literature[1,2,3]**. The insights and conclusion obtained from studying models at this scale are meaningful and generalizable. Thanks for your understanding of the computational constraints.\n\n**Question 4**\n\nThe splitting steps are completely determined by the model itself, that is, the model itself generates the reasoning of any number of steps according to the current question. And the name of each step is also not pre-defined manually, but generated by the model itself. This is the original intention of our step based reasoning, we do not restrict the number of steps of reasoning and let the model reason end-to-end. In section 4.3.5 and Figure 6 of the paper, we analyzed the change curves of the steps of different methods during training, and got some interesting findings such as *longer reasoning steps do not necessarily lead to better performance*. Please refer to section 4.3.5 for details of reasoning length.\n\n**Question 5**\n\nThank you for the suggestion. We have added evaluation results on MMMU-Pro, EMMA, and MegaBench as suggested. Our CoS outperforms basline consistently on these new benchmarks, further demonstrating our design's effectiveness. As for MathVerse and MathVision, we do not report results on these datasets to ensure a fair comparison, as they are already included in our constructed SFT dataset Share-Step-300K.\n\n|                 | MMMU-Pro | EMMA | MegaBench | Average |\n| --------------- | -------- | ---- | --------- | ------- |\n| InternVL2.5-MPO | 34.5     | 22.9 | 30.6      | 29.3    |\n| Cos-InternVL    | 36.5     | 23.5 | 35.8      | 31.9    |\n\n[1] Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models.   \n[2] R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization.   \n[3] R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization."}, {"review_id": "pAO23QMVmM", "reviewer": "Reviewer_zrV1", "summary": "The paper introduces Chain of Step (CoS) reasoning for vision-language models (VLMs), designed to enable accurate assessment of reasoning step quality and better reasoning capabilities. The core innovation is to propose a step-level reasoning dataset, PRM, and iterative DPO training method. Based on the proposed dataset and method, the author validate the effectiveness on different base VLMs and reasoning benchmarks. In addition, exhaustive ablation studies are conducted to analyze each components, and draw some interesting conclusions (e.g., PRM leads to shorter reasoning length but better performance).", "questions": "1. How will the method perform if GPT-4o is replaced with an open-sourced model?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The CoS method is simple and effective. With the proposed method, LLaVA-NeXT-LLaMA3 can get a over 10 points on the average of 6 reasoning benchmarks. I believe the dataset and method will provide a good baseline for the VL community.\n2. The paper's comprehensive ablation studies yield crucial insights into VLM reasoning. Specifically, they highlight the critical importance of individual reasoning steps and emphasize that the quality of steps outweighs their length. These findings are invaluable for the future development of VL reasoning models.\n3. The clarity and conciseness of the method's illustration are commendable. I really like the writing style: presenting each component with remarkable simplicity and directness, which greatly aids in comprehension.\n\n# Weakness\n1. The method seems to be dependent on GPT-4o. It remains unclear how the proposed approach would perform if a less powerful VLM was substituted for GPT-4o in its key components.", "was_revised": true, "final_justification": "The rebuttal addressed my question. Therefore, I maintain my rating as accept.", "rebuttal": "Dear reviewer zrV1,\n\nThank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper. \n\n**Weakness 1 & Question 1**\n\nWe utilize GPT-4o in two components of our pipeline:\n\n1. **SFT dataset construction**. A key goal of this work is to explore step-level reasoning driven by fine-grained rewards. To enable this, we first require the model to generate outputs in a structured, step-level format. The SFT dataset is constructed to guide the model toward this output format. At this stage, the primary requirement is that the teacher model produces reasonably coherent targeted step-level reasoning traces; high performance is not critical. Thus, GPT-4o can be replaced by a moderately capable open-source VLM (e.g., Qwen-VL) without significantly affecting overall performance, as the purpose of SFT is to induce the output structure rather than to boost reasoning quality.\n2. **Step-level process annotation**. This component relies on GPT-4o to evaluate the quality of intermediate reasoning steps. This requires a model with strong reasoning capability, precise instruction-following, and the ability to produce consistent judgments in a target output format. In this case, replacing GPT-4o with weaker open-source models would substantially degrade annotation accuracy, which in turn would negatively impact the training of the PRM and compromise the reliability of the empirical findings."}, {"review_id": "mKu9M0HNaL", "reviewer": "Reviewer_iMMk", "summary": "This work presents Chain-of-Step (CoS) reasoning for vision-language models (VLMs), introducing a structured, step-wise reasoning framework with fine-grained step-level rewards via a Process Reward Model (PRM). The authors construct the ShareGPT-Step-300K dataset using GPT-4o, develop the PRM for step-level evaluation, and integrate both into a direct preference optimization (DPO) training pipeline. Evaluation on several VQA benchmarks shows that this approach outperforms existing state-of-the-art VLMs.", "questions": "Please answer my concerns in weakness.\nBesides, I have two additional questions that I believe need to be discussed.\n1. Some works [2,3] indicate that using LLM-based reward models can introduce biases, which is termed as “reward hacking”. Does this also happen as you use VLM as PRM?\n2. In Table 3, why does InternVL-2.5-38B, the PRM, achieve the highest Step Acc but only the third-highest Answer Acc?\n\n[2] Liu, et al. LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores. ACL (findings) 2024.\n\n[3] Wang, et al. Large Language Models are not Fair Evaluators. ACL 2024.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**[Strengths]**\n\n1. The idea of moving from traditional, coarse-grained chain-of-thought (CoT) reasoning to a fine-grained, explicitly structured process is well-motivated.\n2. The selected benchmarks are extensive.\n\n**[Weaknesses]**\n\n1. It remains unclear whether directly prompting VLMs to reason step by step (as in [1]) could achieve a comparable level of structured thinking. This paper lacks a direct quantitative comparison between the proposed method and simpler prompting strategies. Including such a baseline would clarify the added value of the structured thinking.\n2. The use of GPT-4o to generate the ShareGPT-Step-300K dataset is weird. In my understanding, GPT-4o is not known for fine-grained reasoning; it is unclear how it can generate high-quality, step-wise data as required by your framework.\n3. The PRM is built using a much larger model (InternVL-2.5-MPO-38B) than the reasoning model (RM, e.g., 8B). However, in practical applications, the PRM should be constrained to be the same size as, or smaller than, the RM (Please correct if I am wrong). Thus, I expect that the Table 1 should be re-evaluated, since the 38B model is used as PRM.\n\n\n[1] Kojima, Takeshi, et al. Large language models are zero-shot reasoners. NeurIPS 2022.", "was_revised": true, "final_justification": "After carefully reading the rebuttal and discussing it with the authors, I raised my score from 2 to 4. \n* Why not higher score?\n    1. The presentation of this paper does not highlight its contribution.\n    2. The writing of this paper, especially the experiment section, requires to be improved.\n\n* Why not lower score?\n    1. The proposed iterative DPO is technically novel, since it provides another way to empower MLLM with reasoning capability without using GRPO-based RL.\n    2. During rebuttal, the authors solved my concerns about the experiment setting and provided the experiment results.", "rebuttal": "Dear reviewer iMMK,\n\nThank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper. \n\n**Weakness 1**\n\nThank you for the insightful suggestion. We add an additional ablation study in the table below. Specifically, No Reason denotes a setting where the model performs direct question answering without any explicit reasoning; Direct Prompt means directly prompting VLMs to reason step by step (as in [1]); Cos corresponds to our proposed method using step-level structured reasoning with fine-grained rewards. All methods are trained using the same SFT and RL datasets to ensure a fair comparison. For No Reason, we use only the question-answer pairs (without intermediate reasoning thought) from the ShareGPT-Step-300K dataset for SFT. For Direct Prompt, we concatenate all steps of reasoning (i.e., \"thoughts\") from each instance into a single reasoning sequence. Since both No Reason and Direct Prompt can not output strict step-level thought, we cannot assess their intermediate reasoning quality, so we apply the outcome reward (i.e., correctness of the final answer) during RL training.\n\nThere are two key observations: **i)** Models trained with reasoning-augmented SFT data (i.e., Direct Prompt and Cos) clearly outperform models trained with QA-only data (No Reason) on reasoning benchmarks; **ii)** Our Cos yields much greater improvements during RL compared to outcome-only reward settings. This ablation study directly supports our central motivation for fine-grained reasoning and strengthens the empirical foundation of our conclusions. \n\n|                   | Reason | RL Reward | MathVista | MMStar | M3CoT | Average     |\n| ----------------- | ------ | --------- | --------- | ------ | ----- | ----------- |\n| LLaVA-NeXt        | -      | -         | 45.9      | 43.1   | 45.6  | 44.9        |\n| No Reason SFT     | ×      | -         | 49.1      | 53.9   | 62.1  | 55.0        |\n| No Reason RL      | ×      | outcome   | 51.5      | 56.4   | 63.4  | 57.1 (+2.1) |\n| Direct Prompt SFT | √      | -         | 51.2      | 54.9   | 66.3  | 57.5        |\n| Direct Prompt RL  | √      | outcome   | 53.1      | 58.2   | 69.3  | 60.2 (+2.7) |\n| CoS SFT           | √      | -         | 51.4      | 54.7   | 67.4  | 57.8        |\n| CoS RL            | √      | PRM       | 54.7      | 58.9   | 71.7  | 61.8 (+4.0) |\n\n**Weakness 2**\n\nWe chose to use GPT-4o to construct the Share-GPT-300K dataset for three main reasons: **i)** The primary objective of our SFT stage is not to improve the model's reasoning ability. The core goal of this paper is to explore step-level reasoning driven by fine-grained reward signals. To enable this, we first require the model to generate reasoning in a structured, step-wise format. Therefore, the main purpose of constructing the SFT dataset is to guide the model towards producing step-level reasoning trajectories that facilitate fine-grained reward assessment. At this stage, the key requirement is to generate coherent and logically sound intermediate reasoning steps; **ii)** Ground-truth is provided as reference. In order to ensure high-quality intermediate reasoning, we provide ground-truth answers as references when prompting GPT. This fundamentally alters the difficulty of the task, making the difference of reasoning abilities between 4o and stronger reasoning models (e.g., o1) less important . In such a setup, instruction-following ability and output consistency become more critical. In our preliminary experiments, when GT answers were provided as hints, GPT-4o did not underperform GPT-o1 in terms of reasoning quality obviously. On the other hand, GPT-4o produced more stable outputs and demonstrated stronger adherence to the target format; **iii)** Money cost. Using GPT-o1 would be more than 10× more expensive than GPT-4o, making the cost of money unaffordable. We also use GPT-4o to generate process annotation for the latter two reasons—better instruction-following and significantly lower cost.\n\n**Weakness 3**\n\nThanks for the suggestion. The reason why we use a 38B PRM is that we hope to provide as accurate process rewards as possible, so that our exploration and empirical studies on fine-grained reward driven reasoning can be more precise and solid. Therefore, we use the largest model supported by our computing resources as the PRM. To address your concern, we conduct a further ablation on the rewards in RL using the 8B-PRM (i.e., expanding Table 2 in the paper). Compared with 38B-PRM, 8B-PRM has a lower accuracy of steps and a higher accuracy of answers as shown in Table 3 in the paper. Despite its lower step accuracy, the results support the same conclusion: jointly optimizing rewards for both intermediate reasoning and final answers yields superior performance. Therefore, changing the PRM of 38B to 8B will not significantly undermine the performance in Table 1, nor will it affect our core conclusions and findings.\n\n| Reward                                             | MathVista | MMStar | M3CoT | Average |\n| -------------------------------------------------- | --------- | ------ | ----- | ------- |\n| LLaVA-NeXt-SFT                                     | 51.4      | 54.7   | 67.4  | 57.8    |\n| Answer (PRM-38B)                                   | 53.1      | 57.3   | 69.7  | 60.0    |\n| Answer (PRM-8B)                                    | 53.3      | 57.3   | 70.1  | 60.2    |\n| Outcome                                            | 53.5      | 58.1   | 70.0  | 60.5    |\n| Step&Answer (PRM-38B) (i.e., Cos-LLaVA in Table 1) | 54.7      | 58.9   | 71.7  | 61.8    |\n| Step&Answer (PRM-8B)                               | 54.6      | 58.3   | 71.5  | 61.5    |\n\n**Question 1**\n\nCurrently, we have not observed obvious reward hacking phenomenon in our experiments.\n\n**Question 2**\n\nWhile the results in Table 3 may appear counterintuitive, they are in fact consistent with the underlying training dynamics.  \n\nNotably, **evaluating intermediate reasoning steps is substantially more challenging than evaluating the final answer** for three reasons: **i)** Higher semantic complexity: Step-level evaluation requires a deeper and more holistic understanding of the question. The model must assess whether each step contributes constructively toward the final answer. Similarly, annotating process data is also more cognitively demanding for human annotators[2]. **ii)** More supervision signals. As shown in equation 1 in the paper, PRM is trained to minimize the sum of cross-entropy losses over all intermediate steps and the final answer. A complete reasoning trace contains multiple steps but only a single answer, thus accurately assessing all the intermediate steps is much harder than only assessing the answer. **iii)** Answer memorization. As highlighted in recent studies[3,4], LLMs often exhibit \"answer hacking\"—producing correct answers via memorized question or patterns rather than genuine understanding. In contrast, intermediate steps exhibit higher variance across instances and are less amenable to memorization. \n\nAs we know, **neural networks are inherently short-cut learners**[5]. During training, smaller models (i.e., 8B PRM) tend to prioritize optimizing for final answer accuracy, as it offers a more accessible gradient path to loss reduction. Improving step-level predictions poses a greater optimization challenge. The 38B PRM, with its higher capacity, is better able to model and optimize intermediate reasoning quality. Consequently, it achieves greater improvements in step-level accuracy, which more effectively reduces the total loss—resulting in the performance pattern observed in Table 3. \n\n\n\n[1] Large language models are zero-shot reasoners.   \n[2] Let's Verify Step by Step.  \n[3] SoK: Memorization in General-Purpose Large Language Models.   \n[4] Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding.     \n[5] Shortcut Learning in Deep Neural Networks."}]

Author rebuttal:

Meta-review: Although one reviewer raised concerns about the presentation quality, all reviewers are consistently positive about the paper’s solid technical contributions and strong empirical results. Therefore, I recommend acceptance.
Decision: Accept (poster)