Boosting LLM Reasoning via Spontaneous Self-Correction
Authors: Xutong Zhao
Tengyu Xu
Xuewei Wang
Zhengxing Chen
Di Jin
Liang Tan
Yen-Ting Lin
Zishun Yu
Zhuokai Zhao
Yun He
Sinong Wang
Han Fang
Sarath Chandar
Chen Zhu
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Reasoning
Reinforcement Learning
TL;DR: We propose a spontaneous self-correction approach to improve LLM's math reasoning capability.

Abstract: While large language models (LLMs) have demonstrated remarkable success on a broad range of tasks, math reasoning remains a challenging one. One of the approaches for improving math reasoning is self-correction, which designs self-improving loops to let the model correct its own mistakes. However, existing self-correction approaches treat corrections as standalone post-generation refinements, relying on extra prompt and system designs to elicit self-corrections, instead of performing real-time, spontaneous self-corrections in a single pass. To address this, we propose **SPOC**, a *spontaneous self-correction* approach that enables LLMs to generate interleaved solutions and verifications in a *single inference pass*, with generation dynamically terminated based on verification outcomes, thereby effectively scaling inference time compute. SPOC considers a multi-agent perspective by assigning dual roles -- solution proposer and verifier -- to the same model. We adopt a simple yet effective approach to generate synthetic data for fine-tuning, enabling the model to develop capabilities for self-verification and multi-agent collaboration. We further improve its solution proposal and verification accuracy through online reinforcement learning. Experiments on mathematical reasoning benchmarks show that SPOC significantly improves performance. Notably, SPOC boosts the accuracy of Llama-3.1-8B and 70B Instruct models, achieving absolute gains of 8.8\% and 11.6\% on MATH500, 10.0\% and 20.0\% on AMC23, and 3.3\% and 6.7\% on AIME24, respectively.

Published as a conference paper at COLM 2025
Boosting LLM Reasoning via Spontaneous Self-Correction
Xutong Zhao!?, 12, Tengyu Xu', Xuewei Wang', Zhengxing Chen', Di Jin', Liang Tan', Yen-Ting Lin', Zishun Yu', Zhuokai Zhao", Yun Hel, Sinong Wang', Han Fang', Sarath Chandar', Chen Zhu'
'MetaA,, Mila Quebec Al Institute & Polytechnique Montréal [xutong zhao@mi la. quebec, zhuchen. cetiheenerrieggall....... com)
Abstract
While large language models (LLMs) have demonstrated remarkable success on a broad range of tasks, math reasoning remains a challenging one. One of the approaches for improving math reasoning is self-correction, which designs self-improving loops to let the model correct its own mistakes. However, existing self-correction approaches treat corrections standalone post-generation refinements, relying extra prompt and system designs to elicit celf-correstions, instead of performing real-time, spontaneous self-corrections in a single pass. To address this, propose SPOC, a spontaneous self-correction approach that enables LLMs to generate interleaved solutions and verifications in single inference pass, with generation dynamically terminated based on verification outcomes, thereby effectively scaling inference time compute. SPOC considers multi-agent perspective by assigning dual roles solution proposer and verifier - to the same model. We adopt a simple yet effective approach to generate synthetic data for fine-tuning, enabling the model to develop capabilities for self-verification and multi-agent collaboration. We further improve its solution proposal and verification accuracy through online reinforcement learning. Experiments on mathematical reasoning benchmarks show that SPOC significantly improves performance. Notably, SPOC boosts the accuracy of -lama-3.1-8B and 70B Instruct models, achieving absolute gains of 8.8% and 11.6% on MATH500, 10.0% and 20.0% on AMC23, and 3.3% and 6.7% on AIME24, respectively.
Introduction
Large Language Models (LLMs) have showcased promising results across a broad spectrum of text generation tasks. Among the various domains of LLM applications, mathematical reasoning remains particularly challenging due to its symbolic and structured nature (Shao et al., 2024; Chen et al., 2024). Recent advances in self-correction (Shinn et al., 2023; Madaan et al., 2023) have emerged as a promising paradigm towards self-improvement through iterative critique and refinement of model's own responses.
However, the effectiveness and practicality of existing self-correction approaches remain un- clear. Naive prompting methods may lead to minimal improvement or performance degra- dation without access to external feedback (Huang et al., 2023; Qu et al., 2024). Finetuning- based methods seek to address such issues by post-training the LLM on refinement data collected from oracles (Saunders et al., 2022; Qu et al., 2024) or the learner model itself (Ku- mar al., 2024). Nonetheless, these approaches typically rely on specific prompt after each model response to trigger self-reflection correction (Figures la and 1b), necessitating additional system design to inject these prompts during inference. In other words, existing approaches lack the ability to spontaneously and adaptively self reflect and correct, resulting in ineffective cest-time compute scaling and inflexible deployment in practice.
To address these challenges, introduce SPOC, spontaneous self-correction approach that enables LLMs to spontaneously generate interleaved solutions and verifications in single inference pass. SPOC employs open-loop inference paradigm, which triggers
1

---

Published as a conference paper at COLM 2025
Qucctton
Quesdon
Questonn
Sulttion
Schution
Solution
request
For fuctio requast
Solttionn 2
Rellection Al
Relecion: An
Solution 3
Solution 2
Solutiion
Reflecion
Closed-loop Venticatee/Correcionn
Open-koop
Verfication
(a)
(b)
(c)
Figure Multi-turn generation formalisms. (a) & (b) Sample closed-loop paradigms that require extra system designs and prompting to trigger and terminate correction; (c) Sample open-loop paradigm that spontaneously adapts generations.
self-correction only when the self-verrification identifies errors, and iteratively revises the solution until it passes self-verification, without requiring any external interventions during response generation. dynamically elicits and terminates generations on-the-fly using solely the model's inherent capabilities, thereby effectively scaling inference time compute. We consider a multi-agent formalism that models the alternating solutions and verifications as the interaction between solution proposer and verifier, and adopt self-play training strategy by assigning dual roles to the same model. We adopt a simple yet effective ap- proach to generate synthetic data from the initial model for supervised fine-tuning (Welleck et al., 2022), enabling the model to adhere to the multi-turn generation style, meanwhile developing capabilities for self-verification and inter-agent collaboration without distilling from stronger teacher. We further boost the model's accuracy in its solution proposal and verification via online reinforcement learning, using the correctness of solutions and verifications as the reward.
Our main contributions are threefold:
We demonstrate that generating self-verification and correction trajectories from the initial model's correct and incorrect outputs effectively bootstraps its spontaneous self-verification and correction behavior. We call out the importance of data balanc- ing in achieving high verification accuracy in this stage, which in turn benefits the subsequent RL phase.
We propose the message-wise online RL framework for SPOC, and present the formulation of RAFT (Dong et al., 2023) and RLOO (Ahmadian et al., 2024) as the RL stage of SPOC for enhancing self-verification and correction accuracies. Our results show that RLOO, augmented with process rewards for each solution or verification step, yields stronger results.
We achieve significant improvements on math reasoning tasks across model sizes and task difficulties using our pipeline without distilling from stronger mod- els. SPOC boosts the pass@1 accuracy of Llama-3.1-81 and 70B Instruct mod- els els-improving performance by 8.8% and 11.6% on MATH500, by 10.0% and 20.0% AMC23, and by 3.3% and 6.7% on AIME24.
2

---

Published as a conference paper at COLM 2025
2 Related work
Self-correction. Given that high-quality external feedback is often unavailable across vari- ous realistic circumstances, it is beneficial to enable an LLM to correct its initial responses based on solely on its inherent capabilities. Prior works on such intrinsic self-correction (Huang et al., 2023) or self-refinement can be categorized into two groups based on the prob- lem settings and correction mechanisms: prompting and finetuning. Recent works (Huang et al., 2023; Qu et al., 2024) show that prior prompting methods lead to minimal improve- ment or degrading performance without strong assumptions on problem settings. For instance, Shinn et al. (2023) rely on oracle labels which are often unavailable in real-world applications; Madaan et al. (2023) use less informative prompts for initial responses, re- sulting in overestimation of correction performance. Finetuning methods seek to improve correction performance via finetuning the LLM on refinement data, collected from human annotators (Saunders et al., 2022), stronger models (Qu et al., 2024), the learner model itself (Kumar et al., 2024). However, these works lack the mechanisms that correct errors while generating solutions in a single inference pass (Ye al., 2024). Our work is akin to concurrent works on self-correction (Ma et al., 2025; Xiong et al., 2025). Differently, Xiong et al. (2023) re-attempts a solution within the verification instead of evaluating the previous one; moreover, they only apply RAFT in their learning framework, while we also conduct experiments on RLOO. Ma et al. (2025) uses the more complex GRPO as their RL algorithm, while we show that better performance can be achieved in the same setting (Llama 3.1 8B) by using simpler RL algorithms like RAFT for SPOC.
Multi-agent frameworks. By introducing multiple roles into problem-solving, multi-agent formalisms serve as a different perspective to address complex reasoning tasks. Auto- Gen (Wu et al., 2023) and debate-based frameworks (Du et al., 2023; Liang et al., 2023) solve math problems through customized inter-agent conversations. Despite increased test-time computation, these works lack post-training for different agent roles, which may result in suboptimal performance or distribution shifts at inference time (Xiang et al., 2025). While other works train separate models to perform correction (Motwani al., 2024; Havrilla et al., 2024; Akyürek et al., 2023; Paul et al., 2023), models do not perform spontaneous corrections during solution generations; instead, they require extra system designs to trigger and stop corrections at deployment. In contrast, our method enables dynamic inference-tinn time scaling by improving the model's own inherent deliberation capabilities.
3 Method
In this section, first introduce the multi-turn formalism, in which the agent performs interleaved solution and verification turns. We then discuss how we finetune the agent to ensure it consistently adheres to the multi-turn response style. We finally describe our online reinforcement learning scheme which further boosts the final accuracy of the policy. Figure illustrates the two stages, fine-tuning and online RL, of SPOC.
3.1 Multi-turn formalism
Problem setup. Let X (x;, be a dataset of N math problems, where each pair yx) contains a question x; and the corresponding solution yf with ground-truth final answer. An LLM agent is defined by the policy n0[-1x), parameterized by 0, that generates the solution y to solve the given problem x.
Alternated-turn generation. Suppose given a question x, the LLM generates a trajectory
consisting of L interleaved solutions and verifications = (y1, (u,,,,...,, .I....,yLI. VL, PL), where solution 1/1 indicating the model's I-th complete solution attempt that reaches a final answer, and verification i indicating the 1-th self-verification validating correctness of the solution y/- For clarity, message or turn refers to each single solution y! or verification V, and response or generation refers to the entire trajectory until the end. For brevity, we denote
3

---

Published as a conference paper at COLM 2025
Pair Supervised Fine-Tuning
Online Reinforcement Learning
Question
Quection
D
8
X
S
pair
RL
SFT
S
S
X
Q
Comiit
X
Intames
Solution
[]
Q
Verification
Figure SPOC training overview. Left: PairSFT for initializing multi-turn generation. Right: Online RL for policy optimization.
previous turns by: TΙ = (y13, Vny) and z = (y1.t. P1s-1). The timestep E No indicates single decoding step where the LLM outputs one token from its policy distribution.
Multi-agent formulation. We model the reasoning task as an extensive-form game (EFG) (Osborne, 1994; Shoham Leyton-Brown, 2008), which generalizes the Markov De- cision Process (MDP) (Sutton, 2018) to turn-taking interaction between solution proposer and verifier. At each turn, the proposer outputs solution to the given math problem, and the verifier assesses its correctness. In this context, the EFG is tuple where N' = f1,...,,I n) is the set of и = 2 players (i.e. the proposer and verifier), AA is finite set of actions (i.e. the LLM's token space), S is finite set of states (i.e. each state is question and a sequence of reasoning/ verification steps in context), T C is subset of terminal states (i.e. complete response trajectories
Tr,,I, ?),
No - A C R" is the reward function assigning each player scalar utility at terminal states (i.e. A, = {0,1} characterizes binary outcome feedback), I S - / is player identity function identifing which player acts s (i.e. I(T) and I(T 2), and y E [0,1] is the discount factor.
(y1, p/,..., .1........, UL),
Unlike the general definition of EFGs, we do not distinguish between histories and states due to the deterministic dynamics and perfect-information nature in mathematical reasoning (i.e. +)+1 T U (y1=1,0141)), We denote the proposer's 's and the verifier's action spaces as Asl A and A A, representing the set of solution and verification messages, respectively. We define a per-step reward function for a transition as r(s, a) representing vector of reward to both agents. The return for player i A is defined as Gr,i It T Oz = Ex [G, = s, a).
The corresponding state-action value function under policy π is
To improve reasoning capabilities by learning from both solution and verification ex- periences, we adopt the commonly-used self-play strategy with parameter sharing (Al- brecht et al., 2024), where the proposer policy Λ + A(A*)) and the verifier policy nz S + ( share the same set of parameters 0. The policy πe outputs alternated solution and verification messages depending on the context
Policy optimization. We optimize the policy no by maximizing the KL-regularized leam- ing objective
((0) Es- - 17 ESpp [KL [T,)
(1)
where p indicates the discounted state distribution, d/ > is the KL-regularization coefficient, and TTg is the reference policy parameterized by the initial parameters og. This objective has
Different from the classic self-play in zero-sum games (e.g., AlphaZero (Silver et al., 2017)), ours involves non-symmetrical roles in the sense that two policies are different conditioned the context.
4

---

Published as a conference paper at COLM 2025
vf
vf
VT
C
I
C
s1
sl
sl
C
1,
1
1,
0
1,0
1,
1
1,1
I
0,
0,0
0,0
0, 0
0,0
(a) Corr
(b) Last
(c) All
Figure Reward configurations for policy optimization, where sl, vf, C, indicate solution, verifica- tion, correct, and incorrect, respectively. For Last and All, SPOC optimizes correct solutions (first row in each table) only when the last solution is correct.
a close-form solution for the optimal policy π (als) = Z(s) no, (als) exp(, Q(s,a)) where (s) = Ea-m (-1s) (-16) (exp(, @(s, Given our multi-agent formulation, this objective intro- duces an individual objective for each role, namely
(0) E[Q 17a1 IB[KL (-1s) (-Is) (-1s))] (²) = E(Q - 1f E| (-1s)
(3)
Due to shared parameters across both roles, we jointly optimize both objectives using common generated trajectory experiences. Hence the optimal proposer and verifier poli- cies satisfy z251* (als) Tit (als) exp(, (s, and 7 (a|s) Th (P(() respectively, implying the optimal shared policy increases the probability of outputting high-rewarding solutions/ Ieluuiioss/vriiicaattii /wwriiviaiiiiis....... Note that the optimal policy for the unregular- ized learming objective (n 0) results in the maximizer of the action-value function: n (-1s) = arg maX)+A(A) (Q also yielding high probablity of generating high- rewarding messages.
Reward setting. To obtain a reward signal for each token each message, we evaluate the outcome correctness of each message. In particular, assume access to rule-based checker for the final answer in the solution, and provide a binary outcome reward denoted by y (v, E {0, 1], where (y, y*) = 1 when the model answer matches the ground- truth answer. Similarly, we parse the Yes / conclusion in each verification, and denote ا ( y") indicating the ground-truth verification. Figure 3a shows the joint reward setting, denoted by Corr hereafter. To obtain maximal returns against each other role, our reward setting admits one unique Nash equilibrium (Shoham & -eyton- 2008) with the joint policy (i.e. the shared policy πτ) generating both correct solutions and correct verifications.
the reward function by y't (z, x*) E {0, 1}, with
3.2 Enabling multi-turn generation
Since off-the-shelf LLMs do not adhere to the response style of interleaved solution and verification turns by default, before conducting RL optimization, we first perform an initial finetuning with multi-turn data to enable such behaviour. To collect such data, we imple- ment variant of Pair-SFT (Kumar et al., 2024; Welleck et al., 2022) to construct synthetic correction responses.
In particular, we rollout the base policy ne, collect single-turn responses for each question E X, denoted by } ~ (- (-1x;). For each response, we record its binary correctness using the solution reward function r اlر (y), We obtain the verification message of one single-turn response by pairing it with correct sampled response. To generate verification of one response, either correct or incorrect, we prompt the same base model Ne, to identify the potential error, briefly explain it, and output final binary conclusion indicating correctness of the given solution. The entire verification message is denoted as Di TTp (-|x;, Vi, y ), where y indicates the correct sample. We denote this synthetic multi- turn correction dataset as the Pair-SFT dataset D Dpair {x;, of yy U { "/t o))), where the +/ -/- superscripts indicates correctness of the corresponding solution turn. We

---

Published as a conference paper at COLM 2025
perform SFT finetuning the base model, with tokens in incorrect messages masked out, and denote the finetuned model by a In practice, we observe that reweighting the subsets {x Vi P; Vf; } and VT 0 } to approximately the same scale leads to a Tean with higher verification accuracy and more stable RL training afterwards. The complete training data collection procedure detailed in Algorithm 2.
When generating the verification messages, we adapt the generative critic method (Zhang et al., 2024; Zheng et al., 2024) that prompts the model respond with rationales before judging solution correctness, except that our variant concisely explains the error rather than performing a chain-of-thon -fthoughh (COT) analysis. Obtaining strong COT verifier requires explicit training and it is out of scope of this work. Prompt templates for data construction are detailed in Appendix E.
Algorithm SPOC Message-wise Online Reinforcement Learning
Inputs: Question-ansser dataset D X' x y bf policy model 7i8 parame- terized by 0, number of questions N, number of steps T, number of rollouts per question K, batch size B, rule-based solution correctness reward function (y. E {0,1},
verification correctness reward function (v, v*) E {0,1}
2: for 1, , do do 3: Sample batch D; C D of size B
4: Sample K trajectories for each xj E X;: (T/) (-11xx),, where of = (v T:1y)
5: Label binary rewards: rall i.k.
o(fa)), where oiA1 = sll j.k. 6: Update policy with any policy optimization algorithm (e.g. Algorithm Algorithm 4)
= ( ik ,y),
r = (of -
7: end for
8: return T
3.3 Online reinforcement learning
With the multi-turn problem formulated and the agent adhering to the multi-turn responses style, we conduct online reinforcement learning to improve the policy performance. The overall message-level RL training procedure described in Algorithm 1. While SPOC is compatible with any policy optimization method, we apply RAFT (Dong et al., 2023) unless otherwise specified. The RAFT policy optimization algorithm is presented in Algorithm 3.
Besides the RAFT policy optimizer, we also implement an RLOO (Ahmadian et al., 2024) variant, which replaces the leave-one-out procedure with subtraction of the mean reward across all messages, followed by division by the standard deviation. We refer to this approach as RLOO for brevity. Unlike the best-of-N (BoN) response selection strategy in RAFT, RLOO optimizes the policy using all generated responses, enjoying better sample efficiency. The RLOO policy optimization detailed in Algorithm 4.
Experiments
In this section we present empirical experiments on math reasoning benchmarks. We first overview the tasks conduct experiments We then describe the experimental setup and evaluation protocols. Finally we discuss the results and provide ablation studies.
4.1 Experimental setup
Tasks. We perform experiments on established math reasoning benchmarks. To enable rule-based answer checking, all problems in selected benchmarks require verifiable final output. We evaluate models on benchmarks: (1) MATH500 (Lightman et al., 2023), a curated dataset of 500 problems selected from the full MATH (Hendrycks et al., 2021) evaluation set; (2) AMC23 (AI-MO, 2023), a dataset of 40 challenging competition questions; (3) AIME24 (AI-MO, 2024), dataset of 30 more difficult competition problems.

---

Published as a conference paper at COLM 2025
Approach
MATH500
AMC23
AIME24
-lama-3.-&8B-nnsteee -B--Instruct (Dubey et al,, 2024)
52.2
22.5
3.3
SFT
53.6
32.5
3.3
RAFT
55.2
27.5
PairSFT
53.8
22.5
10.0
Self Refine (w/o oracle)
39.4
20.0
3.3
Self- Reff-Refie (w/ oracle)
57.0
35.0
3.3
R-B-BI (Ma et al., 2025)
49.6
20.0
10.0
SAR-PRL*
53.6
25.0
6.7
S-R-ORL=
55.0
32.5
6.7
SPOC
61.0
32.5
6.7
Llama-3.1-700 3.0m-3.177B-Insttrut (Dubey et al., 2024)
65.8
32.5
16.7
70.4
RAFT
74.2
52.5
20.0
PairSFT
74.8
47.5
23.3
Seff-Refine (w/o oracle)
54.2
42.5
13.3
Self-Refine (w/ oracle)
72.2
47.5
26.7
SPOC
77.4
52.5
23.3
1lamm-3.3-70B--nsss (AT, 2024)
75.6
57.5
SFT
73.6
55.0
23.3
RAFT
76.6
62.5
20.0
PairSFT
75.0
62.5
23.3
Relf-Refine (w/o oracle)
75.4
33.3
Self-Refine (w/ oracle)
76.2
65.0
26.7
SPOC
77.8
70.0
23.3
DeepSeck C1-Distill- Distill-Llama- -lama-8B (Guo et al., 2025)
62.6
62.5
26.7
SFT
76,8
65.0
30,0
RAFT
74,2
6.7
PairSFT
73.2
77.5
16.7
Self-Refine (w/o oracle)
67.4
75.0
10.0
Self-Refine (w/ oracle)
71.2
65.0
40.0
SPOC
77.6
70.0
23.3
SPOC-RLOO
DeegSeek-Ri-DDatll-llaaa--- (Guo al., 2025)
82.8
72.5
60.0
SFT
90.6
80.0
40.0
RAFT
87.4
85.0
50.0
PairSFT
92.6
95.0
63.3
Self-Refine
86.2
80.0
30,0
Self-Refin RaffReeiin (w/ oracle)
88.6
72.5
30.0
SPOC
89.6
85.0
53.3
SPOC-RLOO
94.6
92.5
76.7
Gemini-1.5-Flash (4-shot)s (Team et al., 2024)
54.9
SCoRe= (Kumar al., 2U24)
6444
Llama-3-8B-Instructt (4-shot)s (Meta, 2024)
30.0
Self-rewarding IFT: (Xiong al., 2025)
27.9
Self-rewarding-IFT Gold RMs
33.9
DeepSeck-RD R1- Distill-Llamann 1---RRee Rltok-avg@4+
88.9
92.5
48.3
DeepSeek-R1 MaaaSeek--R--iiin 1t--Ditill--Iamm-----R
82.2
87.5
36.7
DeepSeek-R1 HiisSeekRR--issill-- Distill-Llamm-88.
89.1
50.4
DeepSeck-RR I--iisiill-Lamman -00---00k---g0044
94.3
94.4
65.9
DeepSeek-RR R1- D--Dssill-Llama-700 -lamma-00--- 70B-Rltokt
91.2
80.0
56.7
94.5
70.0
Qwen2 Mathh-7--Bii Math-7B-Instructt (Yang et al., 2024)
82.8
16.7
84.8
72.5
26.7
01+
94.8
74.4
GPT-4os
60.3
9.3
Claude 3.5 Sonnets
78.0
16.0
Table 1: Main evaluation results. Bold and underlined performance scores indicate the best learning and best prompting (if any) results under each initial model, respectively. Blue means ours, and green means other RL based approaches. Baselines marked with are reported by the original works, while those marked with are obtained by evaluating the open-source models on Huggingface using greedy decoding (unless stated otherwise). "R1tok" indicates the model is evaluated using the R1 modified tokenizer and chat configs. "avg@4" indicates the model is evaluated using sampling, with the temperature of 0.6, the top-p value of 0.95, and 4 responses generated per question to compute the mean pass@1 (Guo al., 2025).

---

Published as a conference paper at COLM 2025
Base Model trained w/ SPOC
Base.Acc.
Verif.Acc.8tl
Acc.@tl
Acc.012
Af1),
Ac-i
4i-
Llama-3. 1aamma-31-8B-Insttuuee
52.2
80.2
59.0
61.0
2.0
8/29
18/79
65.8
80.0
77.0
77.4
0.4
3/10
5/8
Llama-3.3-70B-Inst
75.6
81.8
77.8
77.8
0.0
1/4
1/20
Table 2: Performance across first two solution turns on MATH500. Ac-ii and Ap-t indicate Reaa
Me-32 +
and Wi over turns, respectively.
Mi-s
Evaluation protocol. Our primary evaluation metric is the final answer accuracy. We addi- tionally report cross-solution correction accuracy serving complementary evaluation.
For all experiments, we finetune Allma-3-IInse Innstrcctt models (Dubey et al., 2024) (3.1-BB & 70B, 3.3-70B, DeepSeek-Rt-Distill-LLeee using the NuminaMath dataset (LI et al., 2024), which consists of training sets from various data sources, covering a wide range of mathematical topics and difficulty levels. We exclude the Orca-Math dataset (Mitra et al., 2024) and synthetic data subset since their correctness are not human-validated despite their large scale.
8B & 70B) as the base models. We conduct training
For evaluations, we report the pass@1 accuracy of the final answer. We use greedy decoding and zero-shot COT prompting unless otherwise specified. As mentioned in previous sections, we do not utilize additional external instructions to prompt the finetuned model to attempt another solution trial; instead the model spontaneously performs self-verification to determine whether another attempt is needed. Our prompt templates for evaluation are included in Appendix E.
Implementation details. All models are prompted with the original Llama tokenizer and chat configs (Dubey et al., 2024) unless otherwise specified. All models except the DeepSeek-RI-Distil--LUe
based ones are evaluated using the maximum generation length of 6, 144 tokens, while the PreeSee--R--Distil--Innne maximum generation length of 32, 768 tokens, as per Guo et al. (2025), To support training with multi-message responses, we utilize different special termination tokens for each model message. In particular, in each model response each message starts with assistant header tokens, indicating the source of message is the model. Besides, every assitant message except the last ends with an <|eom_ide _id__ddd> termination token, representing the end of one message. The last assistant message ends with an <leot_id|> token, which concludes the entire model response. We implement RAFT (Dong al., 2023) under the CGPO (Xu al., 2024) framework, which allows for filtering out prompts whose all corresponding sampled responses contain no correct solutions or verifications.
based models are evaluated using the
4.2 Results
Table 1 presents the comprehensive evaluation results, showing the comparisons across different initial models and parameter scales. In general, SPOC consistently outperforms the base models on all initialization models across all benchmark tasks. Notably, SPOC en- hances the accuracy of Llama3.1 8B and 70B, reaching gains of 8.8% and 11.6% on MATH500, 10.0% and 20.0% on AMC23, and 3.3% and 6.7% on AIME24, respectively. This result highlights the effectiveness of SPOC across different parameter scales and task difficulties.
SPOC also achieves consistent enhancement when fine-tuned with strong initial models. Despite marginal improvement on (lama3.3-70p) model, SPOC obtains significant overall outperformance compared the baselines after finetuning the DeepSsek-RI-DistillLLnnm models. Respectively on MATH500/ AMC23/ AIME24, SPOC reaches 21..%/770.%%/...3 with the 8B model, and 89.9% 89.9% /53.3% with the 70B model. Furthermore, SPOC achieves more drastic performance improvement using the RLOO policy optimizer, obtain- ing 87 37.2% / 87.5% 50.0% with the 8B model, and (..6% 76.7% with the 70B model. base and their corresponding Rltok results attributed to different tokenizers and chat configurations.
is important to note that the gap between evaluation of -ei-see-----iitilll- models for
8

---

Published as a conference paper at COLM 2025
Table 2 shows performance across the first two solution turns MATH500. Overall, SPOC achieves consistent improvement on the second solution turns over the first. With the smaller Llama3.1-8B model, SPOC shows more inclination to generate second solution turn, resulting in a more significant improvement margin. With larger 70B models that achieve higher final accuracy, on the other hand, SPOC tends to get the first solution message correct in the first place, resulting in an already strong turnl performance and marginal f2), Such behaviour well aligned with our expected Nash equilibrium admitted by the Corr reward setting, where policy optimization encourages the joint policy to generate both correct solutions and correct verifications in the first place. The complete per-turn performance analysis and diagnostics of verifier reliability are presented in Appendix C.
Table shows the performance of applying multiple iterations PairSFT-RL. training proce- dure. Results indicate that the second iteration still leads to overall consistent improvement over all models. Although the overall improvement mainly marginal, the second iteration shows larger gain in challenging competition benchmarks. For instance, with Llama3.1- 70B, iter2 improves over iterl by 10% and 6.7% on AMC23 and AIME24, respectively.
Approach
MATH500
AMC23
AIME24
52.2
22.5
3.3
PairSFT (iterl)
53.8
22.5
10.0
SPOC (iterl)
61.0
32.5
6.7
PairSFT (iter2)
60.8
35.0
SPOC (iter2)
62.0
32.5
10.0
Llama-3.1-70B-Inss.
65.8
32.5
16.7
PairSFT (iterl)
74.8
47.5
23.3
SPOC (iterl)
77.4
52.5
23.3
PairSFT (iter2)
76.4
67.5
20.0
SPOC (iter2)
77.6
62.5
30.0
1lama-3.3770--Immttee
75.6
57.5
26.7
PairSFT (iterl)
75.0
62.5
23.3
SPOC (iterl)
77.8
70.0
23.3
PairSFT (iter2)
79.6
72.5
26.7
SPOC (iter2)
79.8
70.0
30.0
Table 3: Iterative training performance. The second iteration still yields overall consistent improvement across all models.
Ablations
We conduct ablation experiments differ- ent reward configurations, as overviewed in Figure 3. We present comparisons with the default Corr reward setting in Table 4, as the base
Model
MATH500
AMC23
AIME24
Base
52.2
22.5
3.3
SPOC-Corr SPOC-Last
61.0
32.5
6.7
using Llama-3.1-8B-Instrree model. Compared to Corr, the ablation variants Last and All do not yield a unique Nash equilibrium; instead, they promote generating correct solutions regardless of the correctness of verifications. Results show that both variants still improve performance over the baseline; however, they both underperform Corr two out of three tasks. Last and All obtains only one more correct answer than Corr in AIME24 and AMC23, respectively, while the performance discrepancy MATH500 dominates the overall gap. The ablation highlights the importance of jointly optimizing the correctness of both solutions and verifications.
59.8
27.5
10.0
SPOC-AII
58.4
35.0
6.7
Table 4: Ablation experiments under different re- ward settings. Experiments are conducted on the Llama-3.1-8B-Inse
model.

---

Published as a conference paper at COLM 2025
5 Conclusions
In this work, we tackle the mathematical reasoning challenge for Large Language Models by promoting intrinsic self-corrections. We propose SPOC, novel approach that enables spon- taneous, real-time solution proposal and verification within single inference pass. SPOC frames the reasoning process as multi-agent collaboration, where the model assumes both the roles of a solution proposer and verifier. SPOC dynamically elicits and terminates reason- ing generations based on verification results, which flexibly and efficiently scales inference- time compute while improving accuracy. SPOC leverages synthetic data for fine-tuning and further enhances performance via online reinforcement learning, without requiring hu- man or oracle input. Comprehensive empirical evaluations on challenging math reasoning benchmarks showcase SPOC's efficacy, yielding substantial performance improvement.
Our results highlight the potential of spontaneous self-correction as an effective strategy for advancing LLM reasoning capabilities. To address the prohibitive length of long Colls (Marjanović et al., 2025), future work could explore extending SPOC to partial solutions in long reasoning chains, using step-level process rewards to guide RL training and enable dynamic revisions when ertors are detected until reaching the final answer. It would also be interesting to adopt SPOC to broader reasoning domains beyond mathematics, further enhancing its applicability.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXio preprint 4rX1o:2402.14740, 2024.
Meta AI. L.ama-3.3-70binnst 3-705- 10nt0-nnstuttt.... 2024.
AI-MO. American mathematics contest.
httpss///huginngaaccco////tsssts
aimo-validation amc, 2023.
AI-MO. American invitational mathematics examination. https://uuggingface.co dattasstss/ALtoooaaanovvvvaaaaia aime, 2024.
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. Ri4f: Generating natural language feedback reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.08844 088444 2023.
Stefano Albrecht, Filippos Christianos, and Lukas Schäfer. Multi-agent reinforcement learning: Foundations and modern approaches. MIT Press, 2024.
Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao. Autoprm: Automating procedural ural supervision for multi-step reasoning via controllable question decomposition. Xiv preprint ar Xxxi:2402.114555 2024.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Dipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Xiv preprint ar XrXv:2304.06767. 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xrviv preprint ar Xiv:2407.22 21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. Xiv preprint ar Xix::22501 1 1248,, 2025.
10

---

Published as a conference paper at COLM 2025
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yi Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to improve Ilm reasoning via global and local refinements. arXiv preprint ar Xfxo::240..,9963, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arxiv preprint ar Xrx::22003.3877, 2021.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. Xiv preprint ar Xrie:2310.017996 2023.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. Xiv preprint ar XrXio:2409.12917, 2024.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Nu- minamath. aagpeet-nnumnn/aaiiopppppaa httpr
(https: thub. com/ daa.aaattt...dd)) 2024.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint ar Kiv:2305.19118 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint ar Xix:2305. 2023.
Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S r: Teaching Ilms to self-verify and self-correct via reinforcement learning. Xrv preprint ar Xrxx:22502.122 12855, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 46534 46594, 2023.
Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lù, et al. Deepseek-rl thoughtology: Let's think about 1lm reasoning. ar preprint arXiv:2504.07122, 2025.
Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta Al, 2(5):6, 2024.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of sims in grade school math. arXiv preprint ar arXio:2402.14880, 2024.
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent 1lm training. arXiv preprint arXi::2412.01928, 2024.
Martin Osborne. A course game theory. MIT Press, 1994.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. Xiv preprint arXtt:2304.01904 2023.
11

---

Published as a conference paper at COLM 2025
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. ar arviv preprint arXiv:2407.18211 2024.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiuuing models for assisting human evaluators. arXiv preprint ar arXiv:2206.05802 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint ar arXit:2402.03300 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-866666 06524,866,,440 2023.
Yoav Shoham and Kevin Leowon- Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. Xrv preprint Xit::1712.01811, 2017.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint ar arXiv:2211.00053, 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen I1m applica- tions via multi-agent conversation. arXiv preprint ar arXiv:2308.08111
2023.
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system reasoning in Ilms: Learning how to think with meta thain-of-though. arxio preprint arXiv:2501.0466 004662,,,066...... 2025.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. ar Xiv preprint ar axXio:2312.11456, 2023.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang, Self- rewarding correction for mathematical reasoning. ar ar preprint ar XrXio:2502.19613 2025.
Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, et al. The perfect blend: Redefining rlhf with mixture of judges. ar preprint 2rXi::249..0033 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. ar preprint ar arXii:2409.11212 2024.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2 how to learn from mistakes on grade-school math problems. arXio preprint Ar::::2408.11 162993, 2024.
12

---

Published as a conference paper at COLM 2025
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint (rXio:2408.15240, 2024.
Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via chain-of- oottthghtttss critic. arv preprint Xix:2408.113222 2024.
13

---

Published as a conference paper at COLM 2025
A Algorithms
Algorithm Pair-SFT Data Construction
1: Inputs: Question-answer dataset D X y (x policy model no, number of questions N, number of single-turn rollouts per question K, rule-based solution
correctness reward function r (y. E {0,1}, single-turn sampling set Drjs + {h,
multi-turn correction set D pair + (,, verification message validator f {0,1}
2: for N, do
3: // Solution rollout
4: Sample K solutions for each question x; E X: {y/k- no(+1x;)
5: Label binary reward for each solution y/: o = r (1t.vf))
6: Append to rejection sampling set: Drss Dijs U (x
7:
// Obtain verifications
8:
Choose the best/worst worst-of-N samples: k = arg max, r k = arg min;
9:
if &
or ,k 1 then
10:
continue // All correct or all incorrect solutions
else
*t Vi t- y , ,c_flag +- false, i flag <- false for k 1,..., do
12:
13:
14:
if r. = 0
Di ~ x;,
16:
if ft = 1 then
iflag f-
18:
pair + Paii U (xxv), o, ,y))}
19:
end
20:
else if + = and k & k+ then
21:
Df x,,
22:
if (tt(t) = then
f_llag 1
24:
D pair + Daai U ((
25:
end end if
26:
27:
if if cff_lag and i_flag = then
28:
break
end if 29:
30:
end for
31:
end if
end for
33:
return D pair
14

---

Published as a conference paper at COLM 2025
Algorithm 3 RAFT Message-wise Policy Optimization
1: Inputs:
Question-answer batch D; = X2 3/ = ((z) batch size B, policy model
Ta, number of rollouts per question K, generated trajectory (T } k=1 solution correctness
rewards (=(A))) verification correctness rewards {ri, ke(]]_[))))
2: Choose the best-of-N trajectory for each question based on last solution message:
k
max pál JALL1
3: // Apply constraint
4: Filter out questions with no correct final solution no correct verification, i.e. learning batch
Dleam (xj, r -rt
j i (e (ryt [
اكر iit.44
V vf f jc(B]
5: Perform one gradient update on f with Equations (2) and (3) using D Deam
Algorithm 4 RLOO Message-wise Policy Optimization
1: Inputs:
Question-answer batch D, - X; x }; = 9x batch size B, policy model
Te, number of rollouts per question K, generated trajectory { (
) ) solution correctness
rewards (e))) verification correctness rewards (r kE(K)EE))))
2: // Message-wise advantage
3: for = 1,..., max Liir = rs v do
4: Hij = K [kc[K]
5: UH (* LiEjK) -
11p11)
6: A1x11
"Ha Tj
7: end for
8: Learning batch contains all K samples for each question:
Dieam {xj, TH, (Afk (Ayf (Affr)ictual [e[B]]E]]]]
9: Perform one gradient update on 0 with Equations (2) and (3) using D Pleam
15

---

Published as a conference paper at COLM 2025
B
Experimental setup details
Tasks. We evaluate model on test sets as follows:
MATH500 (Lightman et al., 2023). A dataset of 500 problems selected from the full MATH (Hendrycks al., 2021) evaluation set. This test set spans five difficulty levels and seven subjects, which promotes a comprehensive evaluation of reasoning capabilities.
AMC23. A dataset (AMC12) 2023 (AT-MO, 2023). This test set consists of challenging competition questions intending to evaluate the model's capability to solve complex reasoning problems.
problems from the American Mathematics Contest 12
AIME24. A dataset of 30 problems from the American Invitational Mathematics Examination (AIME) 2024 (AI-MO, 2024). This test set contains difficult questions, with few at AMC level and others drastically more difficult in comparison, aim to access the model's abiblity to perform more intricate math reasoning.
Implementation details. We use the A optimizer with f1 = 0.9, P2 = 0.95, weight decay = 0.1, and a constant learning rate 1.0 x 10 We conduct all training runs on 32 NVIDIA H100 GPUs. We set the global batch size to 2048, and train for 256 steps.
C Extra results
C.1 Verifier reliability
We provide detailed diagnostics for verifier reliability in Table 5. Each confusion matrix corresponds to base model and task pair, with the rows and columns indicating the actual and predicted solution correctness, respectively i.e. diagonal cells represent the true positive (TP) and true negative (TN) rates while the off-diagonal cells represent the false positive (FP) and false negative (FN) rates. We observe the following phenomena:
On easier tasks, the proposer has higher solution accuracy, and the verifier tends to show higher TP&FP and lower TN&FN.
Stronger models that reach higher solution accuracy also have higher TP&FP.
The small model's high verification accuracy attributes largely its higher TN.
Base Model
MATH500
AMC2023
AIME2024
90.2 (266/295)
9.8 (29/295)
81.9 (9/11)
(2.111
(0/1)
(1/1)
Llama-3.1-8B-mm
34.1 (70/205)
65.9 (135/205)
24.1 (7/29)
75.9 (22/29)
(0/299)
100 (29/29)
100 (385/385)
(0/385)
100 (21/21)
(0/21))
85.7 (6/7)
14.3 (1/7)
Llama-3.1770B-nnttter
87.0 (100/115)
13.0 (15/115)
84.2 (16/19)
15.8 (3/19)
82.6 (19/23)
17.4 (4/23)
99.0 (385 /389)
1.0 (4/388)
93.1 (27/29)
6.9 (2/29)
100 (7/7)
(0/7)
Llama-3.--70--nnsrruut
78.4 (87/111)
21.6 (24/111)
72.7 (8/11)
27.3 (3/11)
82.6 (19/23)
17.4 (4/23)
Table 5: Diagnostics for verifier reliability at the first turn across MATH500, AMC2023, and AIME2024 benchmarks.
C.2 Per-turn performance analysis
We provide the per-turn performance statistics for AIME24 and AMC23 in Table 6 and Table 7, respectively. The results are consistent with MATH500 analysis in Table 2. SPOC generally improves or maintains performance on the second solution turns. The smaller model has lower final accuracy yet larger turn-wise improvements, while larger models tend to achieve correct solutions sooner at turn. Moreover, turn-wise corrections occurs less in these two challenging competition benchmarks, as they contain significantly fewer
16

---

Published as a conference paper at COLM 2025
Base Model trained w/ SPOC
Base.
Verif.Acc.@11
Acc-@t1
Acc.w012
A(11,
Ac-ai
3.3
29/30
1/30
2/30
1/30
0/1
1/7
1a-ma--3--70-IInsttee
16.7
10/30
7/30
7/30
0/30
0/1
0/1
1lmma-3.37700--mmss Instrrcct
26.7
11/30
7/30
7/30
0/30
0
0/1
Table 6: Performance across first two solution turns on AIME2024, Ac-si and Aj-to indicate over turns, respectively.
and
Base Model trained w/ SPOC
Aase.Acc.
Verif.Acc.@l!
Acc-0t1
Acc.@12
A(11,12
Ag-11
Ap-
1lama-31-88--nmstt Instruct
22.5
31/40
27.5
32.5
5.0
0/2
2/11
-lama-3.1-70B-nnstte
32.5
24/40
21/40
21/40
0
0
0
-lamm-3.3-700-Instte
57.5
30/40
29/40
28/40
-25
1/2
0/2
Table 7: Performance across first two solution turns on AMC2023. Aç- and Ap- indicate and
ne-er Ao-e--ppp
over turns, respectively.
questions than MATH500. We will include both tables in the appendix of our revised manuscript.
Table 2 presents our per-turn performance analysis over turn] -> 2, where the majority of self-correction occurs, In practice, all finetuned models perform multiple rounds of self-reflection. We hereby present the complete results, where the Table 8 shows the turn + performance of all models, and Table shows the all-turn performance of the 8B model (as the other stopped reflection earlier). Results suggest that the 8B model reaches maximum turns while the 70B models reach a maximum of turns across all 500 evaluation questions. This observation aligns with our discussion in Section 4.2, where stronger models tend to achieve correct solutions sooner. We also observe that the amount of questions requiring additional solutions drops turns, aligning with the looping until verified correctness behavior. Overall, SPOC achieves improvement over turns.
Base Model trained w/ SPOC
Base.Acc.
Verif.Acc.012
Acc.012
Acc.013
A(12
Ar-i
Ap
-lama-31-88-Innstee In---nutt
19/22
0/3
1/18
Llamm-3.1-708-Imn Innuruuctt
65.8
0
77.4
77.4
0
1aama--33-7B-Innstree
75.6
4/24
77.8
77.8
0
Table 8: Performance across solution turns + 3 on MATH500.. Aç- and As indicate Reoas naprr -
and
Ri ag
over turns, respectively.
Turn
Verif.Acc.0t;
Acc.@t;
Acc.@ti+1
A
Ai-t
401/500
59.0
61.0
2.0
8/29
18/79
2
19/22
61.0
61.2
0.2
0/3
1/18
3
6/8
61.2
61.0
-0.2
2/2
1/6
4
2/2
61.0
61.0
0.0
0/2
1/1
61.0
61.0
0.0
0/1
6
0/1
61.0
Table 9: Performance across all solution turns on MATH500 for 1lama-.1-8B-Inntrrue base model. over turns, respectively.
Ap- and i indicate to
Ry-al and
D Preliminaries
CGPO (Xu et al., 2024) is a constrained RL framework that allows for flexible applications of constraints on model generations. Denoting the contraints that the LLM generations need to satisfy as (CI...,,Mll the prompt-generation set that satisfies constraint C,, is defined as Σn (xx,y) X y (x,y) satisfies Cm). The feasible region is defined as
17

---

Published as a conference paper at COLM 2025
the prompt-generation set that satisfies all constraints, ie., Σ = n - In the single-task setting, CGPO solves the constrained optimization problem as follows:
max E Xx~xxm)((s)) [r(x, y)
s.t. IP Pr-XyyTa()) (xx,y) E E) > 0,
KL (TT Fet (x)) I KL
where rxx,y) is the reward function. CGPO is compatible with a wide spectrum of policy optimizers. The RAFT (Dong et al., 2023) algorithm prompts the current policy to generate multiple responses for each prompt, and the best-of-N (BoN) response is used to perform one-step SFT update on the policy.
18

---

Published as a conference paper at COLM 2025
E Prompts
Llama 3.1 COT query template
User:
Solve the following math problem efficiently and clearly:
For simple problems (2 steps or fewer):
Provide concise solution with minimal explanation.
For complex problems (3 steps more):
Use this step-by-step format:
## Step 1: [Concise description]
[Brief explanation and calculations]
## Step 2: [Concise description]
[Brief explanation and calculations]
Regardless of the approach, always conclude with:
Therefore, the final answer is: $\ (boxed(answerrs. hope it is correct.
Where [answer] is just the final number or expression that solves the problem.
Problem: {( Question }}
Figure Llama 3.1 COT query template (Dubey et al., 2024).
Simple COT query template
User:
Please reason step by step, and put your final answer within Wboxed(). Question: {( Question }}
Figure 5: Simple COT query template (Guo et al., 2025).
19

---

Published as a conference paper at COLM 2025
Instance reflection query template
User:
Compare a correct solution and potential solution to the same math question, identify the error in the potential solution, if any, and briefly explain the error in two sentences or less in the <reflection> </reflection> tags, and then conclude with:
Verification: the previous solution correct? (Yes/No)
The explanation should "I think the solution is correct. if the potential solution is correct.
DO NOT solve the question. DO NOT include phrases implying the comparison between the two solutions.
Question: {{ Question }}
Correct solution: {{ correct_solution }}
Potential solution: {{ potential_solution }}
Figure 6: Instance reflection query template.
SPOC simple COT query template
User:
Please reason step by step, and put your final answer within \booxed)).
After each solution attempt, reflect on its correctness within <reflection> </reflection> tags.
Your reflection should first concisely evaluate the previous solution, and then conclude with:
Verification: Is the previous solution correct? (Yes/No)
If the verification is "No", rewrite the solution in separate attempt, the correcting the error or choosing a different approach altogether.
Question: {{ Question }}
Figure 7: SPOC simple COT query template.
20

---

Published as a conference paper at COLM 2025
Self-Refine w /o oracle query template
User:
There might be an error in the solution above because of lack of understanding of the question.
Please correct the error, if any, and rewrite the solution.
Be sure to apply the given format and conclude with: "Therefore, the final answer is: $1 a
Figure 8: Self-Refine w/o oracle query template (Madaan et al., 2023).
Self-Refine w/ oracle query template
User:
There is an error in the solution above because of lack of understanding of the question. Please correct the error and rewrite the solution. Ensure you use the information from past attempts. If you arrive at solution you have already had, the answer is incorrect once again, so take that into account and retry necessary.
Be sure to apply the given format and conclude with: "Therefore, the final answer is: $1 bbboxee((anwwer)s..
Figure 9: Self-Refine w/ oracle query template (Madaan et al., 2023).
Example response
We present example responses SPOC finetuned on Llama-3.1-70--Instrud on MATH500.
SPOC sample response
Problem info:
Problem: Positive integers $as, sb$, and $2809s, with $a<b<2009s, form geometric sequence with an integer ratio. What $a$?
Answer: 41
Subject: Algebra
Level:
Unique_id: test/algebra//91.jja
User:
Please reason step by step, and put your final answer within Mboxed([)
After each solution attempt, reflect its correctness within <reflection> </reflection> tags.
Your reflection should first concisely evaluate the previous solution, and then conclude with:
Verification: Is the previous solution correct? (Yes/No)
If the verification is "No", rewrite the solution in a separate attempt, either correcting the error or choosing a different approach altogether.
Question: Positive integers $a$, $b$, and $2009s, with $acb<2009s, form geometric sequence with an integer ratio. What $a$?
21

---

must both be integers and Published as a conference paper at COLM 2025
Assistant:
## Step 1: Understand the problem and recall the formula for geometric sequence. A geometric sequence is defined by the formula V( ar*2, Aldots, ar<[n=1] V) where )) the first term and \ V) the common ratio. In this problem, we have (( a, b, 2009 \) as the terms of the geometric sequence, with \ < b 2009 V), Therefore, we can write \( b-b = ar \) and \( 2009 = ar*2 \).
## Step Express ( 1) in terms of a 1) and \rr V) and 2009 \( 1) and Ar v) to find the relationship between V( 1) and A V).
terms of Given b ar 1) and \( 2009 = ar*2 V), we can see that \( \(r= \frac(b)(a) V) and M( r=2 \frac(2009{(a) V). Since \( (r 1) is an integer, M( (rr2 1) must also be an integer, which means V( \frac{2009]{a} \) must be integer. Therefore, A( a \) must be factor of 2009.
Find the factors of 2009 to determine possible values for \( a V). To find the factors of 2089, we can start by checking for its prime factorization. 2009 is prime number, which means its only factors 1 and 2009. However, since \ < 2009 V), A( 1) cannot be 2009. Thus, the only possible value for \( a 1, but we need to verify if this satisfies the condition of forming a geometric sequence with an integer ratio.
## Step 3:
## Step Verify if a( 1) satisfies the conditions of the problem.
If a =1 V), then \(b = \cdot r r \) V( 2009 ar*2 = Acdot r^2 r*2 V). This implies \( r22 2009 V), but since 2009 is not a perfect square (it is prime number), \(r cannot be an integer. Therefore, \( = V) does not satisfy the condition of having integer ratio \( V). We need to reconsider based the prime factorization of 2009 and the properties of geometric sequences.
approach
#ff Step 5: Reconsider the prime factorization of 2009 and its implications for \( \) and V( \).
Given that 2009 is prime number, for V( 2009 \) form geometric sequence with \(
< 2009 V) an integer ratio A V), we must have \ ar \) and \( 2009 ar*2 \). Since A( r \) is an integer and \( 2009 = ar \), A( r*2 \) must divide 2009. However, since 2009 is prime, the only integer divisors of 2009 are and 2009 itself. This means \( r~2 = 2009 \) is not possible for any integer \( since 2009 is not perfect square. The only other option is \( r*2 \(r=1 \) or -(r=11 )) but V< r 1 V) would not give geometric sequence with distinct terms, and \(r== \) would not satisfy \( ((ab < 2009 \) since multiplying by -1 would change the arder of the terms. We need to think about the factors of 2009 in different way, considering that \( \) and A \(
V), which gives
2009 \).
#tf Step 6: Consider the factorization of 2009 in the context of geometric sequences. Since 2009 is prime number, for \< b, 2009 \) to be in geometric sequence, to think about how \( \) \( \) can be chosen such that b ar \) and A( 2009 ar*2 \) are both integers and knowing 2009 is prime, we realize that for \( r*2 \) to divide 2009, A r*2 \ must be 1 (since 2009 is prime and has other divisors besides and itself). This implies \(r= \) or = V), but discussed, A( ( does not yield geometric sequence with distinct terms, condition \( b 2009 \) because multiplying by -1 would reverse the order. We must reconsider the relationship between \( )), \ b V), and 2009 in terms of their prime factorization and the properties of geometric sequences.
V(
2809 \). Given 2009 ar*2 V),
A( -1 V) does not satisfy the
## Step Apply correct relationship between \ V) A V), and 2009 considering the geometric sequence and integer ratio. Given \ bb ar \) and \( 2009 = ar \), dividing the second equation by the first gives \( \frac{2009](b} of 2009.
V). Since A( is an integer, N( b \) must be factor
The factors of 2009 are 1 and 2009 (since 2009 is prime), but V V) cannot be 2009 because \( 2009 \). Thus, A( b = \) is not possible since A( a V), implying \( a would have to be less than 1, which is not possible for positive integers. We
22

---

Published as a conference paper at COLM 2025
need to find factor of 2009 that fits the geometric sequence criteria. However, analysis has been misguides ded because overlooked the simplest factorization that allows for geometric sequence with integer ratio. Let's correct this by directly applying the condition that a( b, 2809 \) form a geometric sequence with \( \) being an integer.
## Step Correctly apply the geometric sequence formula and the fact that 2009 is a prime number.
Since \( 2009 = ar*2 \) and V \) is an integer, we look for integer factors of 2009. However, since 2009 is prime, its only factors are and 2009. For V( a, b, 2009 \) to be in geometric sequence, is an integer and V( < 2009 V), examine the possibility of
must have \(b = \) and V 2009 ar*2 V). Given \
2009 v) and this does not because V( b \) would then be 2809, violating \( b 2009 v) The correct approach involves recognizing that since 2009 is prime, the only way A( a, b, 2009 \) can form geometric sequence with integer V(r V) is if A(r \) itself is factor of 2009, but since 2009 is prime, \(r must effectively be 1 2009 for the sequence to hold with integer values. However, \( (rr11 V) does not yield a geometric sequence with distinct terms, and A( ( 2009 \) does not fit the \( < b < 2009 1) condition. We need to think about the factors of 2009 in a way that allows for geometric sequence with \( a < 2009 V) and A< being an integer.
A( 1),
## Step Re-evaluate the mions for \( 1), M(b V), and 2009 to form geonetric sequence.
Given \( b ar \) and \( 2009 = ar*2 V), and knowing 2009 is prime, for \ (rr) to be an integer, \( \) must be such that A( ar 2009 \) holds true with \( r V) being or 2009 \) is not
integer. Since 2009 prime, V( r*2 V) must divide 2009, implying \ r^2 2009. \(r2 r*2 possible for integer \(r since 2009 is not perfect square. Thus, V( r+2 V), giving M r 1 \) \(r= )). However, A< r = \) does not yield geometric sequence with distinct terms, and > would imply \bb= \), which does not fit ( a 2809 \) for positive integers. We must consider nature of geometric sequences and prime numbers to find \( a )).
\rr22 2009 \) since these are the only divisors
#tll Step 10: Apply the properties of geometric sequences and prime numbers correctly. For \( a, b, 2009 \) to be a geometric sequence with integer V( V), and given V r \) such that V( 2009 ar*2 \) and \ ar V). Since \( r \) must be an integer and 2009 is prime, \ r \) must be a factor of 2009, \) (since 2009 is prime and has other square factors). This N( r===-- 1), but \( (r== \) does not give geometric sequence
is
prime, we look for A( a \)
which means (( r-2
= > with distinct terms, and \( ((==-) \) does not the condition \( a b 2009 \) for positive \( \) and \(b \). We need to think about the prime factorization of 2009 and how it applies to geometric sequences. Given 2009 is prime, for \( b, 2009 \) to form geometric sequence, \ must be 1, and \ \) must be such that M r*2 2009 \), but since 2009 not perfect square, this approach flawed. basic properties geonetric sequences
implies
reconsider the given condition tiontioos.
## Step 11: Correctly identify a(a based on the geometric sequence and prime number conditions.
Given N( a, b, 2009 \) form geometric sequence with integer V( V), and 2009 prime, we have \ ar \) and \( 2809 = ar*2 V). Since \ r \) is an integer and 2009 is prime, X( r*2 \) must divide 2009, implying \ r*2 = \) or \( r*2 2009 V). Since \( r*2 2009 \) is not possible for integer \( \) (2009 is not perfect square), we consider r~2 )), giving X< =(r=1 \) or = V). However, >rr=1 1 does not yield a geometric sequence with distinct terms, and \ ((r-11 \( b < 2009 \) for positive integers. The error lies in assuming A r*2 V) must directly divide 2009; instead, we should consider how ((a \) and \( relate through the geometric sequence formula. Since 2009 is prime, and \( a, b, 2009 \) are in geometric sequence, we look for \( a \) and V( b = ar V), with \( \) being an integer. Given 2009's primality, A \) must be for \(r22 \) to be an integer factor of 2009, but since 2009 is not perfect square, V). We need to reconsider the relationship
does not fit
such that \( 2009 ar \) and
X \) cannot be an integer if \( )), and 2009 in the context of geometric sequences and pr prim
between \( a V), numbers.
23

---

Published as a conference paper at COLM 2025
#i Step 12: Finalize the correct approach for finding A( V).
Given A( 2009 form geometric sequence with A( V) as an integer, and 2009 is prime, we must find A( a \) such that V( = ar v) and (( 2009 ar )). Since 2009 is prime, for \( r-2 \) to divide 2009, r(r22 1) must be (since 2009 is not a perfect square). This implies M( r = \) or A( ((r=l) V), but A( r \) does not yield geometric sequence with distinct terms, and V( [(r=11 v does not fit M( < 2009 \) positive V(
r^2
\( )). The key insight prime, \( \) must be such that \( ar 2009 \) holds with V( \) being an integer. Given \( r*2 = \) is the only integer square factor of 2009 (since 2009 is not perfect square), must consider how \( 1) and Ar relate through the geometric sequence. If positive integers. Thus, we need think about the factors of 2009 a way that allows for geometric sequence. Since 2009 is prime, the only factors are 1 and 2009, implying M \) could be if V( r-2 2009 V), but \(r cannot be an integer in this case. The correct approach involves recognizing that A( must be factor of 2009 such that for V( r \) to be an integer, given M( r-2 \) as the only viable integer square factor of 2009.
recognizing that since 2009 is
V), then \ b
V) which does not fit
2009 for
A( v) can be an integer, but since 2009 is prime, \( a V) must effectively
Therefore, the final answer is: \boxed( [<moomiddl
<reflection>
The error lies in the incorrect assumption that \( r*2 v) must directly divide 2009, leading to flawed analysis of possible values for A( \) and \( V). This mistake results in an incorrect conclusion that A( \) must be 1, when in fact, \ a \) should be derived from considering the factors of 2009 and the properties geometric sequences with integer ratios.
Verification: Is the previous solution correct? No </reflectionx<(emmiddl>
Understand the problem em and recall the formula for a geometric sequence. A geometric sequence is defined by the formula M(a_n a_1 \cdot r'((n-)))), where M(an\)) is the nth term the sequence, M(a1\) the first term, \(r)) the common ratio, and \(n)) the term number. For the given problem, we have \(a_1 al), = b)), and ((a_3 = 2009)). Therefore, we can write the equations \(b = arl) and \(2099 ar*
#ff Step 1:
## Step 2: Express \(b) in terms of M(a) \(rV), \(20091) terms \(a)) \(r)).
From the geometric sequence formula, we have \(b = ar\) and \(2009 = ar*
## Step Solve for M(r) using the information given.
Dividing the equation \(2009 ar*2\) by \(b ar\), we get \(\frac(2009)((b) r\). Since \(b ar\), we can substitute \(b) in the equation to get \(\frac(2009)(ar) r\), which simplifies to \(2009 ar*2)). We already know this, so let's directly use the relationship between M(a)), \(b)), and ((2009)) to find \(r)). Given \(b = ar\) and \(2099 2r),)), dividing these equations gives \(r \frac(2809){b))).
## Step 4: Determine the possible values of \(r)) and \(al). Since \(a\), \(b)), and ((009)) form geometric sequence with \(a < b 20091), and \(r) integer, need find integer factors of ((2099)) that could serve as M(N). The prime factorization of (20091) \(7^2 \edot 411). Thus, possible values for \(r)) could be \(1), \(7V), (441)), or (491), considering the factors of ((2099)). However, since \(a < 20091), \(rl) must be greater than \(T)).
## Step Test possible values of \(rl) find \(a\).
Let's test \(r = 7\): If \(r = 7V), then \(2009 = Nedot 7*2)), which gives Ma = \frac((009)(49))). Calculating this, \(a = 411), which is an integer and satisfies \(a < < 20091).
Let's test \(r 411): If 41V) then \(2099 \cdot 41^21), which does not yield an integer \(a)) since \(4122 1681)) and \(20e9 16811) not an integer. Let's test \(r 491): If \(r 491), then ((2089 Nedot 49*2N) does not apply since
24

---

Published as a conference paper at COLM 2025
\(49*2 = 24011), which is larger than \(2009\), making it impossible for \(a\) to be integer this context.
## Step 6: Verify the solution.
Given V(r = 7V) and V(a 411), we find \(b ar = 41 Acdot 7 = 287\). Thus, the sequence is \(41, 287, 2009)), which satisfies the given condi tions.
The final answer is: $1boxed(4133</eom_ddd)
<reflection>
think the solution correct, AnnVVerfiiaation: the previous solution correct? Yes </reflecctonxx(ot_iddl
25


Reviews:
[{"review_id": "jQXvuedb4q", "reviewer": "Reviewer_Vfpt", "summary": "This paper proposes SPOC, a training method that allows LLMs to interleave solution generation and verification in order to improve math reasoning abilities of an array of SLMs. The paper differs from prior work as it interleaves the model's ability to generate solutions and perform self-verification in the same reasoning trace, instead of separate prompts. Their method combines pair-wise SFT strategy to enable multi-turn (solution and verification) generation, followed by an online RL training phase (RAFT or RLOO).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is fairly well written and easy to follow\n- Empirically strong results and improvements on math reasoning tasks with a variety of SLMs show that their methodology is effective", "reasons_to_reject": "- The paper lacks a GRPO baseline which would test the model could learn to interleave solving and verifying with online RL on the same data -- without which it is hard to gauge their method's effectiveness.\n- The main contribution is *not* properly placed in recent literature, specifically, it is unclear how SPOC compares to other long CoT methods, which have also demonstrated interleaved solving, backtracking, and verification in longer CoT generations (see below):\n     - https://arxiv.org/abs/2503.01307\n     - https://arxiv.org/abs/2504.07128", "questions_to_authors": "See weaknesses", "extra_scores": {}}, {"review_id": "hgId6IUX3J", "reviewer": "Reviewer_Cxxp", "summary": "The paper proposes SPOC (Spontaneous Self-Correction), a training-and-decoding framework that interleaves a solution proposer and a verifier within a single inference pass. The authors first generate synthetic “solution + verification” pairs (Pair-SFT) to teach the model this format, and then apply online reinforcement learning with a reward that jointly incentivizes correct solutions and correct verifications. Experiments on three math-reasoning benchmarks (MATH500, AMC23, AIME24) across multiple Llama-3 and DeepSeek models show consistent accuracy gains.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Clarity and Organization.** The writing is clear and easy to follow, and the paper’s structure is well organized.\n\n**Integrated Self-Correction.** SPOC embeds solution generation and verification into a single inference run, eliminating external prompt loops and simplifying deployment.\n\n**Clear Two-Stage Pipeline.** The synthetic Pair-SFT stage followed by online RL is precisely defined and delivers substantial accuracy gains across both 8B and 70B models on multiple benchmarks.", "reasons_to_reject": "**Limited Per-Turn Analysis.** Table 2 provides a detailed turn-by-turn breakdown only for MATH500; the same analysis is missing for AMC23 and AIME24, making it difficult to assess SPOC’s effectiveness across all datasets.\n\n**High Computational Cost.** The online RL phase requires 32 × H100 GPUs (Appendix B), which is prohibitively expensive for most research groups and small teams.\n\n**Incomplete Verifier Reliability Reporting.** Although the paper reports a single Verif.Acc.@t1 metric (e.g., approximately 80 % on MATH500), it does not provide deeper diagnostics such as false-positive/false-negative rates, confusion matrices, or robustness to subtly incorrect justifications, leaving the verifier’s reliability only partially evaluated.", "questions_to_authors": "1. Could you provide a per-turn performance analysis, similar to Table 2 for MATH500, for the AMC23 and AIME24 benchmarks to better understand SPOC's generalizability?\n\n2. Beyond the reported Verif.Acc.@t1, were the verifier's false positive and false negative rates measured? Additionally, was the verifier's behavior on slightly incorrect explanations examined, and could you share any related statistics such as a confusion matrix or an error-type breakdown?", "extra_scores": {}}, {"review_id": "4g6OxUAMtG", "reviewer": "Reviewer_VrYp", "summary": "The paper proposes SPOC (Spontaneous Self-Correction), a pipeline that teaches a single model to alternate between proposer and verifier roles within one forward pass. Synthetic “solution + verification” dialogues are first generated with Pair-SFT; the policy is then refined via message-level RL (RAFT or RLOO) using a strict 0/1 reward that requires both a correct answer and a validated proof.\n\nThe writing is generally clear and the figures convey the workflow well. Ablations on reward design and iteration depth help justify key choices. Still, section 3 is notation-heavy for a paper without formal theory, trimming some symbols or moving details to an appendix would improve readability.\n\nEmpirically, SPOC consistently lifts several Llama-family backbones and their DeepSeek-distilled variants on MATH500, AMC 2023 and AIME 2024. To strengthen the method's applicability to other model families, it would be helpful to show results on a different architecture (e.g., Qwen 2.5) or at least discuss why the method should transfer beyond Llama, especially that Qwen has been a popular base model to train for long CoT and reflective reasoning.\n\nIn terms of related work discussion, [1-2] are very relevant in terms of relying on the conversations of multiple role-playing LLMs (proposer, verifier, ect.) for synthetic data curation and further distillation. In the attempts to reproduce o1 thinking, there have also been works discussion on how to construct long CoTs (self-reflection, ect.) for SFT and later RL [3]. The concurrent line of work on distillation is also highly relevant [4-5], though I understand that these are quite recent. \n\nIn general, the manuscript would benefit from clarifying what is the main advantage of SPOC. Moreover, multi-agent data generation causes extra latency as compared to distillation from a strong model or simply forcing reflective keyword such as \"wait\" during rollouts or inference time for data collection.\n\n[1] Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning\n\n[2] Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation\n\n[3] O1 Replication Journey: A Strategic Progress Report – Part 1\n\n[4] s1: Simple test-time scaling\n\n[5] LIMO: Less is More for Reasoning", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, the work is technically solid and the empirical results showed notable improvements. \n- The paper is mostly well written and its diagrams easy to follow. Ablation studies provide interesting insights.", "reasons_to_reject": "- The experiments are done within the same model line (Llama).\n- The concept of first curating data with reflective reasoning behaviors and distilling it via SFT or RL is not quite new. I'd be curious to learn the unique finding of this paper.\n- Multi-turn (agent) conversation naturally has a high latency cost as compared to direct distillation from strong models or simply forcing reflections by appending words like \"wait\" or \"let me double check\" at inference time (or rollouts) for data construction. It's not clear why or on what scenario would this approach be more beneficial.\n- Math notations are heavy and some unnecessary, given that the paper does not present theoretical analysis.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x6lheARohv", "reviewer": "Reviewer_MWdF", "summary": "This paper introduces SPOC (Spontaneous Self-Correction), a novel approach for enhancing mathematical reasoning in large language models (LLMs). Unlike existing self-correction methods that rely on external prompts or post-generation refinement, SPOC enables LLMs to interleave solution generation and verification in a single inference pass, dynamically terminating based on verification outcomes. By framing the process as a multi-agent interaction (solution proposer and verifier) within the same model, SPOC uses synthetic data for fine-tuning and online reinforcement learning (RL) with rewards for both solution correctness and verification accuracy. Experiments on benchmarks like MATH500, AMC23, and AIME24 show significant improvements: Llama-3.1-8B/70B achieve gains of 8.8%/11.6% on MATH500, 10.0%/20.0% on AMC23, and 3.3%/6.7% on AIME24, respectively. The RLOO RL variant further boosts performance, reaching 94.6% accuracy on DeepSeek-R1-Distill-Llama-70B for MATH500.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- SPOC outperforms baseline methods across model sizes and task difficulties\n- By modeling the LLM as both a solution proposer and verifier, SPOC introduces a self-play training strategy that avoids the need for separate models or stronger \"teacher\" supervision", "reasons_to_reject": "- SPOC with RLOO performs much better than RAFT, so it makes more sense to use RLOO as the default RL algorithm and compare it with RLOO as the baseline.\n- SPOC relies on synthetic data generated by the base model for fine-tuning, which may introduce bias if the initial model has systematic errors.\n- SPOC only integrates with RAFT and RLOO. It is not clear whether it will be compatible with other RL algorithms.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "0hD3eQ3iMe", "reviewer": "Reviewer_VSGv", "summary": "The paper argues that the existing methods rely on external system prompts for verification and solution generation to perform self-correction, which may be cumbersome to deploy. To this end, the paper proposes SPOC, a technique to allow the model to perform self-correction adaptively by acting as a solution generator and verifier in a single inference call. Specifically, the model is first trained with paired SFT data which motivates the model to perform multi-turn generation-verification. Subsequently, the SPOC performs RL-tuning to explore diverse scenarios on the fly and learn from the oracle rewards (e.g., solution correctness and Yes/No correctness for verifier). Interestingly, the model policy is optimized for both solution generation and verification jointly. The paper achieves large gains over the baseline models across diverse math datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper highlights an important problem that closed loop self-verification/correction can be cumbersome at inference due to heavy reliance on different prompting or allocation of multiple models to act as solution/verification generators. \n- The proposed solution of performing self-correction in the single inference call is interesting and allows the model to make decisions adaptively. \n- The performance improvements over the relevant baselines is quite strong across different benchmarks.", "reasons_to_reject": "- While the presentation and theory in the paper is general-purpose, the experiments were performed with a lesser number of self-correction turns. For instance, the D_pair (Line 186) data indicates that the model is taught to self-correct within one-round (question, wrong answer, verification, right answer). In reality, you might need multiple rounds of self-correction. Table 2 also suggests that there is only one-round of self-reflection in the experiments. However, Figure 2 (Right) indicates that there are multiple rounds of self-correction where an example goes through three rounds of solution generation till the final verification is correct. It is confusing how the model is generating more rounds in the RL stage than it has been taught in the SFT stage. \n- It is unclear how the method is required in the light of thinking models that can perform self-verification/self-correction before generating the final answer. The introduction never touches on this aspect. We observe that the authors show that SPOC works on top of Deepseek-Distill models too so it will be helpful to have more nuanced discussion on why self-correction is needed for thinking models too. Related to the above, it is not clear how well the method performs beyond one-round of self-reflection. In contrast, the thinking models can perform multiple rounds of self-verification, backtracking, alternative solutions, and reflection in their reasoning traces. \n\n[1] https://arxiv.org/abs/2501.12948", "questions_to_authors": "- In Line 38, the authors mention that lack of adaptive self-reflection leads to “ineffective test-time scaling”. This statement does not make a lot of sense since prior work has studied test-time scaling in traditional self-reflection too [1]. \n- In Table 1, the LLama-8B-Instruct Self-Refine w/ Oracle achieves better performance than SPOC on AMC-23 but it is not highlighted. Feel free to fix the typo.\n- I think the paper presentation requires a lot more work. For instance, Figure 3 presented in Page 4 is not relevant uptill the experiment in page 9 which is quite confusing. \n\n[1] https://arxiv.org/abs/2501.19306", "extra_scores": {}}]

Author rebuttal:

Meta-review: The study proposes SPOC, which is a process that allows an LLM to generate interleaved solution+verification messages in a single forward pass. This is trained using message-level rewards via RAFT/RLOO. The reviews were mixed for this submission, trending negative, with several concerns and strengths raised. The main ones I'll highlight that were common across multiple reviews are:

Strengths:

* Strong results across multiple math datasets showing the effectiveness of SPOC.
* Technical novelty of "closed-loop"/single-pass generation and verification.

Weaknesses:

* More RL baselines (such GRPO) in addition to RLOO
* Some presentation details were not clear, or mathematical notation is too heavy.

Additional concerns or strengths that are not as pertinent to the scientific contribution of SPOC include and were either addressed by the authors or only raised as a concern by one reviewer include: missing related work (addressed in rebuttal), concerns about the number of turns in data/analysis (addressed in rebuttals), limitation to Llama-derived models instead of Qwen-series, computational cost, or using synthetic data for fine-tuning. 

Ultimately, I'm convinced the contribution and novelty outweighs missing RL-related baselines, especially since RLOO and GRPO are similar methods and +RLOO is not only contribution in this work (although that is what leads to the best performance; the conclusions would not be different if +GRPO was the winner): This paper can be presented at COLM.
Decision: Accept