Reinforce LLM Reasoning through Multi-Agent Reflection
Authors: Yurun Yuan
Tengyang Xie
Venue: ICML 2025 poster (2025)
Keywords: Post-training
LLM-based multi-agents
Reinforcement learning
Mathematical reasoning
Primary area: deep_learning->large_language_models
TL;DR: This paper introduces DPSDP, a reinforcement learning algorithm that trains an actor-critic LLM system to iteratively refine answers via direct preference learning on self-generated data.

Abstract: Leveraging more test-time computation has proven to be an effective way to boost the reasoning capabilities of large language models (LLMs). Among various methods, the verify-and-improve paradigm stands out for enabling dynamic solution exploration and feedback incorporation. However, existing approaches often suffer from restricted feedback spaces and lack of coordinated training of different parties, leading to suboptimal performance. To address this, we model this multi-turn refinement process as a Markov Decision Process and introduce DPSDP (**D**irect **P**olicy **S**earch by **D**ynamic **P**rogramming), a reinforcement learning algorithm that trains an actor-critic LLM system to iteratively refine answers via direct preference learning on self-generated data. Theoretically, DPSDP can match the performance of any policy within the training distribution. Empirically, we instantiate DPSDP with various base models and show improvements on both in- and out-of-distribution benchmarks. For example, on benchmark MATH 500, majority voting over five refinement steps increases first-turn accuracy from 58.2% to 63.2% with Ministral-based models. An ablation study further confirms the benefits of multi-agent collaboration and out-of-distribution generalization.

Reinforce LLM Reasoning through Multi-Agent Reflection
Yurun Yuan Tengyang Xie
Abstract
ing additional computation inference time. Mechanisms such as best-of-N sampling (Charniak and Johnson, 2005; Stiennon et al., 2020; Chow et al., 2024), self-consistency
Leveraging more test-time computation has proven to be effective way to boost the rea- soning capabilities of large language models (LLMs). Among various methods, the verify- and and-improve paradigm stands out for enabling dynamic solution exploration and feedback in- corporation. However, existing approaches of- ten suffer from restricted feedback spaces and lack of coordinated training of different parties, leading to suboptimal performance. To address this, we model this multi-turn refinement pro- cess as Markov Decision Process and intro- duce DPSDP (Direct Policy Search by Dynamic Programming), a reinforcement learning algo- rithm that trains an actor-critic L.LM system to iteratively refine answers via direct preference learning self-generated data. Theoretically, DPSDP can match the performance of any pol- icy within the training distribution. Empirically, we instantiate DPSDP with various base models and show improvements on both in- and out-of- distribution benchmarks. For example, on bench- mark MATH 500, majority voting over five re- finement steps increases first-turn accuracy from 58.2% to 63.2% with Ministral-basee models. An ablation study further confirms the benefits of multi-agent collaboration and -fist--distiiuttion generalization.
(Wang et al., 2022), explicit reasoning (Wei et al., 2022;
Yao et al., 2023; OpenAI, 2024), and refinement through critique and revision (Qu et al., 2024; Kumar et al., 2024) exemplify this approach.
Among the various test-time scaling methods, the verify- and-improve paradigm offers distinct advantage by en- abling interaction with external environments to explore the solution space and incorporate feedback into response gen- eration. For instance, when faced with tasks like using an unfamiliar coding library that is not covered in the LLM agent's knowledge. the LLM agent needs to compose trial programs and interact with the compiler and tests to refine its attempts. In contrast, methods like chain-of-thoughe and self-consistency remain confined to the knowledge already embedded within the LLM, limiting their adaptability in dynamically evolving tasks.
Various prior works have explored methods to enhance LLMs' ability to refine their responses. One line of re- search focuses on enabling LLMs correct their own er- rors by extracting and utilizing latent knowledge that was not effectively applied in the initial attempt (Kumar et al., 2024; Qu et al., 2024), despite the findings from other works that LLMs is unable to reliably correct their errors without external information (Kamoi et al., 2024; Huang et al., 2024). Another line of research introduces exter- nal feedback mechanisms to guide refinement, leveraging resources such as compilers, external tools, and verifier models (Welleck et al., 2022; Havrilla et al., 2024; Chen et al., 2024a; Shinn et al., 2023). By incorporating exter- nal feedback, L.MMs gain access to new information that can improve their prior responses. Nonetheless, these ap- proaches face limitations such as: (1) restricted feedback space (e.g., compiler messages in code generation tasks or output of fixed set of tools for tool-assisted refinement), and (2) the absence of joint training processes among LLM agents and feedback providers, resulting in suboptimal in- teraction between L.MMs and their environments.
1. Introduction
Large language models (LLMs) have shown strong capabil- ities in solving reasoning tasks such mathematical prob- lems and coding (Lozhkov al., 2024; Shao al., 2024; Team et al., 2024). A series of effective methods to im- prove the reasoning performance of L.MSs involve leverag-
'epartment of Computer Sciences, University of Wis- consin-Madison, Madison, WI, USA. Correspondence Yurun Yuan yyurun-yuane wwwn@ec..iis..duu Tengyang Xie <tx <tx@esswisc.coo wd.....sc.duu>
Recent progress in multi-agent systems have shown great promise across various tasks, highlighting new opportuni- ties for response improvement (Guo et al., 2024b; Mot-
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reinforce LLM Reasoning through Multi-Agent Reflection
wani et al., 2024). Instead of treating feedback resources as fixed, multi-agent systems incorporate different parties into the training-time optimization process, enabling better coordination among agents. Building on this foundation, we propose DPSDP: Direct Policy Search by Dynamic Programming, an RL algorithm designed to train multi- agent LLM system to iteratively improve its responses for reasoning tasks with self-generated data. DPSDP intro- duces actor model that generates and refines responses over multiple turns, guided by feedback from critic model at each turn. This approach enables broad and flexible feedback space by utilizing the diverse and dynamic re- sponses generated by LLM agents. Furthermore, the joint training process optimizes the collaboration between the actor and critic, hamessiin the strengths of both models to achieve more effective response refinement.
induces greater distribution shift and leads to degraded per- formance.
2. Problem Setup and Preliminaries
Consider a scenario involving two agents: an actor Ta- which generates answers to questions, and a critic Tes which provides feedback on the actor's responses help refine them. Let Prob a;); represent a dataset of N questions and corresponding ground truth answers, with n, a' ~ Drrob denoting a question posed by hu- man user along with its correct answer. When the actor receives the question x, it generates an initial response dg ~ Fal- æ). The critic then evaluates this response, providing feedback di ~ =T(-- [ar, aol) that identifies po- tential errors or inaccuracies. Using this feedback, the ac- tor refines its response, generating 02 ~ TT(- [xx, Go aıl). This critique-and-refine process can be repeated multiple times, enabling the actor to iteratively improve its answers. The final output is determined by majority voting across all answers. We illustrate this process in Figure 1.
Contribution Our contributions are three-fold. First, we introduce DPSDP, an RL algorithm that enables LLMs to iteratively refine responses with collaboration. We for- mulate the multi-turn improvement process as a Markov Decision Process (MDP) and design a direct preference learning algorithm to teach LLMs from self self-generated data. Furthermore, we theoretically prove that the policy pro- duced by DPSDP competes with any policy under single- policy concentrability and bounded in-distribution gener- alization error. Finally, demonstrate the effectiveness of our algorithm by instantiating our method across vari- ous model families, including Ministral (Mistral AI team, 2024), Llama-3.1 (Grattafiori et al., 2024), and Qwen2.5 (Yang et al., 2024a), and evaluating their performance on multiple benchmarks. Specifically, by sampling five se- quential answers on problems from MATH 500, Ministral- based models improve their first-turn accuracy from 58.2% to 63.2%, Llama-3.1 1-asee models from 55.8% to 58.4%, and Qwen2.5- also show that our models generalize effectively to out- of-distribution benchmarks, from wrade-school-lever prob- lems to challenging Olympiad-lever benchmarks.
Cast to a Markov Decision Process (MDP) This multi- turn conversation can be naturally formulated as a stan- dard episodic Markov Decision Process (MDP). Let æ = (Ta. Te) represent the joint policy governing the iterative refinement process. The state turn h, denoted as 8%, cap- tures the conversation history up to that point. The actions in this MDP correspond to the responses on generated by either the actor or the critic at each turn, and the policy π determines the action taken given the current state, i.e., On ~ x(- sn). The new state 8/+1 incorporates the new response on and full or partial conversation history from last state. We use deterministic function 0(5%, as) to in- tegrate the new response into the conversation history to produce the next state: 8h+1 = Ö(sn, an). We define the state space at turn h as S₁, and action space as A.. The state distribution over S₁ induced by policy π is represented as dj. If the critique-and-refin process is performed L times, the MDP will have horizon of H 2L + 1, comprised of an initial answer from the actor and following L rounds of feedback from the critic and refinements by the actor. The reward function r(s) incentivizes the actor to produce correct answers. Specifically, a reward is assigned when the actor's response matches the ground truth answer a", defined as r(32141) = I[az; = a*) for i E (0,1,...,44 L}.
models from 60.4% to 62.0%. We
Additionally, we explore the factors that contribute to the strong empirical performance. First, we replicate the train- ing process on a single LLM agent, and our findings indi- cate that while training single LLM to both solve ques- tions and reflect on them can improve reasoning capabili- ties, it struggles with more challenging benchmarks, such as MATH 500 and the Olympiad Bench, highlighting the benefits of specialized LLM agents. Furthermore, we eval- uate the models in non- Markovian settings, where agents have access to the full history of prior refinement iterations. This setup deviates from the MDP defined by our frame- work, which assumes states are based only on the most recent answer. While providing the full conversation his- tory offers richer context, our results show that this setting
We also define the value functions Q7 a) Ex - r(st) | 8h s, = a] and V = (u (x,a)) a)] as the expected undiscounted cumu- lative returns starting at step h for h E [H]. Specifically, we define J(T) as the expected return of the entire trajec- tory and we aim to learn a policy that maximize the number of correct answers: max, [(x) == Exg=d [V (s)), where
2

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Probiem
Actor
Ag
Anrwer
Feedback oi
Answer
Feedback
Critic
RsL
Figure I. Inference time. Given problem az, the Ta generates initial response ag- The critic me then provides feedback di. identifying potential errors in ng- The actor iteratively refines its response based on the feedback, continuing this process for L rounds. Finally, majority voting applied all generated answers determine the final response
dī denotes the distribution over initial states.
KL-regularized objective with respect to Tif Specifically, for each fixed h E {H- 22 .0. we aim obtain policy Th such that
Policy Search by Dynamic Programming (PSDP) PSDP (Bagnell et al., 2003) is classic reinforcement learning algorithm designed to optimize undiscounted re- wards over a fixed horizon. For a non-stationary policy N - where each x² E IT is sta- tionary policy deployed at time step h, the PSDP algorithm shown as in Algorithm iteratively optimizes policies in reverse order from h H 1 to h 0, selecting T² from Il at each step maximize the expected future rewards fol- lowing the policy sequence (T²,, nh+ ...., π -11,, starting with states sampled from a bascline distribution PA. PSDP provides a theoretical performance guarantee, even when the maximization step arg ma can be done only ap- proximately (Bagnell et al., 2003).
-d - oon (110a) [ [Q (on, an)
TB = max
#EA(An)
(I)
- -BDLL [ sn]]]-
:TS =
Here, DKL denotes the Kullback-Leibler divergence and parameter 3 is used to balance the gap to the unregu- larized objective. Prior works show that there exists a closed-from solution to this problem: (a 8x) x Tref sn) exp Q% - (sh, a)). and this objective can be learned in direct learning manner (Rafailov et al., 2023; Rosset al., 2024).
Particularly, we define the cross-entropy loss as
LCE (T, Trefi P.
3. DPSDP: Direct Policy Search by Dynamic Programming
HB ( - Q(s,a
= [ 02
In this section, we develop DPSDP, an RL algorithm in- spired by PSDP, enable LLMs to generate enhanced an- swers via collaboration. We first induce ideal version of our algorithm, consisting of data collection and Q-value learning. We then provide theoretical proof for perfor- mance guarantees. Lastly, we modify the algorithm to ob- tain a practical version which accommodates implementa- tion difficulty and computational efficiency.
a a
Blog T(a' (s)
x(a2 5) neet (a²
-Blog
Tee (a) (8)
where HB 2) -2 log 8 (1- log(1- is the cross entropy of two Bernoulli distributions. With dataset D consisting of (sh, at, a.)), where Sy ~ dft.., of Trer(: $1), E {1,2), 2). we can achieve Eq. (1) via optimizing min, LCE I,, Teff; U[DA], o.th 11) ")) In brief, at each step, DPSDP first samples various responses from Tref- then ob- tains policy that minimizes the cross-entropy loss Eq. (2) over the collected pairwise dataset. The reference policy Tref induces the baseline distributions. We formally outline DPSDP algorithm in Algorithm 1.
3.1. Algorithm Development
We start with a reference policy Tee. which represents LLM agents that are capable of but not specialized in rea- soning or response refinement.
Motivated by PSDP, pected return each turn following the optimized pol- icy from the last iteration, i.c, Tx(- 3%) argmax, 1 (s,, a). However, accurately solving this requires a finite action space, which is unrealistic for a model's s open-ended responses. We alternatively consider
want to maximize the ex-
3.2. Algorithm Analysis
In this section, we analyze the performance of DPSDP un- der appropriate assumptions. We first assume that &ee pro- vides fair coverage of the state distribution of the optimal
3

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Algorithm DPSDP
The policy produced by DPSDP theoretically competes with any policy under single-policy concentrability and bounded in-distribution generalization error. We defer the proof of Theorem I to Appendix C.
Input: horizon H. reference policy Tref, dataset D prob. and
Q-value functions (s, a)
for h = H- - 1, H ,,. do Initialize D + 0.
for (z, a²) E Prob do
3.3. Practical Algorithm
Sample m pairs { (sh: a/. az)) m' where
While Algorithm achieves theoretical performance guarantee, we also aim for practical and efficient imple- mentation. Achieving this requires carefully refining sev- cral key aspects.
Sh ~ (. So = x), a/ ~ Tree(. 8h), az
TTer(- 8p).
Collect the pair: DA t DA U (sh, ane aa)) Im
end for
Let an < min, LCe (T,, reeti U[Da], 0% 2h!tt) where LCE is defined as Eq. (2). end for
Reduced Context and Horizon Generalization Previ- ous works that model multi-turn conversations MDPs maintain the full conversation history at each state, i.e., they define 8(sh, 0%)) concal an] (Qu et al., 2024; Xiong et al., 2024). However, in iterative refinement tasks, the most recent answer and its feedback are more crucial than earlier responses, as the actor no longer needs to ad- dress errors that have already been corrected in previous iterations. Based on this heuristic, we design Ta to re- fine its previous answer while observing only its most re- cent response and the corresponding feedback. Likewise, the critic provides feedback exclusively based on the lat- est answer. Formally, function ò is defined as 0( an) == [x, an. if h is even, else concat [K an..
Let the final policy be = Rg.
Ourput: DPSDP policy म
policy x" and generates more diverse responses than other policies, as formalized in Assumption 1.
Assumption 1 (Coverage). We assume the concentrabil- ity coefficients with respect to space distribution and ac- tion space defined below are bounded, i.e., C.s +00 and CA < +00.
di (sh) C3 = max h,8h df (sh)
CA = max max- T(an sh) Tlll h.89 Teef (ah 8h)
This design simplification not only reduces the model's context length requirements but also enables generaliza- tion longer test-time horizons than train-time horizon. Intuitively, if the actor and critic are trained to effectively improve diverse initial answers, the two-agent system can iteratively refine an answer multiple times, even if it was only trained for a single refinement step. Since the ac- tor and critic only observe the last answer, future refine- ment iterations are not affected by test-time horizon shift. Therefore, our practical version of DPSDP involves only one refinement step at train-time, resulting in compact MDP with horizon H = 3. Our evaluation results in Sec- tion 4.3 demonstrate that DPSDP policies are effective to improve answer over turns regardless of the horizon dif- ference, while the reduction horizon enables easy and efficient implementation.
This ensures the training data generated from Teer is suf- ficiently exploratory, providing an opportunity for DPSDP to learn the optimal actions.
Additionally, we assume that Algorithm 1 maintains a bounded in-distribution generalization error, as stated in Assumption 2.
Assumption 2 (In-distribution reward learning). We as- sume the policy से obtained with Algorithm I satisfies that for any 1,,,
-1,,
a(an 8n) E, oet say aagermeeeeee (-llali blnnn (8los
(an I sn)
log a(ah s8)) (n 8t)
27 )
Qu(s,, Qa QR ah)
Estat-
Estimation of Q-Values Algorithm assumes access the Q-value function **,**(*,,, az), which is impracti- cal. In practice, we approximate the Q-value by rolling out the policy and evaluating the correctness of its an- swers. Particularly, for the refined answer 02, its unbi- ased estimated Q-value is naturally given by its own cor- rectness, 2 (32, 42) r(s3). For the feedback a₁ we use the reference policy Feef as an approximation of î to generate a refined answer al based on a1, and esti- mate the expected return using the correctness of al. ie..
By Lemme C.5 of Xie et al. (2024), we can immediately in- fer that Estat is small with high probability for large dataset.
Under these assumptions, we establish the following per- formance guarantees for DPSDP.
Theorem 1. Under Assumptions I and 2, if we choose 3
/C3C CCCCm then DPSDP policy से satisfies J(x²))
0
log C.A
[())= = VH CCCCAest))
4

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Problem
Annwer
Feedback
Answer
Anser ajv
Feeibook oj
Anmver dix
Answer aj
Feealback af
Ansver agv
DPSDP Data Creation
DPSDP Training
(s, ag (st, 41, oij a.))
(2, a
MOTO
DPO
DPO
Figure 2. Model training. DPSDP first samples a complete trajectory T = (at, ag, 01, 02) from the reference policy Tee-- At each state along this trajectory, it generates responses explore possible answers and feedback. Q-values of these candidate responses then estimated as in Section 3.3 and pairwise preference dataset is extracted for subsequent DPO training on both the actor and critic.
O ()) = Eag (1sa)) (1))) (8{, d-))]- Since we fo- cus on KL-regularized target in Eq. (1), Ther is a proper approximation of T2. and our results Appendix E.3 also supports our approach. For the first-turn answer ap. estimate Q (sp, dn) based on its correctness, i.e., 2 (sg,aa x r(s1) = 1|ag = a*). This approach motivated by three key considerations: (1) in an ideal scenario where the policy from the last iteration is op- timal for all actions after h 0, i.e., मn Q (so, an) Q5" (o), ao) = r(s1) + H-1, fers from r(s1) by only a constant shift. (2) we want to encourage correct first-turn answers, as they serve as the foundation for subsequent refinements. (3) empirical re- sults indicate that only small fraction of first-turn answers change correctness in later turns relative the total num- ber of problems. This suggests a strong positive correlation between r(s1) and Q (so: on). We analyze the impact of Q-value estimation on the performance guarantee in Ap- pendix C.3.
and I are scaled by the same factor. Motivated by this, we amplify the estimated Q-values from their original val- ues, which are either 0 or a positive constant. For each pair (sh, a ax) in Algorithm 1 where the estimated Q-values differ, we relabel the actions as (a aj = {1), a such that 2% @en+1 (sh, a ) > 0 and Q% 2 (sh, (8h,07) = O. We in- sufficiently large positive value
crease 2 (sh,
while leaving 2% (sp, 0g unchanged. By substituting the amplified Q-values into Eq. (2) and leveraging the fact that a (+oc) 1, we find that Eq. (2) can be effectively estimated with a DPO loss:
n², then 2, which dif-
LDPO(M, Redt; D) -B InD
(3)
aaa sn))
3 log कल 81)
log a Blog
That (aj, (sh)
Toat (0 sn))
This significantly simplifies the implementation of DPSDP without compromising performance.
Furthermore, we sample a complete trajectory т (sg, ag; ",,,,,,,,,.............. from Tee for each problem æ and use the resulting states sn as an alternative directly sam- pling from d' The practical implementation of DPSDP is presented in Algorithm 2 and illustrated in Figure 2.
Remarkably, all estimated Q-values can be obtained by rolling out Tref, which effectively eliminates the loop- carried dependencies across different in Algorithm 1. This allows consolidate all pairwise datasets into single dataset, Dx U Dn, and perform single op- timization step.
3.4. Preliminary Training
Reduction of Cross-Entropy Loss to DPO Loss In practice, experiments typically use 3 0.1, which may be significantly larger than the value suggested by The- orem I: According to Lemma C.5 of Xie et al. (2024), Estat is of order O(1nn, where n is the number of sam- ples used to solve the objective, indicating that & should be small. However, it is important to note that the ob- jective in Eq. (1) remains unchanged if both the Qavalues
We find direct adoption of DPSDP on off-the-shelf mod- els leads to inferior improvement, as the base models strug- gle to reflect on prior responses or refine their answers based on feedback. Aligned with prior works (Qu et al., 2024; Xiong et al., 2024), we observe it is beneficial to su- pervised fine-tuning base models on feedback and refined answers generated by a capable model as preliminary
5

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Algorithm 2 Practical DPSDP
back-refinement trajectories and fine-tunes the actor and critic using only trajectories that produce correct final an- swers. During SFT, messages from the other LLM agent are masked: the actor's loss is computed solely from an- swers, while the critic's loss is based only on feedback. (2) STaR-DPO follows the same data collection strategy as STaR but incorporates both correct and incorrect answer trajectories. It applies DPO to train the actor and critic us- ing a pairwise preference dataset derived from self-gener- ated trajectories. Same as STaR, loss computation is re- stricted to the messages of the individual LLM agent, with messages from other agents masked. (3) Oracle-RISE (Qu et al., 2024) samples multiple sequent answers under the assumption of access to an oracle model that provides ground-truth feedback on the actor's previous response. At each refinement iteration, the actor is informed of the cor- rectness of its last answer and then generates new at- tempt. Our implementation of Oracle-RISE. retrains base models starting from the preliminary training stage using binary feedback, followed by RL training. All baselines are trained on the same problem set as their DPSDP coun- terparts to ensure fair comparison.
Input: reference policy ree = (Trefay Trefe); B, dataset
Pprob. and number of samplings n
Let H =3 for (x, a²) E Prob do
Sample a trajectory T ~ Tref with 30 = x.
(Su, 0o 01, (z).
for = 0, 1,..,..... H 1 do
Sample n responses dj ~ Teer(- sh), where i=
1, 1, n2, and estimate their Q-values 0% (sn, (h) as described in Section 3.3. Extract m pairs (a (n)) from {a/] where 0% (sh, a Q7 (Sh; ) collected as Dr- end for end for Let ãt = (fer Rt) be
na + min CDPO refai Do U D2)
me + min LDPO (T,, Trefoi D1)
where LDPO is defined as Eq. (3).
Ourput: DPSDP policy म
Implementation We conduct experiments using Llama-3.1-8B-Inttue and Qwen2.5-3B as base models. We adopt subset of OpenMatthnnsruutt-22 as the problem set. For each state, we sample n = 8 additional answers or feedback using a tem- perature of 1.0. We construct at most one chosen-rejected pair from these 8 candidate outputs using the estimated Q- values described in Section 3.3, and apply DPO loss with the collected dataset to train the actor and critic separately. Beyond a generative critic that provides verbal feedback, we also implement a value-based critic, referred to as non-gen critic NG which gives only binary correctness feedback. Under this setting, the critic is trained using cross-entropy loss, while the actor is fine-tuned to refine its answers based on binary feedback. More details of our implementation can be found in Appendix D.
training phase before DPSDP. This phase enables the ac- tor to better utilize feedback for refinement and equips the critic to provide more insightful reflections. We defer train- ing details of this phase to Appendix D.I.
4. Experimental Evaluation
4.1. Experiment Setup
We focus on mathematical problem- solving tasks, evaluating our approach on MATH 500 (Hendrycks et al., 2021b) and GSM8K (Cobbe et al., 2021) benchmarks. In general, MATH 500 is more challenging than GSM8K. We use problems from the OpenMathInstruut-2 (Toshniwal et al., 2025) for train- ing, which are sourced or augmented from MATH and GSM8K-the same datasets used for benchmarking. To assess generalizability of our models to oit-of--istribution problems, we evaluate on two additional benchmarks: MMLU-Pro Math (Wang et al., 2024b) and Olympiad Bench (He et al., 2024). Among them, the Olympiad Bench is the most challenging benchmark, featuring Olympiad-level scientific questions.
Tasks and Datasets
4.2. Main Results
We evaluate our methods and present the main results in Table and more comprehensive set of results is pro- vided in Appendix E. Following Qu al. (2024), we use three key metrics:
pass @turnl (pl @t!) measures the accuracy of the actor's initial response without any reflection or re- finement.
Baselines We compare our algorithm against methods adapted to enhance LLM-generated responses: (1) STaR (Zelikman et al., 2022) fine-tunes the models on self- self-gener- ated correct answers. Specifically, for each problem in the training set, STaR samples multiple complete answer-feed-
maj1 tturn5 (ml @t5) computes the accuracy of the majority voting answer over five generated answers: the initial response followed by four refinements from the actor with feedback from critic. To mitigate the inherent randomness of LLM evaluations, unless oth-
6

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Approach
ID
OOD
MATH 500
GSM8K
MMLU-Pro Math
Olympiad Bench
pl@t
ml@r5
pl@t5
plêtl
ml@t5
plets
pl@tl
ml@t5
pl@t5
pl@ti
mi@rs
pl@t5
Ministral-8B-It
55.8
53.4
58.4
83.4
81.9
84.7
52.1
50.6
55.7
22.8
22.7
24.8
+SFT
55.4
57.2
68.0
82.3
82.3
88.2
50.4
49.7
62.2
22.1
23.4
30.7
+DPSDP (ours)
58.2
63.2
70.0
87.8
89.1
92.7
53.1
54.2
64.3
25.8
27.0
32,9
Non-Gen Critic
+SFT
57.0
59.0
62.0
84.5
85.1
86.2
50.6
51.7
54.4
24.2
24.8
27.2
+DPSDP
59.2
62.0
64.2
88.9
90.4
91.0
52.8
53.3
57.8
25.8
26.6
30.0
Single-Agent
57.0
63.8
89.9
90.4
91.2
54.0
54.9
58.0
23.6
26.3
28.2
Non-Markovian
58.2
62.6
67.8
87.8
88.9
91.3
52.8
59.7
25.8
26.4
29.5
Llama-3.1-8B-It
49.2
43.2
60.2
83,4
67.0
88.5
50.1
41.7
62.0
18.1
14.8
26.6
+SFT
51.4
53.4
56.8
83.3
78.9
86.6
55.7
56.1
61.4
20.5
22.3
24.6
+DPSDP (ours)
55.8
87.5
88.4
91.2
56.6
58.0
62.1
22.4
23.0
Non-Gen Critic
+SFT
53.2
54.2
57.4
84.2
85.1
87.1
55.4
55.8
60.0
20.5
21.7
23.3
+DPSDP
56.0
56.2
60.2
88.6
89.8
91.2
56.9
56.5
62.1
20.2
20.6
23.1
Single-Agent
53.4
54.8
58.0
87.9
87.6
90.4
56.1
57.3
62.0
23.0
21.5
25.1
Non-Markovian
55.8
57.2
61.2
87.5
88.2
91.0
56.6
57.0
60.5
22.4
22.6
24.6
Qwen2.5-3B
57.6
48.0
58.6
78.6
75.2
79.4
47.4
41.2
48.4
24.0
22.0
24.5
+SFT
60.0
60.4
64.6
79.1
77.7
81.5
50.9
51.4
56.00
23.9
24.8
26.4
+DPSDP (ours)
60.4
62.0
65,2
79.9
79.9
84.2
52.6
53.2
57.1
24.0
24.0
26.0
STaR
Ministral-8Bi
54.6
64.6
84.9
85.7
89.9
58.6
21.5
22.7
29.4
Llama-3.1-885
50.8
52.2
56.8
83.6
81.3
87.5
53.8
54.6
58.5
20.5
20.3
22.4
Qwen2.5-3B
59.0
59.6
64.8
80.3
79.5
83.7
51.9
51.8
56.8
23.3
22.6
24.8
STaR-DPO
Ministral-8B-It
56.6
67.8
87.6
89.5
92.7
51.6
52.6
63.2
25.2
26.4
31.8
Alama-3.1-8B--.
54.2
55.6
59.2
87.5
87.4
90.3
54.8
55.0
60.3
20.9
21.5
24.3
Qwen2.5-3B
60.4
60.2
64.8
79.4
78.9
82.6
51.2
52.3
55.9
23.1
22.8
28.9
Dracle-RISE
Ministral-88 -B-IIt
59.2
65.4
65.8
88.9
92.6
92.9
52.8
61.3
62.4
25.8
30.6
30.9
Table 1. Performance comparison of various approaches. DPSDP effectively improves reasoning performance from supervised fine- tuned models enables agents to achieve higher accuracies hy generating more answers. The highest accuracies achieved by the same base models (excluding Oracle-RISH RISE, which has access the ground-trutt answers) under each metric underlined.
erwise specified, questions with no more than two cor- rect responses considered incorrect.
phase. By rolling five answers, DPSDP policy enables the actor to increase the correctness of its first answer by 5.0% on MATH 500 and 1.3% GSM8K. In contrast, pervised fine-tuned models achieve only 1.8% improve- ment on MATH 500 and show no gain on GSM8K.
pass1 @ turn5 (p1 @ t5) is also based on five generated answers but considers problem solved if at least one answer is correct. This metric particularly valuable in settings where an oracle reward model is avail- able to verify correctness.
Comparison with Baselines We implement the STaR and STaR-DPO baselines using the supervised fine-tuned models obtained during the DPSDP training pipeline. In- terestingly, STaR, which does not incorporate incorrect fi- nal answers during training, leads to negligible improve- ment with five refinement iterations. STaR-DPO achieves near-saturated performance on GSM8K but offers only a
The effectiveness of our algorithm is demonstrated in Ta- ble 1. For instance, DPSDP improves the five-turn majority voting accuracy of the Ministral-based model from 57.2% 63.2% on MATH 500 and from 82.3% 89.1 % on GSM8K compared to the models after preliminary training

---

Reinforce LLM Reasoning through Multi-Agent Reflection
modest accuracy gain over the first-turn answer on the more challenging MATH 500 benchmark (e.g., 2.2% for Ministral-based models). The contrast between STaR and -PO highlights the importance of incorporating neg- ative data during training, which helps prevent response degradation in subsequent attempts. While STaR-DPO demonstrates the potential for iterative answer refinement, its lack of the restarting mechanism used in DPSDP lim- its train-time exploration. On Ministral-based models, this results in a five-turn accuracy of 58.8%, which is notably lower than the 63.2% achieved by DPSDP. This gap is also evident in Llama- and Qwen-based models. We further in- stantiate Oracle-RISE using Ministral base models. Our results show that DPSDP models achieve maj@15 accu- racies approaching those of Oracle-RISE on challenging benchmarks such as MATH 500 and Olympiad Bench, de- spite Oracle-Rie RISE having access ground-truth feedback. Notably, our models consistently outperform Oracle-RISH on @15s suggesting that the actor-guided by critic feedback explores the solution space more actively in- stead of relying solely on initial responses.
level problems, Ministral-base models improve their first- answers over five attempts by 1.2%, demonstrating that they have internalized the ability to iteratively refine responses rather than merely memorizing in-distribution data. STaR-DPO also exhibits some generalization to OOD benchmarks, but its limited training-ting exploration re- sults in suboptimal performance compared to DPSDP.
Generative Critic vs. Non-Generative Critic In addi- tion to using generative LLM as the critic, we also exper- imented with a value-basee reward model, where a value head is applied over the transformer's outputs to serve as the critic. We denote this non-generative critic as =NG. Given a problem æ with a ground truth answer a" and an actor-generated answer a/. the non-generative critic pro- duces probability estimate Ph+1 ~ N. | [ar, aA)) between O and 1, approximating the probability that an matches a". If Ph+1 > 0.3, the critic provides feedback ah+1 affirming the previous answer; otherwise, it warns the actor that the response may contain an error. Unlike the generative critic, AN does not provide detailed feedback beyond correctness estimation. To evaluate the effective- ness of NN we train the critic alongside corresponding actor IN which is specialized in processing binary feed- back, forming the system NN = (=NN ,,NG). Our results show that DPSDP can be effectively applied in scenarios where only binary feedback is available, exemplified by the five turn accuracy improvement in Table 1. It performs par- ticularly well on elementary problems like GSM8K, even surpassing the generative critic in this setting. We sus- pect this is due to the over-thinking phenomenon, where the generative critic tends to overanalyze less challenging problems, leading to performance degradation over addi- tional turns, as observed in prior works (Chen et al., 2024a; Shridhar et al., 2024). We also provide qualitative analysis supporting this hypothesis in Appendix E.4. However, the limited feedback space of non-generative critic constrains its ability to achieve the same level of refinement as the generative version on more challenging benchmark MATH 500 and OOD benchmarks. Notably, NG consistently un- derperforms the generative critic in pass @turn5, suggest- ing that fails to sufficiently explore the solution space at inference time and makes less modifications its previous answers.
Comparison with Self-Consistency Wang et al. (2022) highlight the effectiveness of majority voting over multi- ple sampled responses. A potential concern is that the im- provement in majl @t5 may stem from sampling multiple answers rather than the cooperative refinement of multi- agent interactions. To investigate this, we evaluate the ac- tors on maj5 @tl (m5 @tl), which measures the accuracy of majority voting over five independently sampled first-turn answers. As shown in Table 2, the results confirm that the performance gains primarily come from the models' abil- ity to identify and correct errors from previous responses rather than from majority voting alone- particcllllllrly challenging benchmarks like MATH 500.
DPSDP
MATH 500
GSMSK
pl@1
m1@15
m5@11
pl@d
m1@15
m5@u
Ministral
58.2
63.2
59.2
87,8
89.1
88.9
Llama-3.1
55.8
58.4
55.2
87.5
88.4
88.2
Table 2. Comparison between ml @115 and m5@t1. The perfor- mance gains stem mainly from the model's ability to identify and correct errors, rather than relying solely on majority voting.
4.3. Ablation Study and Discussion
Generalization to Out-of-Distribution (OOD) Problems Since DPSDP is trained exclusively on problems from OpenMathInstruct222aa0 augmented collection of MATH and GSM8K problems-a key question is whether it can generalize to unseen benchmarks. To assess this, we evaluate our models on MMLU-Pro Math and Olympiad Bench. The results show, even on challenging Olympiad-
Single-Agent vs. Multi-Agent We also explore whether utilizing multiple LLM agents with specialized roles con- tributes to the performance advantage. To investigate this, we replicate the training process on a single base model and report the evaluation results in Table 1. Our findings indi- cate that DPSDP effectively enhances performance across most benchmarks even with single agent, enabling it to
8

---

Reinforce LLM Reasoning through Multi-Agent Reflection
both solve problems and reflect on its responses. For in- stance, the Ministral-based single-agent policy improves its first-turn accuracy from 57.0% to a majority voting accu- racy of 59.6% on MATH 500. However, the single-agent approach underperforns compared to multi-agent policies on more challenging benchmarks such as MATH 500 and Olympiad Bench. Interestingly, we find that the single- agent framework has a slight advantage over the multi- agent approach benchmarks containing grade-school- level problems, such as GSM8K and MMLU-Pro Math. This suggests potential performance boost through posi- tive transfer between the two related tasks of answer gener- ation and feedback provision. For example, the actor may internalize reflection skills and preemptively avoid errors in its initial response, while the critic could enhance its feedback by learning from the answer-generation process. These observations align with prior research showing that integrating both tasks into the training objective can be ben- eficial (Zhang et al., 2024a; Yang et al., 2024b). We view this as a promising avenue for future research.
problem æ, we sample n full trajectories ~ Feer( (- x)) and construct the DPO dataset accordingly. As shown in Table 3, our results indicate that incorporating the restart mechanism generally have a comparable or more substan- tial improvement performance across multiple answering attempts, supporting the idea that restart-style data collec- tion facilitates better exploration.
Approach
MATH 500
GSM8K
pl@tl
m1@1
p1
pl@ti
ml@15
pl
Ministral
w/o restart
56.6
61.6
68.0
88.9
89.5
92.9
restart
58.2
63.2
70.0
87.8
89.1
92.7
Llama-3.1
w/o restart
56.2
57.6
61.4
87.9
88.3
90.1
w/ restart
55.8
58.4
62.0
87.5
88.4
91.2
Table 3. Comparison of data collection with and without restart.
Markovian vs. Non-Markovian As discussed in Sec- tion 3.3, we define the states to incorporate only the most recent answer with the heuristic assumption that the last answer has a more profound impact on future refinements, which corresponds to the Markovian setting. The main ad- vantage of this design choice is that our models are less affected by distribution shift, and the policy produced by DPSDP can be used to refine answers arbitrarily many times without the constraint of training-time horizon. To support our argument, we reevaluate the models with each agent having full access to previous messages, which vi- olates the defined MDP and which we denote as the non- Markovian setting. Results in Table show that, under non- Markovian scenarios, the policy can still improve its first- turn answer through iterative refinement. However, de- spite having more information available, agents suffer from test-time distribution shift and underperform compared to agents seeing only the last answer. One intuitive reason why the non-Markovian setting underperforms is that in Assumption 1. Ct represents trajectory-level distribution shift in the mon-Markovian setting. While if the Markov assumption holds, Markovian setting has state-level distri- bution shift for C3 under our defined state, which is always strictly smaller than the trajectory-level shift.
Additional Analyses and Insights In addition to our main findings, provide extensive analyses in Ap- pendix E to further validate the effectiveness of our ap- proach. We examine how performance evolves as the num- ber of refinement iterations increases, scaling up to 10 itera- tions, and analyze the corresponding transitions in response correctness. We also ablate the preliminary training phase to demonstrate its necessity prior reinforcement learning. Furthermore, we empirically validate the Q-value estima- tion methods described in Section 3.3. A comprehensive qualitative analysis, including common failure patterns, is also presented. These results offer deeper insights into the mechanisms behind the observed performance gains and highlight the key design choices that contribute to our method's 's success.
5. Conclusion and Future Research Direction
We introduce DPSDP, practical RL algorithm for multi- agent iterative solution refinement. Our algorithm offers both strong theoretical performance guarantees and sub- stantial empirical improvements.
Restart vs. Non-Restart Data Collection The data col- lection process in Algorithm 2 follows a restart-style ap- proach: we sample complete trajectory estimate Sh det and generate multiple responses from this state. We ar- gue that this strategy enhances action space exploration by enabling more diverse responses from each sampled state
A promising direction for future research is developing an online or iterative algorithm that adapts to changes in state distribution during training, analogous to prior work on on- line/iterative DPO (Guo et al., 2024a; Dong et al., 2024). Furthermore, it is possible to explore a mixed generation objective for agents for transfer learning, as detailed in Sec- tion 4.3.
Sj.- To test this hypothesis, we modify the data collec- tion method to a trajectory-level approach, where for each

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Impact Statement
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak lan- guage models to strong language models, 2024b. URL https: http://axxvvvvvo/ org/ abs 022400133335
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vin- cent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Alek- sandra Faust. Inference-aware fine-tuning for best-of- n sampling in large language models. arXiv preprint arXiv:2412.15285 2024.
References
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ash- win Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs, 2023. URL https: / /ww.iv.vvvv..///aa//..///aaa oogg/b/g/aaoo 8844.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong. Saining Xie, Dale Schuurmans, Quoc V Lc. Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17106 2025.
abs/ org/abs/2305. 2305.0
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to un- derstand learning from human preferences, 2023. URL https: / /awxivvvvvv.///aa///a.//aaaaavva www.iiiiw//.aa.........eeoooeeee wwgg/bgg////ooooo..gog.../...bre wtw.///10000 222022....
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training ver- ifiers to solve math word problems. arXiv preprint arXiv:21 10..21111..418, 2021.
J. Bagnell, Sham M Kakade, Jeff Schneider, and An- drew Ng. Policy search by dynamic programming. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003, URL https:: neurips ps. 1e
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingho Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint 0rXiv:2405.07863 2024.
aper. pdf.
Eugene Charniak and Mark Johnson. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Kevin Knight, Hwee Tou Ng, and Kemal Oflazer, editors, Pro- ceedings of the 43rd Anmual Meeting of the Association for Computational Linguistics (ACI, (05), pages 173- 180, Ann Arbor, Michigan, June 2005, Association for Computational Linguistics. doi: 10.3115/121984..11 1219 862. URL https: /awlantho ogy. org/P05-1 022/.
Silin Du and Xiaowei Zhang. Helmsman of the masses? evaluate the opinion leadership of large language models in the werewolf game, 2024. URL https: /ttp///axxii arxiv. org/ abs abs//////000 /2004 01602.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenen- baum, and Igor Mordatch. Improving factuality and rea- soning in language models through multiagent debate, 2023. URL https: / //axivivvvv/..////////..../aaa.v org/ abs./30..... 4325.
Justin Cao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. Magicore: Multi-agent, iterative, coarse-to-fine refinement for rea- soning, arXiv preprint arXiv:2409.12147 2024a.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL https:
/ //aaxivvorg//aee /20../02222..2.......200
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answer- ing dataset. In Houda Bouamor, Juan Pino, and Ka- lika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889-7901, Singapore, December 2023, Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 2023.emnlp hallp-mmii.444 main.489. URL https: /aclantoooo gy oo.org/ 2023.. mmnlppmaa in 489/.
Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. Empowering biomedical discovery with ai agents. Cell, 187(22):6125-6155 2024a.
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Ocrtell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen
10

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Sun. Rebel: Reinforcement learning via regressing rel- ative rewards, 2024b. URL htt.ttt/p//axiiii
Jiwoo Hong, Noah Lee, and James Thome Orpo: Mono- lithic preference optimization without reference model. arXiv preprint arXiv:2403.0704 2024a.
2bs/2404.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yu- jiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, 2024. URL pttps://arxiin org/o/////o.o. s/2305.11733
Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao. Chenglin Wu, and Jürgen Schmid- huber. Metagpt: Meta programming for a multi-agent collaborative framework, 2024b. URL https::/s xiv. org/ abs 2200. 00352.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama herd of models. arXiv preprint arXiv:2407.21783 2024.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2024. URL 10. 01798.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model align- ment from online ai feedback, 2024a. URL https: / //aaxxiv.orr////////./ abs/ wwwp////2..... 04792.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holis- tic and contamination free evaluation of large language models for code, 2024. URL /tLht:ps://aat http//axxvvv.r////// abs 2403. 07974.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xi- angliang Zhang. Large language model based multi- agents: A survey of progress and challenges, 2024b. URL https: /ttp:v//aavvvv///a//aa/eeeeeawaa org/ abs/ 2402. 01680.
Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang. Weijie J Su, Camillo Jose Taylor, and Tanwi Mallick. Multi- modal and multi- -ggett systems meet rationality: sur- vey. In ICML 2024 Workshop on LLMs and Cognition, 2024. URL https: / wpwpeeeeeewwweee/////e/ee wet.iiiwweeeeeeeeeeeeeeeee.eeeee forum ? ?id=9Rtm2gAVjo.
Tiankai Hang, Shuyang Gu, Dong Chen, Xin Geng, and Baining Guo. Cca: Collaborative competitive agents for image editing, 2024. URL https://aaxiii/
Edward Junprung. Exploring the intersection of large lan- guage models and agent-based modeling via prompt en- gineering, 2023. URL https: http///axvvvvv/// org/ / 2.308 C7411.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yie Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to improve Ilm reasoning via global and local refinements, 2024. URL https::///...........t...tt.tttt / //e arxiiivvie org/ / a abs / 24.2.1 0963.
Sham M. Kakade and John Langford. Approximately op- timal approximate reinforcement learning. In Interna- tional Conference on Machine Learning, 2002. URL https: //aps:///pp..// http:/app..emmee. www.sticsmcnneeee..... orq/ usID:31442909.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yu- jie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilin- gual multimodal scientific problems, 2024.
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can Ilms actually correct their own mistakes? a critical survey of self-correction of lims. Transactions of the Association for Computational Lin- guistics, 12: 12:1417-1440, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learn- ing Representations (ICLR), 2021a.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Lan- guage models can solve computer tasks, 2023. URL
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes. Avi Singh, Kate Baumli, Sharig Igbal, Colton Bishop. Rebecca Roelofs, et al. Training lan- guage models to self-correct via reinforcement learning.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. Measuring mathematical problem solv- ing with the math dataset. NeurlPS, 2021b.
arXiv preprint arXiv:2409. 12917, 2024.

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Mistral Al team. Un ministral, des ministraux. https:
/ //misttaall a / /eww///////// /mi /tt///miisiiiie /mwimmttaauuti..iii//// October 16 2024. Accessed: 2025-01-14.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the poten- tial of slms in grade school math, 2024. URL https:
Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. Lim-based agent society investigation: Collabo- ration and confrontation in avalon gameplay, 2024. URL https: / /awxiv..vvvv.///// org/ has//ab///// /2310 14985.
//arxiv. org/ abs 2402 1180021148800 14830.
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Markian Rybchuk, Philip H. S. Torr, Ivan Laptev, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent Ilm training, 2024. URL https: //axiivaaiivv////a/iiai org/ //ab s s/2412. 928.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujuu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint 13X.::33005.11188 2023.
Rémi Munos, Michal Valko, Daniele Calandriello, Mo- hammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Gir- gin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learn- ing from human feedback, 2024. URL https: xiv. org/ abs 2212.33122222 00886.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhen- peng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey, 2024a. URL https///axxxv..roo /bs/ 2409. 02977.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tau Yih, Sida I. Wang, and Xi Victoria Lin. Lever: Learn- ing to verify Linuaggeto-cood generation with execu- tion, 2023. URL https: /tttpss//arxxt /ttpxvaxvvv/////////////////.av. org/abs/22 08.68468.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statis- tical rejection sampling improves preference optimiza- tion, 2024b. URL https: //////aavvvv////////aaa//eeleaaa wwwo/rgg/////oooooo///////////ss 09. 06677.
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang. Jianfeng Gao, and Armando Solar- self-repair silver bullet for code generation?, 2024. URLhttps: / //arxiv. /awxiivorg//bb//2230 wttp://///.........///// 09896.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang. Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, al. Star- coder 2 and the stack v2: The next generation. arXiv preprint arXiv: 191.71933, 2024.
OpenAI. Learning to reason with Ilms, 2024. URL http s: / /t//ppenai... wwtp://ww/eenieennaeeeeeeeeeeeee wtwpeeeeweeeannnn--teeeee rea son hon-with-iie with-1lms/ Accessed: 2025-01-14.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He Hc, Sainbayar Sukhbaatar, and Jason Weston. It- erative reasoning preference optimization, 2024. URL
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hal- linan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Her- mann, Sean Welleck, Amir Yazdanbakhsh, and Pe- ter Clark. Self-refine: Iterative refinement with self- feedback, 2023. URL https: wwwaaaiiv.rgg/.o s/ 8/2303. 17651.
https:/ http://aaivv..oo/ab//////// 197333...99..
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, and C. Zhang, editors, Advances in Neural Infor- mation Processing Systems, volume 37, pages 55249- 55285, Curran Associates, Inc., 2024. URL https: / "pppceeeingg..euurpps..c//pppppp V/aper//22///iie/333022E22222220 aba51100Bffaa-pppee--oCoreeeeeep
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Sim- ple preference optimization with reference-frn reward. Advances in Neural Information Processing Systems, 37: 124198- 124235, 2024.
12

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Di- rect preference optimization: Your language model is se- cretly reward model. Advances in Neural Information Processing Systems, 36:53728-53744 2023.
Proceedings of the International Conference on Learn- ing Representations (ICLR) 2025, 2025. URL https: / //openreview.oo net forum 1ididddddwwww2mmm
Nisan Stiemnon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with hu- man feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural In- formation Processing Systems, volume 33, pages 3008- 3021. Curran Associates, Inc., 2020. URL https: / baaroceedinggs.eurips....co httpeeaarrrrrr..ppppp.__..... //ccp///sssssseoorrooo/o/o/a//pe
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q": Your language model secretly a q- function, 2024. URL httpps://axxvv.oo/ s/2404. 12358.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Di- rect nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint 035::2404.007115, 2024.
(aappe//0000/EL1e/11888888833332 e -1
pdf.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhi- wei Steven Wu, and Alekh Agarwal. A minimaximal- ist approach reinforcement learning from human feed- back, 2024. URL https: //apiv/aaiiv.////a org/ abs//////////// 2abs/24 /4 01 04006.
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. Multi-turn re- inforcement learning from preference human feedback. arXiv preprint ar arXiv:2405. 14655, 2024.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Er- mon, Chelsea Finn, and Aviral Kumar. Preference fine- tuning of iims should leverage suboptimal, on-policy data, 2024. URL https: //tpa/ii....//aaa...../////// 04.14367.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junx- iao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the lim- its of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawende F. Bissyande. Codeagent: Autonomous communicative agents for code review, 2024a. URL https:///xxxvvorr/aab///////
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Pro- cessing Systems, volume 36, pages
021722222.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Har- vey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment, 2024b. URL https: arxiv. org/ abs /202 05749.
Curran Associates, Inc., 2023. URL httpttttttss//rroeeeeet ng s. http:/pppssccpppppe_ieeeee/eeeee hL11/D24437763372223344488888883 O-Paper-Conferencee pdf..
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu. Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, and Asli Celikyilmaz. The ART of LLM refinement: Ask, refine, and trust. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- fies: Human Language Technologies ( (Volume Long Papers), pages 5872-5883, Mexico City, Mexico, June 2024. Association for Computational Linguisties. doi: 10. 18655/v1/2024..no/l 12..6655////2224mmaaaeoooo..... naacl- URL. https: //ac lanthology lanthology.orrg22244 2024. naacl 1ong.1oon..... 327/.
CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua How- land, Nam Nguyen, Siqi Zuo, Andrea Hu, Christo- pher A Choquette-Choo Jingyue Shen, Joe Kelley, et al. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406. /409, 2024.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. OpenMathhInstrutt::: Accelerating Al for Math with Massive Open-Source Instruction Data. In ICLR, 2025.
Leandro Von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https: //gi thub. com/hugg ng fac.//rrl, 2020.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling parameters for reasoning. In
13

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08995, 2023.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yim- ing Yang, and Quanquan Gu. Self-play preference op- timization for language model alignment, 2024a. URL attps://axivv.ot htwpg///aa///..000//00
Zengqing Wu, Run Peng, Shuyuan Zheng, Qianying Liu, Xu Han, Brian Inhyuk Kwon, Makoto Onizuka, Shao- jie Tang, and Chuan Xiao. Shall we team up: Explor- ing spontaneous cooperation of competing Ilm agents, 2024b. URL htttt:t/// www.a.iw.oogo..ooooeeoeoooooiaoe ab2/g/2b 2327.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating College-Level Scientific Solving Abilities of Large Language Models. In Proceedings of the Forty-Fire First International Conference on Machine Learning. 2024a.
Zhiheng Xi. Dingwen Yang. Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shi- han Do, Wenyu Zhan, Xiao Wang, Rui Zheng, Tao Ji, Xiaowei Shi, Yitao Zhai, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Zuxuan Wu, Qi Zhang, Xipeng Qiu, Xuanjing Huang, and Yu- Yu-Gang Jiang. En- hancing IIm reasoning via critique models with test-time and training-time supervision, 2024. URL https: //arxiv. org/ abs///b//////
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171: 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024b.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harness- ing implicit q* at-appoxximatin for sample-efficient rlhf. arXiv preprint arXiv:2405.21046 2024.
Jason Wei. Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou, Chain-of-thought prompting elicits rea- soning large language models. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors.AAvancee Advances in Neural Information Processing Sys- tems, volume 35, pages 24824-24837 Curran Asso- ciates, Inc., 2022. URL https: URt /proceedis..ceee/gs.... neurips http://cccccpppeeeeeeeeppprr///a 1e/ /tt////000ii22eeeaaa11aaaraaaaae aper-Confereneee pdf..
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang. Han Zhong. Heng Ji, Nan Jiang, and Tong Zhang. Itera- tive preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.1 1456, 2023.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosen- berg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Salch, et al. Building math agents with multi-turn iterative preference learning. arXiv preprint wrXiv:2409.02392, 2024.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Gen- erating sequences by learning self-correct, 2022. URL https: / /aaaiaivvv..//..// wttp:iiioorr...ooooooo abs/ 221122111111 00053.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: It- erative preference optimization with the pairwise cringe loss, 2024. URL https:////// : aaxii.iiivv...a. org/ abs/ /2312 .16622.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davi- son, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexan- der M. Rush. Transformers: State-of-the-ar natural lan- guage processing. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguis- tics. URL ht t pps: gy/2020 emn 1 1 demos. 6.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Day- iheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learn- ing generalizable reward model for llms. In Advances in Neural Information Processing Systems, 2024b.
/ /ww.//ww..moom//.o. www.w..wwwwbb......o.w..o/w..... aclweb. gagghooooo
14

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 'rree of thoughts: Deliberate problem solving with large lan- guage models. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances in Neural Information Processing Systems, vol- ume 36, pages 11809-11822, Curran Associates, Inc., 2023. URL http::::ppccceeeaiiiii neurips. cc/paper_fiies 9922881144d7aaaf8888dd0777339993 ference.pdf.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Informa- tion Processing Systems, 2022. URL https: / /open review... net/ forum? ? d== dedddd=__________
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mchran Kazemi, Aviral Kumar, and Rishabh Agarwal. Genera- tive verifiers: Reward modeling as next-token prediction. arXiv preprint 2r0iv:2408. 15240, 2024a.
Xianren Zhang, Xianfeng Tang. Hui Liu, Zongyu Wu, Qi He, Dongwon Lee, and Suhang Wang. Divide-verify- refine: Aligning Ilm responses with complex instruc- tions, 2024b. URL httph:t///seee http:://aaxxv.rrrg abs/ 2410.12207.
Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf, 2024. URL org/ abs 22..2404..888 18922.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024.
15

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Appendix
A. Related Work
A wide range of prior research focuses on improving LLM performance by enabling models to revise their answers for greater accuracy and robustness. Within this domain, subset of works investigates intrinsic self-correction, where LLMs refine their outputs without relying on external feedback (Qu et al., 2024; Kumar al., 2024). While these methods equip models with abilities to address their own errors, studies reveal mixed results, with some highlighting the difficulties LLMs face when self-correcting without external input (Kamoi et al., 2024; Huang et al., 2024). In contrast, other works leverage external feedback to enhance responses, utilizing tools such as compilers and verifiers (Welleck et al., 2022: Havrilla et al., 2024; Chen et al., 2024a; Shinn et al., 2023). More recently, multi-agent systems have gained attention as a promising approach, training two or more LLMs to collaborate or compete to generate improved solutions for reasoning tasks (Xi et al., 2024; Motwani et al., 2024). Our work aligns with this direction, proposing a novel reinforcement learning algorithm to enhance the interaction and coordination between models.
Intrinsic Self-Correction Several prior works have developed techniques to enable LLMs to correct emoos in their previous responses. Some approaches prompt LLMs to reflect on their answers and attempt revisions when necessary (Shinn et al., 2023; Kim et al., 2023; Madaan et al., 2023). Among these, Kim et al. (2023); Shinn el al. (2023) assume access to ground-truth answers during self-correction. which is often unrealistic. Other works explore algorithms for training models to enable models of self-correction. Kumar et al. (2024) demonstrate that standard supervised fine-tuning insufficient for improving LLMs' ability to self-correct. Instead, they formulate self-correction as a Markov Decision Process (MDP) and apply a two-stage reinforcement learning algorithm to enhance performance, which focuses on single refinement. Qu et al. (2024) train models to iteratively refine their responses over multiple turns, which is relevant to our work. In contrast, our approach explores leveraging flexible feedback from another model to guide self-correction under this multi-turn setting.
Self-Correction with External Feedback Another line of research explores self-correction using additional feedback from the environment. A common setting is code generation, where feedback is derived from unit test results or compiler messages (Chen et al., 2024b; Jain et al., 2024; Olausson et al., 2024; Ni et al., 2023). Other works incorporate external tools, such as scripts or search engines, to provide feedback for refining responses (Gou al., 2024; Zhang et al., 2024b). Additionally, some approaches utilize feedback generated by other models (Welleck et al., 2022; Havrilla et al., 2024). However, these works treat the model generating answers and the entity providing feedback as two separate components, focusing either on leveraging feedback from a fixed set of sources (Olausson et al., 2024; Ni et al., 2023; Gou et al., 2024) or training a dedicated corrector model for refinement (Welleck et al., 2022). In contrast, our algorithm trains a multi-agent system jointly, enabling seamless interaction and collaboration between models.
Multi-Agent Collaboration LLM-based multi-agent systems have shown promising results across a variety of tasks (Guo et al., 2024b; Gao et al., 2024a; Liu et al., 2024a; Jiang et al., 2024). The communication paradigms in these systems can be broadly classified into two categories (Guo et al., 2024b): (1) Debate or competitive agents, which interact by defending, and critiquing viewpoints or solutions (Junprung, 2023; Du and Zhang, 2024; Lan et al., 2024), and (2) Coop- erative agents, which work together toward shared goals, exchanging information to enhance collective solutions (Hong et al., 2024b; Hang et al., 2024; Tang et al., 2024a; Wu et al., 2024b). In the context of reasoning, Du et al. (2023); Liang et al. (2023) explored improving LLMs' reasoning capabilities by employing debate-like interactions among agents. Other works (Chen et al., 2024a; Shinn et al., 2023; Xi et al., 2024; Motwani et al., 2024) refined LLM-generated responses through collaborative multi-agent frameworks, often involving specialized roles such as actor models for response gener- ation and critic models for evaluation and feedback. However, some of these approaches lack specialized model training (Akyürek et al., 2023; Chen et al., 2024a; Shinn al., 2023) omit joint training of agents for collaboration (Xi al., 2024). The most related work is Motwani et al. (2024), which trains three-model system to generate refined answers after an initial attempt. In contrast, our work focuses on an algorithm with theoretical performance guarantee that enables multi-turn refinement, fully encompassing the 2-turn cases.
Many prior works have applied reinforcement learning (RL) algorithms to LLM training. One line of research focuses on single-turn settings, where response generation is formulated as a token-level Markov Decision Process (MDP), and pairwise preference data is used align models with human preferences (Rafailov et al., 2024; Gao et al.,
RL for LLMs
16

---

Reinforce LLM Reasoning through Multi-Agent Reflection
2024b; Zhong al., 2024). Among these, Direct Preference Optimization (DPO) (Rafailov al., 2023) demonstrates that LLMs can be aligned with human preferences by maximizing the margin between the implicit rewards of chosen and rejected responses. Building on this, many algorithms have been proposed to address specific limitations of DPO (Tang et al., 2024b; Azar et al., 2023). Some prior works explore online or iterative RL algorithms, providing theoretical performance guarantees through either high coverage of the reference policy or active exploration (Xiong et al., 2023; Guo et al., 2024a; Xu et al., 2024; Pang et al., 2024; Mitra et al., 2024; Dong et al., 2024; Xie et al., 2024). Another subset of these works focuses on a extended problem of general preference learning where binary labels are augmented to numeric probabilities representing the likelihood of one response being preferred over another (Wu al., 2024a; Swamy et al., 2024; Munos et al., 2024; Rosset al., 2024). Other aspects include on-policy training (Tajwar et al., 2024; Liu al., 2024b), direct utility function maximization (Ethayarajh et al., 2024), and and simplified algorithms that eliminate the need for reference policy (Hong et al., 2024a; Meng et al., 2024).
Another line of research focuses on improving LLM performance in multi-turn settings, where the prompt and conversation history are treated as the state, and each response is an action. Some works aim to generally align models with human preferences multi-turn conversations (Shani al., 2024; Zhou et al., 2024), while others specifically enhance step-by- step reasoning (Snell al., 2025; OpenAI, 2024; Wang et al., 2023) or integrate tool usage (Xiong et al., 2024). Qu al. (2024); Kumar et al. (2024) are most related to work. They apply RL algorithms train models for self-correction through multiple sequential attempts. In comparison, we formulate the conversation each state contains only most recent answer, mitigating test-time distribution shift effect regarding number of refinements
B. A Additional Notation and Pseudocode
We provide the pseudocode for original PSDP algorithm described in Section 2.
Algorithm PSDP
Input: horizon H. policy set II, and baseline distributions μn for h = 0, 0, ..
1: for h = H1 - 1, 2, ..,..,
0 do
2: Th + arg max Es-na
(a
(s)
h
where (x, ( ... H
denotes a non non-stationary policy composed of stationary policies n, m/41, ,: ,T H-1
3:
end for
C. Theoretical Proofs
C.1. Proof of Theorem 1
We first introduce episodic version of performance difference lemma (Kakade and Langford, 2002).
Lemma (Performance difference lemma). For any policy n' and n. we have
J(T') J(T) = Σ
sq d (ag
1Q (sh, az)] - Vē (sw)]
h=0
Let Sn. be any function, then from Lemma 2 we have
-(
E
Bannd
(ag [o (s2, (s))aa)) V/ (sn)
(4)
h=0
H
Σ
Bennd (1er))))))))) ( (sn, an) (n(on,an)
h=0
H
+
E
Ban (1on) (fn an) - Ens (((on) (fn an)]
k=0
17

---

Reinforce LLM Reasoning through Multi-Agent Reflection
#
[a(sn an) (s an)
E sn aa m+())o))
==
Let (h(sh, = log Alap (an (86)
c(sh), where c(s)) = Ban [ R(enlen) [se
- (T an)) then
(15))-1(1 =
1-1
E ruy on mx @R(snran) an) - log
a(an | 188) + c(sh)
(I)
Tref(ap (t))
h=0
H
(aswe)()on) log i(an 11h)
aaa shh) Tr (an 8h.)
3
E Bsa-di wdhh"
Bas (aan)1((1110 (H1l) log
(II)
Tras Trat(ann 88)
H-1
a(an 184)
ax) (sh -c((s)))
+
E
Bandd a()on)
Goog
(III
Tret | shh)
h=0
For term I, from Cauchy-Schwartz Schwartz inequality, we have
H
T(an 81))
Q (sn; an) - -8log
+ c(8h)
Tee (0h 8hh)
k=0
(axh 8h)
E Ban sn an rr (1
2 (Rh. -Blog
+ c(su))
Treel (an shh)
hm0
H 11
(
(ee (on(sn an) - Blog
27 7(an 8h) + c(sh)) Teef (ap st)
E sy mdy on anrrr
h=0
From Assumption 1, we have
(oRten Blog
#(an 88) + c(sn)) 27
B Ban annarl(e
TTeef (an (th)
[( (coieenne)) - - Blog #aan 1 st)) + c(sh))
27
I C. (remmellen))
(ra sh)
We now introduce the following lemma further bound term Lemma 3. Under Assumption 2, have for any h E 10 .. 1}:
( (Blog
a(an sh))
(i an) - -c(s)))
27
Estat
E Ba (1
$
Treel (an sh)
(ap
2
where c(s)) Eagcaaa(le
Blog
alanles) mo((nla))
(sh, (()ss,an)))))
Then from Lemma 3 we know
(oRlon log
7(an 8h)
c(sh) 27
H
E Bnnay an aa (((m))
Teed (ap st))
k=0
#-1
(
7(an 8x) + c(sh)) (Rllon)an)) -Blog
27
C3CAH H
E Es ofd (1so)))
Fee (ap 8h)
h=0
As
H
2
18

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Similarly, term III can be bounded
=1
a(an 8h)
E
Spnndh ,an (+(lo)))
Glog Teet (an sh))
(sn, an) - -c(s))
A=0
H-
T(an 81))
Q7 (9h, an) c(8h)) 27
sn an (+(l)o))
āe (ap sh))
H-1
( (8los log
iaan sh.)
27
CSCAH H
Bn (113))
17 an) - c(sh)
Tree (ah 18h)
1=0
C's CAEtat
< H
2
Now, let's consider term II. Given fixed h and fixed Sh.
a(an
sh.
log a(ab (hh)
Eyyarr (+len)
log
(-1lo)
ref ah (hh)
Teef (ap, sh,)
(ah
5h))
log n (ap. shh)
log
a(an shh)
-Eas (1ea)
log
+ (anonr(--on))
Ban an)
π (an 8h)
Traf (ap (8))
To (an 88h)
log π² (an sh.)
Fas oor (Hlaa)
DRI.(TT(- (-)x) (x.]] DRL. ( 88 (1)))s)))see [|Tert
[h)]]
Fref (an [8h)
From Assumption we have
x² (a 8hh)
E (-1on)
log
DKI.[TT" (11 8a)l(a(. (n)) - Dki.F(( 8x) sx]l|Tee 8n)]
Tref (ah 88))
< Ean ~xx (-1on)
[log Cal 0
= log CA
103Caema log Ca
Therefore, with
term II can be bounded as
h
log (aa sn)
7(an 8n) Trat (an 8p)
E snndd""
Banwny (1Ia))
Eag (()oa)
log
< 3H log CA = H HCCCAAS
Teot (an sn)
A=
In conclusion,
CHVCCCAe (
C.2. Proofs for Supporting Lemmas
Proof of Lemma 2 Let prove by induction. When 1, the trajectory consists of single step, and the statement trivial.
Now assume that for horizon k the statement holds for any policy n' and m:
k Σ
Je(T') - Tm(TT) =
sn (())))))))))))) (sn. an) - V. (s+))
h=1
For H = k 1, expand Te+1 (T') - Jk+t (TT):
k+1
x+1
JK+I(TT) T I = Ex
Σ
rsh an)
Ex
Σ hm
r rsn an)
19

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Ex
rssh an)
(V2
(I)
h=2
k+1
E (r(s), a1)] + Ex [V2 (82)
Ex
Σ
r(sh: an)
(II)
Term corresponds to the difference between expected returns of n' and MDP with horizon k. Apply the inductive hypothesis to term I, we have
1++1
H
E
Σ 2
Er(sh, an)
- -E [82)] = Σ
F h=2
dn (1
18% (Sh. an) - Vi (8h))
For term II, we have
1++1
E (r(81, + + (82)
(82) - Ex
Σ L1=1
rssh, an)
= a1) -.
E syway' [Bar) (s,, -V7
-V
Summing up term I and II concludes our proof.
Proof of Lemma 3
a(an 81) Blog 7(d 1 8h)
27
E sn Ba an -aa
Blg Trer(an st)
Qh(sh an) + Q7 (shh:
Trell/ ss))
E Bennd an (e -aaaa--
((s
( Blogg log i(an [8h)
Q7 an) - c(sa)) -(3) log
(aah 8h) ree (a/, |8b)
(K(, -chh) -dde))))) -ccsn)))
Frre (0h 8h)
(
2B sy oft of an marl-
(Bl log
iian 181)
(8n, - c((h))
Tooe Tiet(an (8h)
2B e/ vdfiea (Ean-matioon)
7(an |8h)
Qi(sn -c(sn) 1)
Glog
Tret (ap (/))
{810g log
7(an 9h)
(sh, -css)())))))))))))())))))))))- c(sh) |
2B odffed marl-)))))
Tref (ah sh))
The last step follows from the definition of c(st). Under Assumption we have (Blog log
(
az) aaan | sh) Qah an) - c(8x)
)
)}
27
Estat 2
E En monelllow)
Tref Fre (0h 8h)
C.3. Analysis of the Approximation of Q-Values
In this section, we analyze the Q-value approximation in practical DPSDP described in Section 3.3. As only one feedback and refinement step is used during training, assume H 3. Let be the resulting policy, and let Qf denote the estimated Q-values, replacing Q in Assumption 2.
We define advantage function as AR an) (T(s,,, (sn),, and Ag an) E Ear ) (7) (Sh, a.)]. Then Eq. (4) can he rewritten as
QT an)
#-1
J( -
Σ
E
Sy M aa (11on)
[A((s))
(5)
h=o
20

---

Reinforce LLM Reasoning through Multi-Agent Reflection
We analyze each term in Eq. (5) as follows:
1. At h 2, the estimated Q7 (s2, 02) - r((3)) is exact. Therefore, (3h,, an)) A (shs an).
2. A = 1, the estimated Q-value is: (81,)) = Bog (r(a)] (81, We define the approximation eπror:
= E sanatt as (1)a)) [A (sh; an) - (sh, ($h,,an))
3. h = 0, we have 200 (so, an) = r(1)) + H-1 = Q7" (sp. ao). Therefore.
2
So1o))(-1s)))) [A (sh, an)) so Bay (1a)) (Sys an) = O,
where the last equality follows from the definition of A7.
Following steps in Appendix C.I, we obtain the approximate upper bound by adding | to the theoretical bound.
To assess the impact of A we performed ablation using the step-by-step DPSDP variant in Appendix E.3, which uses QT in the DPO-style loss. The results showed no significant performance gain, indicating that | has minimal effect. For simplicity and efficiency, we use the original version in the main paper.
D. Implementation Details
D.1. Preliminary Training
As described in Section 3.4, we begin by supervised fine-tuning the model to initialize the critic for providing effective feedback and the actor for refining previous answers. To acquire high-quality SFT dataset, we generate first-turn responses by rolling out the reference policies, exposing errors that learner actor might make. An oracle model then generates feedback and corresponding refined answers, demonstrating expected behavior for critic and actor. After this initial stage, we proceed with DPSDP from the boosted model.
In experiments, we use a subset of mathematical problems from OpenMaathnnstruct-2 (Toshniwal et al., 2025) and sample diverse first-turn answers with the base models quality feedback and refined answers, we employ capable models Distral-Large-IIsttuee to demonstrate how to evaluate and improve initial responses. After filtering (e.g., deduplicating problems and removing degraded answers), obtain final dataset of 381K samples for supervised fine-tuning. We then fine-tune the base models separately: the critic is trained on feedback generation, while the actor is trained on refining answers. Both DPSDP and baseline methods are subsequently trained on these fine-tuned models.
-inttautttttttttt--4444 and Llama-3. 1-8----uettt To generate high- and Llama-3.3-70B-Instuu
D.2. Benchmarks
(1) MATH 500 (Hendrycks et al., 2021b) consists of problems from mathematics competitions, including the AMC 10, AMC 12, AIME, etc. The dataset covers diverse mathematical topics, including algebra, geometry, statistics, number theory, linear algebra, and calculus. Following Lightman et al. (2023), we augment the MATH training set with 4500 problems from the test set and report results on the remaining 500 problems, denoted as MATH 500. (2) GSM8K (Cobbe al., 2021) includes 1319 grade-school-evvel math word problems, which are generally simpler than those in MATH. (3) MMLU-Pro Math (Wang et al., 2024b) is mathematical subset of MMLU-Pro, a challenging multi-task benchmark. It consists of 1351 multiple choice questions augmented from TheoremQA (Chen al., 2023), SciBench (Wang et al., 2024a), and original MMI.U (Hendrycks et al., 2021a). (4) Olympiad Bench (He et al., 2024) is an Olympiad-leve scien- tific benchmark, containing 674 open-ended text-only mathematical problems in English, covering algebra, combinatorics, geometry, and number theory.
D.3. Training Hyperparameters
For supervised fine-tuning (SFT), we experimented with learning rates of 1 x 10 6, 5 x 10-5, and x 10-5, selecting 1 x 10-6 for Llama-based models and 5 10-6 for Ministral- and Qwen-based models. Base models were trained on the
SET dataset for 1 epoch, using gradient accumulation steps of 64 and per-device train batch size of 1 on 4x H100 80GB GPUs.
21

---

Reinforce LLM Reasoning through Multi-Agent Reflection
For direct preference optimization (DPO), we tested learning rates of 2 10-7 and 10-*, 7 choosing 2 10-7 for Ministral-based actor and critic, Llama-based actor, and Qwen-based actor, and 10-7 for Llama- and Qwen-based critics. For the KL coefficient 3, we evaluated values of 0.1, 0.5, and 1.0, selecting 0.1 for all actor model training, 1.0 critic (trained for epoch), and 0.1 for Llama-3.
for Ministral-8B- -es--Instucttbaiiieee epochs) and Qwen2.553BBbaal critic (trained for epochs). Same as in SFT, we used gradient accumulation steps of 64 and per-device train batch size of I on 4XH100 80GB GPUs.
critic (trained for 2
The non-generative critic was trained with learning rate of x 10-0 6 for epoch.
D.4. Inference Hyperparameters
All evaluations were performed with a temperature (I) of O, except for self-consistency (maj5@turnl) where we set T 0.5. To generate dataset used in the preliminary training phase (SFT), reference models (Ministral-8B- -nsssrrutt 2410 and 1-ama-3.1-8B-Istiiie and 1imma-3.3-70B-Instruee then produced feedback and refined answers with T 0. In the data collection stage of Algorithm 2, policies were sampled with 7 1.0 to ensure diverse responses.
generated first-turn answers with 7 0.8. Oracle models Lattral--arg--Instrrce
D.5. Codebase
We adapted code from TRL (von Werra et al., 2020) for both SFT and DPO training. The non-generative critic trained using the Hugging Face Transformers framework (Wolf et al., 2020) with customized loss function. For inference, we utilized the vLLM offline engine (Kwon et al., 2023) and adapted scripts from Xiong et al. (2024). Evaluation code was adapted from Yang et al. (2024a); Grattafiori et al. (2024) to compare L.LM-generated answers with ground-truth solution.
D.6. Prompts
Prompts used for first-turn answer feedback and refined answers
For first-turn answer You are an AI language model designed to assist with math problem-solving. In this task, I will provide you with math problems. Your goal is to solve the problem step-by-step, showing your reasoning at each step. After you have finished solving the problem. present your final answer as |boxed Your Answer). {problem}
For feedback Take moment review your previous response for accuracy and completeness, Briefly identify any errors gaps that might lead to incorrect answers, but don't worry about fixing them-just focus on pointing them out.
For refined answer response. Ensure your final answer addresses any identified issues and present it as (boxed( Your Answer). Pay close attention to the errors you pinpointed during the reflection.
Using your reflection as guide, carefully make any necessary corrections to your previous
Verbal feedback from non-generative critic
The solution appears to be correct. Please review it to confirm its accuracy, and present the verified final answer the format {boxed Your Answer).
Correct
Incorrect There is an error in the solution. Please review it, correct the mistake, and present the revised answer in the format boxxd Your Answer].
E. Additional Results
E.1. Answer Improvement Over Turns
The training process of DPSDP involves single-step refinement rather than multi-turn enhancement. Despite this discrep- ancy between training- and test-time conditions, we find that DPSDP effectively enables models to improve accuracy over multiple turns. In Figure 3, we plot the per-turn turn accuracy, majority voting accuracy, and pass@t-k accuracies on MATH 500 and GSM8K. For this section only, we do not consider questions with no more than two correct responses as incorrect, which is adopted in Section 4.2. The results show general increase in accuracy as the number of refinements grows, with
22

---

Reinforce LLM Reasoning through Multi-Agent Reflection
models steadily improving majority voting performance successive turns. Additionally, the increasing pass @ tturnkk scores indicate that models progressively solving problems they previously could not, demonstrating the effectiveness of iterative refinement.
Accuracy @ turn-k
Majority @turn-k
Pass @turn-k:
Accuracy MATH 500 (%
72
63
65
90
70
93
Accuracy GSM8K (%)
89
63
89
68
92
61
88.5
61
66
91
64
59
88
59
62
90
87
57
60
89
57
87.5
55
86
58
88
56
55
87
53
85
87
1
2
3
4
5
6
8
1011
2
3
4
6
7
8
9 101
2
3
4
5
6
7
8
9
1011
Number of turns &
Number of turns k
Number of turns
Ministral MATH 500
Llama - MATH 500
Ministral GSM8K
Llama GSM8K
Figure 3. Various metrics under different tums. Accuracies improve as the number of refinements increases. The rising pass @turn-k scores indicate that iterative refinement enables models to solve previously unsolved problems. Note that the decrease majl accuracy arises from the requirement that both responses (2 out of 2) must be correct count toward majl @1j.@12.
We further analyze how the proportion of responses that change in correctness evolves throughout the multi-turn refinement process. Table 4 reports results using Ministral-based models on the MATH 500 dataset. We focus on the fraction of responses transitioning from incorrect to correct, denoted A and from correct to incorrect, denoted A c-i at each refinement step. Notably, Ai-ce i-ve consistently exceeds A e- indicating that the refinement process generally beneficial. Interestingly, both i- and A decline successive iterations, suggesting initial phase of active correction followed by stabilization in later stages. The consistently low level of < suggest that over-refinement is not significant concern in practice. However, to further eliminate the possibilities of over-refinement, propose monitoring performance on validation set at each refinement step and stopping early if accuracy begins to decline potential solution.
Turn
12
13
14
15
16
17
18
17
/10
/11
A
(%)
7.8
4.0
3.4
3.0
1.8
2.0
1.4
1.0
1.0
1.2
A
(%)
4.0
3.6
2.8
2.8
1.6
2.4
1.0
1.2
2.0
1.4
Table 4. Fraction of responses that transition from incorrect to correct (A (-) and from correct incorrect (AC+t) at each turn.
E.2. Necessity Preliminary Training Phase
As detailed in Section 3.4, our algorithm begins with models that have been fine-tuned either utilize provide feedback. To assess the necessity of this preliminary training, we conduct ablation study by removing this phase and applying DPSDP directly to base models. The results, shown in Table 5, indicate that this fine-tuning step is essential for enabling the actor and critic models to follow instructions effectively. Without it, applying DPSDP yields negligible performance gains with additional response attempts. For instance, the models achieve only 53.8% accuracy on MATH 500-an improvement of just 1.2% over a single-response baseline. In contrast, incorporating the preliminary training leads to 5.0% absolute gain in accuracy. These findings are consistent with previous work (Chu et al., 2025), underscoring the importance of cosructtio-foolowinng capabilities in models used starting points for RL. training.
E.3. Empirical Impact of Q-Value Estimation
In Section 3.3, we present the development of practical DPSDP approach, leveraging ree to approximate Q-values that are otherwise infeasible or computationally costly obtain. We are especially interested in investigating our approximation of
23

---

Reinforce LLM Reasoning through Multi-Agent Reflection
Approach
MATH 500
GSM8K
MMLUUPro Math
Olympiad Bench
p1611
m1@t
ples5
p1@11
@M
pl@15
p1 (111
m1@45
p1615
p1
m5
p1@15
Ministral-8B-It
55.8
53.4
58.4
83.4
81.9
84.7
52.1
50.6
55.7
22.8
22.7
24.8
DPSDP w/o SFT
52.6
53.8
54.4
90.6
90.8
90.9
54.1
54.4
55.5
26.0
26.1
26.7
DPSDP (ours)
58.2
63.2
70.0
87.8
89.1
92.7
53.1
54.2
64.3
25.8
27.0
32.9
Table 5, Performance comparison of DPSDP with without preliminary training stage (SFT).
(s1,)1,1), the Q-values for the feedback step. To evaluate the impact of it, we conduct experiments on a smaller dataset primarily consisting augmented problems from MATH. first replicate the training process from Algorithm 2 on this dataset, using ref for estimation and combining training across all turns into single step for each agent, which we refer as unified training. In contrast, we implement step-by-step training procedure based Algorithm I with a DPO loss, with each turn being trained separately. We denote this approach step-by-step training. Specifically, sample second-turn answers, which are then used to construct DPO dataset for actor training, yielding an intermediate actor model Ā.. Next, we sample diverse feedback from the base critic and use x2. instead of Tref as in unified training,
first ref to
generate refined answers based on this feedback, obtaining an accurate value QT We then conduct DPO training on the critic, producing a refined critic model मf. Finally, we improve the actor's first-turn responses based on 72. following the same approach in Algorithm 2. To ensure broad exposure to different problems, use three distinct problem sets each stage of step-by-step training, each matching the size of the dataset used in unified training. We evaluate both methods on MATH 500 using dlmaabbase models, as shown in Table 6. Notably, we find that the policy trained via unified training outperforms the step-by-step approach, suggesting both the accuracy our estimation strategy and the potential advantage of jointly learning answers alongside their refinements.
Approach
MATH 500
p1@11
m1@t5
p18t5
Llama-3.1 Unified
54.4
55.2
59.2
Llama-3.1 Step-by-Step
Trained on d2
53.0
54.2
58.6
Trained on di
53.0
54.0
58.8
Trained on do
52.6
54.2
59.0
Table 6. Comparison between unified training and step-by-step training. The experiments are conducted on a smaller, MATH-dominated dataset.
E.4. Qualitative Analysis
We present example output from DPSDP Ministral-based models!
Critic captures errors in actor's 's first answer
Problem A regular hexagon be divided into six equilateral triangles. If the perimeter of one of the triangles is 21 inches, what is the perimeter, in inches, of the regular hexagon? Answer 1 To solve this problem, we need to understand the relationship between the equilateral triangles that make up the regular hexagon.
1. Identify the Perimeter One Triangle: We are given that the perimeter one equilateral triangle is 21 inches. Since all sides of an equilateral triangle are equal, each side of the triangle 3
21 7 inches.
2. Determine the Perimeter of the Hexagon: A regular hexagon is made up of six equilateral triangles. There-
All modifications to the model-generated responses presented in this section were purely formatting-related and did not affect the underlying meaning or content.
24

---

Reinforce LLM Reasoning through Multi-Agent Reflection
fore, the perimeter of the hexagon the sum of the perimeters of these six triangles.
3. Calculate the Total Perimeter: Since each triangle has a perimeter of 21 inches, the total perimeter of the hexagon is x 21 126 inches.
Thus, the perimeter of the regular hexagon is 126 inches.
Feedback 1 Upon reviewing the previous response, I noticed conceptual enror in the calculation of the hexagon's perimeter. Specifically, the perimeter of the hexagon should not be calculated by summing the perime- ters the six triangles, as this would double-count the sides. Instead, the perimeter of the hexagon is simply the sum of the lengths of its six sides. Here the points to consider:
1. Perimeter of One Triangle: The perimeter of one equilateral triangle is given as 21 inches. Each side of the
triangle 21 3 = 7 inches.
2. Perimeter of the Hexagon: The hexagon has six sides, each of which is 7 inches long. Therefore, the perimeter of the hexagon is 6 7.
The еттоr in the previous response was incorrectly summing the perimeters of the triangles instead of directly calculating the perimeter from the side lengths of the hexagon.
Answer 2 To solve this problem correctly, need understand the relationship between the equilateral trian- gles that make up the regular hexagon and how to calculate its perimeter accurately.
1. Identify the Perimeter of One Triangle: We are given that the perimeter one equilateral triangle is 21 inches. Since sides of an equilateral triangle are equal, each side of the triangle = inches.
2. Determine the Perimeter of the Hexagon: A regular hexagon is made up of six equilateral triangles. Each side of the hexagon is the same length as a side of these triangles.
3. Calculate the Total Perimeter:
The perimeter of the hexagon is the of the lengths of its six sides. Since
each side is 7 inches long, the perimeter of the hexagon is 6 x 7 42 inches.
Thus, the perimeter of the regular hexagon is 42 inches.
Critic iteratively helps refine the answer
Problem
Let be complex number such that 2
1 and x
A 1. Compute
1
1
2 +
+ 2 +
222
Answer
Given that is a complex number satisfying
25
and
+ 1, we need compute the expression
1
1
2
+
22
2
22
First, note that since 25 = 1, is fifth root of unity. The fifth roots of unity given by 2 for 0, 2, 4. Since 2 * 1, we have 25/i/5 for k 1,2, 3.4. Now, let's compute the expression step-by-step:
1. Compute z
+
Since z is unit magnitude complex number (as it is root of unity), |2| 1, and thus ž = Therefore,
25

---

Reinforce LLM Reasoning through Multi-Agent Reflection
2. Compute 2² :
22
+
= x² +
+ 1
x²
x²
x2
Since 2² is also a unit magnitude complex number, x²
Therefore,
23 =
+
x² + 22
22
22
3. Combine the results:
1 + (z² )
1
+
22
(=
2 2²
22
2² + and knowing that the sum of all fifth roots of unity is zero (as they form geometric series with sum to zero), we have:
Since + = = 7 and 2² + 1/2
+
Thus, the final answer is:
Feedback Upon reviewing the previous response, noticed a few points that need attention:
1. Sum of Fifth Roots of Unity: The statement that the sum of all fifth roots of unity zero correct. However, the reasoning provided in the solution does explicitly this property correctly. The of the fifth roots of unity is indeed zero, but the solution should clearly state this and use it to simplify the expression.
2. Simplification of the Expression: The solution does not clearly show how the expression 2 1 + 2 +
simplifies using the properties of the fifth roots of unity. The steps should be more detailed to show the intermediate calculations.
3. Use of Conjugates: The solution mentions the use of conjugates but does not explicitly show how the conju- gates are used in the simplification process. This could be made clearer for better understanding.
4. Final Calculation: The final calculation should be more explicit in showing how the sum of the terms equals zero. The solution should clearly state that the sum of the fifth roots of unity (excluding 1) is zero, and thus the expression simplifies to zero.
By addressing these points, the solution can be made more accurate and complete.
Answer 2 Given that is complex number satisfying ~
and + need compute the expression
1
1
2 +
+
2
+
2
First, note that since 25 1, is fifth root of unity. The fifth roots of unity given by 2nki/5 for 1, 2, 4. Since * 1, we have 2 for k 3,4,
Now, let's compute the expression step-by-step:
1. Sum of Fifth Roots of Unity: The sum of fifth roots of unity is zero. This can shown using the formula for the sum of geometric series:
1 1+ +
Since * 1, we exclude from our consideration.
26

---

Reinforce LLM Reasoning through Multi-Agent Reflection
2. Compute + ::
+
2
Since a unit magnitude complex number (as is a root of unity), |2| 1, and thus 1. Therefore,
3. Compute =²² gz:
22
x² + +2 2²
Since 2² is also unit magnitude complex number,
x² z Therefore,
2²
x² 2 22
4. Combine the results:
22 (2 + (2² +2)
2
Since the sum of all fifth roots of unity (excluding zero, we have:
2 = + 2 + xマ
Thus, the final answer is:
Feedback Upon reviewing the previous response, noticed a few points that need clarification or correction:
1. Sum of Fifth Roots of Unity: The statement that the sum of all fifth roots of unity is zero is correct. However, the sum of the fifth roots of unity excluding is not necessarily zero. The sum of the fifth roots of unity excluding is actually -1.
2. Compute 2 + and 2² + : The steps to compute z + and 2² + 3 are correct, but the final combination step needs to be revisited based on the correct sum of the fifth roots of unity excluding 1.
&
3. Final Combination: The final combination step should consider the correct sum of the fifth roots of unity excluding 1, which is 11
These points highlight the need to correct the final combination step to ensure the accuracy of the solution.
Answer 3 Given that is complex number satisfying ف and + need compute the expression
1 2 +
1
= +
22
First, note that since 25 1, fifth root of unity. The fifth roots of unity are given by 2nki/5 for 0, 1, 4. Since 2 * 1, we have 2 for le 3,4, Now, let's compute the expression step-by-step:
1. Sum of Fifth Roots of Unity: The sum of all fifth roots of unity is zero. This can be shown using the formula for the sum of geometric series:
1 1 # 1
22 + 3 +
Since z * 1, we exclude from our consideration. The sum of the other four fifth roots of unity is:
+ 2 +
27

---

Reinforce LLM Reasoning through Multi-Agent Reflection
2. Compute + :
+
2
Since z is a unit magnitude complex number (as it is a root of unity), | = 1, and thus ? = 1.- Therefore,
+ = =
3. Compute 2² +
2²
=² x2
22
Since 2² is also unit magnitude complex number,
22
zz Therefore,
2 + 22
2² + 22
4. Combine the results:
2 + 2
(=
(=+ (=² +22)
We know that 2 + 22 + 23 + + 1. Since 2 + and -2 + z are real numbers (as they are sums of complex conjugates), we can write:
ス+ 2 =
Thus, the final answer is:
Failure Pattern: Over-thinking easy tasks
Problem Marissa is hiking 12-mile trail. She took hour to walk the first 4 miles, then another hour to walk the next two miles. If she wants her average speed to be 4 miles per hour, what speed (in miles per hour) does she need to walk the remaining distance?
Answer To solve this problem, we need to calculate the total time Marissa has spent hiking and the distance she has covered so far. Then, we can determine the speed she needs to maintain for the remaining distance achieve an average speed of miles per hour.
1. Calculate the distance covered far: Marissa has walked the first miles in 1 hour and the next miles in hour. So, the total distance covered so far is:
4 miles + 2 miles 6 miles
2. Calculate the time spent so far: Marissa has spent hour walking the first miles and another hour walking the next miles. So, the total time spent so far is:
hour 1 hour hours
3. Determine the total time required to complete the trail at an average speed of miles per hour: The total distance of the trail is 12 miles. To maintain an average speed miles per hour, the total time required to complete the trail is:
12 miles miles per hour
= 3 hours
28

---

Reinforce LLM Reasoning through Multi-Agent Reflection
4. Calculate the remaining time complete the trail: Marissa has already spent hours hiking. Therefore, the remaining time to complete the trail is:
3 hours hours hour
5. Calculate the remaining distance: The total distance the trail is miles, and Marissa has already covered miles. Therefore, the remaining distance is:
12 miles 6 miles miles
6. Determine the speed required to cover the remaining distance in the remaining time: To cover the remaining miles in I hour, Marissa needs walk speed of:
miles
1 hour = 6 miles per hour
Therefore, Marissa needs to walk at speed of 6 miles per hour to cover the remaining distance and maintain an average speed of miles per hour for the entire trail.
Feedback Upon reviewing the previous response, noticed a few points that need attention:
1. Average Speed Calculation: The average speed for the entire trail should be calculated based on the total distance and the total time. The initial calculation of the total time required to complete the trail at an average speed of miles per hour is correct.
2. Remaining Distance and Time:
The caleulation of the remaining distance and the remaining time is correct. However, the speed required to cover the remaining distance in the remaining time should be re-evaluated to ensure it aligns with the average speed requirement.
3. Speed Requirement:
The speed required to cover the remaining distance in the remaining time should be recalculated ensure it meets the average speed requirement for the entire trail.
These points highlight areas where the solution might need further verification adjustment to ensure accu- racy and completeness.
Answer Upon reflecting on the previous response, let's correct and clarify the steps to ensure accuracy.
1. Calculate the distance covered so far: Marissa has walked the first miles in 1 hour and the next miles in hour. So, the total distance covered so far is:
miles + 2 miles = miles
2. Calculate the time spent so far: Marissa has spent hour walking the first 4 miles and another hour walking the next miles. So, the total time spent so far is:
hour + 1 hour = hours
3. Determine the total time required to complete the trail at an average speed of miles per hour: The total distance of the trail is miles. To maintain an average speed miles per hour, the total time required to complete the trail is:
12 miles
miles per hour 3 hours
4. Calculate the remaining time complete the trail: Marissa has already spent hours hiking. Therefore, the remaining time complete the trail is:
3 hours hours hour
29

---

Reinforce LLM Reasoning through Multi-Agent Reflection
5. Calculate the remaining distance: The total distance of the trail is miles, and Marissa has already covered miles. Therefore, the remaining distance is:
12 miles 6 miles miles
6. Determine the speed required to cover the remaining distance in the remaining time: To cover the remaining miles in hour, Marissa needs to walk at speed of:
6 miles hour
miles per hour
However, we need to ensure that this speed aligns with the average speed requirement for the entire trail. The average speed calculated as the total distance divided by the total time. Marissa has already walked 6 miles hours, which is an average speed of:
miles
= miles per hour
2 hours
To maintain an average speed miles per hour for the entire 12-mile trail, she needs to cover the remaining miles in:
6 miles
4 miles per hour 1.5 hours
This means Marissa needs to adjust her pace to cover the remaining 6 miles 1.5 hours, which is a speed of:
6 miles
= miles per hour
1.5 hours
Therefore, Marissa needs to walk speed of miles per hour to cover the remaining distance and maintain an average speed of miles per hour for the entire trail.
Failure Pattern: Over-pessimistiy feedback
The results of a cross-country team's training run are graphed below. Which student has the greatest average speed?
Problem
[asy] for int i 1: <= 7: ++i
draw
for (int i 1; <= 5; ++i )
draw (00,))--8,i))
draw ((-0.5.0) .0), linewidth (1));
draw (00,-0.5) -00,)) linewidth (1));
IaboS" (0,0), SW);
label
distance". (0, 3), W): (4, 0), S);
label
((12(1 4.5));
label (.85)* " "Evelyn", (1.25, 4.8), N);
30

---

Reinforce LLM Reasoning through Multi-Agent Reflection
(2.., 2.2)):
(scale (o4.2255..52)
(2.5, 2.2), S);
(scale
(4.25, 5.2), SE);
dot 2.8));
label (.85)*" "Debra", (5.6, 2.8), N);
( 1.4));
(scale (.85)** * "Angela", (6.8, 1.4), E);
[/asy]
Answer
[Resoning...] Comparing the average speeds, we find that Evelyn has the greatest average speed.
Evelyn
Feedback 1
Upon reviewing the previous response, I noticed a few errors and gaps: [Feedback...]
Answer 2
[Resoning...] Comparing the average speeds, we find that Carla has the greatest average speed.
Carla
Feedback 2
Upon reviewing the previous response, I noticed a few errors and gaps: [Feedback...]
Answer 3
[Resoning...] Comparing the average speeds, we find that Debra has the greatest average speed.
Debra
Upon reviewing the previous response, noticed a few errors and gaps: [Feedback..] Answer [Resoning...] Comparing the average speeds, we find that Angela has the greatest average speed.
Feedback 3
Angela
31


Reviews:
[{"review_id": "QwpS1eTaBw", "reviewer": "Reviewer_Hay1", "summary": "This paper introduces DPSDP (Direct Policy Search by Dynamic Programming), a reinforcement learning algorithm for training multi-agent LLM systems to iteratively refine responses on reasoning tasks. The authors formulate the multi-turn refinement process as a Markov Decision Process with an actor that generates answers and a critic that provides feedback. The algorithm uses direct preference learning on self-generated data to optimize both agents together. A key contribution is the practical adaptation that allows models to generalize to out-of-distribution horizons at test time through a simplified state representation. Theoretical analysis shows that DPSDP achieves performance equivalent to any comparator policy covered in the training distribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper! \n\n# Theoretical Claims\n**Q: Analysis of practical algorithm**\n\nWe analyze the Q-value approximation in practical DPSDP, where only one feedback and refinement step is used during training (Algorithm 2), assuming $H=3$. Let $\\hat{\\pi}$ be the resulting policy, and let $\\widetilde{Q}_h^{\\hat{\\pi}}$ denote the estimated Q-values, replacing $Q_h^{\\hat{\\pi}}$ in Assumption 2.\n\nWe define advantage function as $A_h^\\pi(s_h,a_h)=Q_h^{\\pi}(s_h,a_h)-V_h^{\\pi}(s_h)$, and $\\widetilde{A}_h^\\pi(s_h,a_h)=\\widetilde{Q}_h^{\\pi}(s_h,a_h)-\\mathbb{E}\\_{a_h\\sim \\pi(\\cdot\\mid s_h)}[\\widetilde{Q}_h^{\\pi}(s_h,a_h)]$. \n\nAs detailed in Section 3.3 (*Estimation of Q-values*), we define the estimated Q-values as follows:\n1. At $h=2$, the estimated Q is exact.\n2. At $h=1$, the estimated Q is: $\\mathbb{E}\\_{a_2 \\sim \\pi_\\mathsf{ref}(\\cdot \\mid s_2)}[r(s_3)] = Q_1^{\\pi_\\mathsf{ref}}(s_1, a_1)$.  \n\n   We define the approximation error:\n   $$\n   \\Delta = \\mathbb{E}_{s_h \\sim d_h^{\\pi^\\star},a_h \\sim \\pi^\\star(\\cdot \\mid s_h)}[A_h^{\\hat{\\pi}}(s_h,a_h) - \\widetilde{A}_h^{\\hat{\\pi}}(s_h,a_h)]\n   $$\n3. At $h=0$, we have $\\widetilde{Q}_0^{\\hat{\\pi}_1}(s_0, a_0) = r(s_1) + \\frac{H-1}{2} = Q_0^{\\pi^\\star}(s_0, a_0)$. Therefore, \n\n   $$\n   \\mathbb{E}\\_{a_h \\sim \\pi^\\star(\\cdot \\mid s_h)}[A_h^{\\hat{\\pi}}(s_h, a_h)] \n   \\approx \\mathbb{E}\\_{a_h \\sim \\pi^\\star(\\cdot \\mid s_h)}[A_h^{\\pi^\\star}(s_h, a_h)] = 0,\n   $$  \n   where the last equality follows from the definition of $A_h^{\\pi}$.\n\nFollowing steps in Appendix C.1, we obtain the approximate upper bound by adding $|\\Delta|$ to the theoretical bound.\n\nTo assess the impact of $|\\Delta|$, we performed an ablation using the step-by-step DPSDP variant (Appendix E.5), which uses $Q^{\\pi_2}$ in the DPO-style loss. The results showed no significant performance gain, indicating that $|\\Delta|$ has minimal effect. For simplicity and efficiency, we use the original version in the main paper.\n\n# Weaknesses\n\n**Q: Other model sizes and families**\n\nWe further tested our algorithm on Qwen2.5-3B to demonstrate its effectiveness over different model sizes and model families. \n\nTask|Model|Pass@t1|Maj@t3|Maj@t5|Pass@t5\n-|-|-|-|-|-\nMATH500|Qwen2.5-3B|57.6|50.0|48.0|58.6\n||SFT|60.0|60.4|60.4|64.6\n||DPSDP|60.4|61.6|62.0|65.2\nGSM8K|Qwen2.5-3B|78.6|76.2|75.2|79.4\n||SFT|79.1|78.4|77.7|81.5\n||DPSDP|79.9|80.2|79.9|84.2\nMMLU-ProMath|Qwen2.5-3B|47.4|42.0|41.2|48.4\n||SFT|50.9|51.2|51.4|56.0\n||DPSDP|52.6|53.2|53.2|57.1\nOlympiad-Bench|Qwen2.5-3B|24.0|22.6|22.0|24.5\n||SFT|23.9|24.3|24.8|26.4\n||DPSDP|24.0|23.9|24.0|26.0\n\nOur results show that the proposed algorithm generalizes effectively on smaller models such as Qwen2.5-3B. Furthermore, DPSDP-trained models demonstrate strong generalization capabilities on out-of-distribution benchmarks, such as MMLU Pro Math.\n\n**Q: Other reasoning tasks**\n\nWhile our focus has been on mathematical reasoning to showcase the effectiveness of our approach, we anticipate that similar performance gains would extend to other complex reasoning tasks. Exploring these tasks presents an exciting direction for future research.\n\n**Q: SCoRe and RISE baselines**\n\nSee reply to Reviewer X1o306, Table 1.\n\n# Questions For Authors\n\n**Q: Define the optimal number of refinement iterations**\n\nWe scaled the number of refinement iterations up to 10. Our results show that accuracy begins to plateau after approximately 5 to 7 iterations as presented in reply to Reviewer jmss, Table 3. \n\n\n**Q: Comparison on training and inference cost with baselines**\n\nBoth DPSDP and baselines have similar time complexity, dominated by quadratic-scaling attention mechanism during forward and backward passes. On 4× H100 80GB GPUs, DPSDP and STaR-DPO each took ~6 hours to train (5h 45m and 5h 55m, respectively), while STaR completed in 3h 10m. Inference costs are comparable, with responses all refined through 4 iterations.\n\n**Q: How essential is the preliminary training (SFT) stage for DPSDP's performance?**\n\nPrior work [1,2] has shown that SFT is essential before reinforcement learning RL, especially when the base model struggles to follow instructions.\n\nOur results support this: we trained DPSDP on Mistral-8B-instruct both with and without SFT. Without SFT, performance degrades notably on  MATH and OlympiadBench, and iterative refinement offers little benefit—as seen in the small gap between accuracy@t1 and majority@t5 or pass@t5.\n\nTask|ModelVariant|Pass@t1|Maj@t5|Pass@t5|\n-|-|-|-|-|\nGSM8K|DPSDP|87.8|89.1|92.7|\n||w/o SFT stage|90.6|90.8|90.9|\nMATH500|DPSDP|58.2|63.2|70.0|\n||w/o SFT stage|52.6|53.8|54.4|\nMMLU-ProMath|DPSDP|53.1|54.2|64.3|\n||w/o SFT stage|54.1|54.4|55.5|\nOlympiadBench|DPSDP|25.8|27.0|32.9|\n||w/o SFT stage|26.0|26.1|26.7|\n\nThese findings highlight the importance of SFT in enabling models to give and use feedback effectively.\n\n[1] Recursive Introspection: Teaching Language Model Agents How to Self-Improve\n\n[2] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training", "claims_and_evidence": "The paper's claims are generally well-supported by empirical evidence and theoretical analysis. The authors make two principal claims: (1) DPSDP improves reasoning performance through multi-agent interaction; and (2) the approach generalizes to out-of-distribution benchmarks. These claims are substantiated through comprehensive experiments across multiple model families and mathematical reasoning benchmarks.", "methods_and_evaluation_criteria": "The methods and evaluation criteria employed in this paper are appropriate and well-suited to the problem of improving LLM reasoning through multi-agent collaboration.", "theoretical_claims": "I reviewed Theorem 1 and its supporting Lemmas (Lemma 2 and Lemma 3), which establish the theoretical performance guarantee for the DPSDP algorithm. The proofs appear to be mathematically sound with appropriate use of Markov Decision Process theory.\n\nOne minor issue is that the formal relationship between the practical implementation (Algorithm 2) and the theoretical version (Algorithm 1) could be more explicitly addressed in the theoretical analysis, particularly regarding how the practical approximations affect the theoretical guarantees.", "experimental_designs_or_analyses": "The experimental design and analyses in this paper are generally sound. The authors evaluate their approach on appropriate mathematical reasoning benchmarks (MATH 500, GSM8K, MMLU-Pro Math, Olympiad Bench) using standard metrics (pass@turn1, maj@turn5, pass@turn5) that effectively measure both initial and refined performance. \n\nOne minor issue is while the authors mention hyperparameter selection in the appendix, a more systematic hyperparameter sensitivity analysis would strengthen the experimental rigor.", "relation_to_broader_scientific_literature": "This paper builds on and extends several key research directions in the LLM reasoning literature. The verify-and-improve paradigm it explores connects to prior work on self-correction and external feedback mechanisms. The formulation as an MDP extends the application of reinforcement learning approaches to LLM alignment, building particularly on Direct Preference Optimization (Rafailov et al., 2023) and Policy Search by Dynamic Programming (Bagnell et al., 2003).", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Strengths**\n1. The paper effectively adapts PSDP to the context of LLM-based agent training, creating a theoretically grounded yet practical algorithm for multi-agent response refinement.\n2. The authors develop several practical modifications to make the algorithm computationally efficient, particularly the Markovian state reformulation that enables generalization to longer refinement horizons at test time than seen during training.\n3. The method shows consistent improvements across different base models and benchmarks.\n\n**Weaknesses**\n1. While the experiments cover two model families (Ministral and Llama), they only use 8B parameter versions. Testing with a wider range of model sizes like meta-llama/Llama-3.2-3B-Instruct, meta-llama/Llama-3.3-70B-Instruct would better demonstrate scalability and generalizability of the approach.\n2. The evaluation focuses exclusively on mathematical reasoning tasks. Including other reasoning domains (e.g., logical reasoning, coding) would provide a more comprehensive assessment of the method's capabilities.\n3. While the paper compares against relevant baselines, it would benefit from comparisons to SCoRe[1] and RISE[2].\n\n[1] Kumar, Aviral, et al. \"Training language models to self-correct via reinforcement learning.\" arXiv preprint arXiv:2409.12917 (2024).\n[2] Qu, Yuxiao, et al. \"Recursive introspection: Teaching language model agents how to self-improve.\" Advances in Neural Information Processing Systems 37 (2024): 55249-55285.", "other_comments_or_suggestions": "- line261 h=0, H-1", "questions_for_authors": "1. The paper demonstrates results with five answer attempts, but it's unclear how this number was determined. What process did you use to select the optimal number of refinement iterations?\n2.  The paper lacks a detailed comparison of computational efficiency between DPSDP and baseline methods like STaR and STaR-DPO. Could you provide quantitative metrics on training time, inference costs?\n3. The paper mentions using oracle models (Mistral-Large-Instruct-2411 and Llama-3.3-70B-Instruct) for generating high-quality feedback and refined answers during the preliminary training phase. How essential is this oracle guidance for DPSDP's performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6VIvFerTvN", "reviewer": "Reviewer_jmss", "summary": "- The focus of the paper is on verification and refinement with an actor and critic model, using a method that trains on self-generated data\n\t- the actor model generates and refines responses based on feedback from a critic\n\t- the actor and critic are jointly trained with RL\n- The authors propose a dynamic programming-based approach to optimizing the policies of the actor and critic jointly\n\t- an analysis of the approach is given in Theorem 1\n- The approach is evaluated in practice on two LLMs (Ministral-8B-Instruct and Llama-3.1-8B-Instruct), which are first SFT'd on feedback data and then finetuned wtih DPSDP. \n\t- evaluation is done on four math datasets, two of which are out of domain\n\t- the method is compared against STaR and STaR-DPO in some of the settings\n- On Ministral-8B, DPSDP outperforms STaR-DPO on MATH 500 and MMLU-Pro MATH. It performs comparably on GSM8K and Olympiad Bench.\n\n## update after rebuttal\nThe rebuttal by the authors has helped address several of my larger concerns and I have chosen to increase my score to a 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the efforts in reviewing our paper! We will take your suggestions, fix the typo and revise the presentations accordingly in the next revision!\n\n# Methods And Evaluation Criteria\n\n**Q: unclear how $a_1$ and $a_2$ are labeled as chosen and rejected actions**\n\nAlgorithm 1 presents the theoretical version of our method and does not explicitly label $a_1$ and $a_2$ as chosen or rejected. Instead, it assumes access to the Q-value function, and the Q-values of $a_1$ and $a_2$ are directly used in the cross-entropy loss (see line 176, left column).\n\nIn the practical implementation (Algorithm 2), we estimate the Q-values as described in Section 3.3, under *Estimation of Q-values*. Specifically, we approximate the Q-values based on the correctness of responses. For each action pair $(a_1, a_2)$, the action with the higher estimated Q-value is labeled as \"chosen,\" and the other as \"rejected.\" We justify the reliability of this estimation both intuitively (Section 3.3) and empirically (Section 4 and Appendix E.5).\n\n**Q: Notation issues -- definition of $d_h^{\\pi}$**\n\nWe adopt standard notation from the reinforcement learning literature, where $d_h^{\\pi}$ denotes the distribution over the state space at step $h$ when following policy $\\pi$ (see lines 97–98, right column). Specifically, when $h = 0$, $d_0^{\\pi}$ represents the initial state distribution—i.e., the distribution over prompts drawn from the prompt set or provided by users. When $\\pi = \\pi_\\mathsf{ref}$, $d_h^{\\pi_\\mathsf{ref}}$ refers to the state distribution at step $h$ under the reference policy $\\pi_\\mathsf{ref}$. \n\n# Experimental Designs Or Analyses\n\n**Q: Baseline implementations with Llama-based models**\n\nTo further validate the effectiveness of our algorithm, we benchmarked it against STaR and STaR-DPO using two additional base models: Llama 3.1-8B-Instruct and Qwen 2.5-3B. We also replicated the single-agent setup using Llama-based models as discussed in Section 4.3. Across both settings, our algorithm consistently outperforms the baselines, demonstrating robust and superior performance.\n\n- Llama\n    |Task|Model|pass@t1|maj@t5|pass@t5|\n    |-|-|-|-|-|\n    |MATH|DPSDP(Llama)|55.8|58.4|62.0|\n    ||STaR|50.8|52.2|56.8|\n    ||STaR-DPO|54.2|55.6|59.2|\n    ||Single-Agent|53.4|54.8|58.0|\n    |GSM8K|DPSDP(Llama)|87.5|88.4|91.2|\n    ||STaR|83.6|81.3|87.5|\n    ||STaR-DPO|87.5|87.4|90.3|\n    ||Single-Agent|87.9|87.6|90.4|\n    |MMLU-ProMath|DPSDP(Llama)|56.6|58.0|62.1|\n    ||STaR|53.8|54.6|58.5|\n    ||STaR-DPO|54.8|55.0|60.3|\n    ||Single-Agent|56.1|57.3|62.0|\n    |OlympiadBench|DPSDP(Llama)|22.4|23.0|25.1|\n    ||STaR|20.5|20.3|22.4|\n    ||STaR-DPO|20.9|21.5|24.3|\n    ||Single-Agent|23.0|21.5|25.1|\n\n- Qwen\n    |Task|Model|pass@t1|maj@t5|pass@t5|\n    |-|-|-|-|-|\n    |MATH|DPSDP(Qwen)|60.4|62.0|65.2|\n    ||STaR|59.0|59.6|64.8|\n    ||STaR-DPO|60.4|60.2|64.8|\n    |GSM8K|DPSDP(Qwen)|79.9|79.9|84.2|\n    ||STaR|80.3|79.5|83.7|\n    ||STaR-DPO|79.4|78.9|82.6|\n    |MMLU-Pro Math|DPSDP(Qwen)|52.6|53.2|57.1|\n    ||STaR|51.9|51.8|56.8|\n    ||STaR-DPO|51.2|52.3|55.9|\n    |OlympiadBench|DPSDP(Qwen)|24.0|24.0|26.0|\n    ||STaR|23.3|22.6|24.8|\n    ||STaR-DPO|23.1|22.8|28.9|\n\n# Questions For Authors\n\n**Q: Stopping Criterion and Overrefinement**\n\nWe did not implement a specific stopping criterion for the refinement process, aside from a fixed limit on the number of refinement iterations, set to 5 in our experiments. To study the potential issue of over-refinement, we extended the number of iterations to 11. Our results show that accuracy generally improves with more refinement steps—reflecting the benefit of increased test-time computation—until it plateaus around 5 to 7 iterations. Beyond that, performance remains stable and shows minimal degradation, suggesting over-refinement is not a significant concern in practice.\n\nTask|Model|maj@t1|maj@t3|maj@t5|maj@t7|maj@t9|maj@t11\n-|-|-|-|-|-|-|-\nMATH|Ministral|58.2|61.8|63.2|63.6|63.6|63.6\n||Llama|55.8|57.8|58.4|58.4|58.2|58.4\nGSM8K|Ministral|87.8|89.1|89.1|89.2|89.5|89.2\n||Llama|87.5|88.2|88.4|88.6|88.6|88.4\n\nTask|Model|pass@t1|pass@t3|pass@t5|pass@t7|pass@t9|pass@t11\n-|-|-|-|-|-|-|-|\nMATH|Ministral|58.2|68.2|70.0|70.8|70.8|70.8\n||Llama|55.8|61.2|62.0|62.0|62.0|62.4|\nGSM8K|Ministral|87.8|91.6|92.7|93.0|93.3|93.3\n||Llama|87.5|90.7|91.2|91.5|91.6|91.7\n\nWe further conducted analysis on failure patterns in reply to Reviewer 2FHw, Table 1. In qualitative analysis beyond what is reported in the paper, we observed that *iterative refinement can help correct over-refined answers*. For instance, a correct answer may be incorrectly altered due to faulty feedback, but later iterations may recover the correct answer, leading to correct majority voting answer.\n\nWhile iterative refinement helps mitigate over-refinement, it does not entirely eliminate the risk. As a potential solution, we propose monitoring performance on a validation set at each refinement step and stopping early if accuracy begins to decline.", "claims_and_evidence": "- The authors claim the method outperforms baselines across datasets. This is only partially supported, as the baseline is only implemented for one of the two models.", "methods_and_evaluation_criteria": "The method is not clear, and requires a bit more intuition. Specifically, the explanation in L153-159 (right) does not make it clear how the pairwise data is collected and verified, i.e. how are $a^1$ and $a^2$ determined? According to Algorithm 1, they are both sampled from $\\pi_{\\text{ref}}$ but it's not clear how one is marked as preferred and the other as dispreferred.\n\nNotational issues:\n- as far as I can tell, $d_0^\\pi$ and $d_h^{\\pi_{\\text{ref}}}$ are not defined (line 126, equation (1)). I assume these are trajectories?", "theoretical_claims": "Based on Assumptions 1 and 2, the authors claim that the policy resulting from their method competes with ``any policy under single-policy concentrability and bounded in-distribution generalization error.''. \n\nTo me, this theoretical claim does not add much as-is; the success of this kind of paper rests on its results in practice.", "experimental_designs_or_analyses": "I am concerned by the fact that the baseline was only implemented on one model. What was the reason for omitting the STaR and STaR-DPO baseline for Llama3.1 8B?", "relation_to_broader_scientific_literature": "The related work section is in the appendix but is fairly complete, covering most relevant work.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Jointly training the actor and critic for reasoning refinement is an interesting direction and seems to be novel.", "other_comments_or_suggestions": "- I think there's some kind of spacing issue with L264-274\n- Overall the spacing/presentation of the paper could be refined.\n\nTypos:\n- L154 (right): cross-entropy\n- L156 (right): a collected pairwise dataset.\n- L308: challenging\n- L314: Olympiad-level", "questions_for_authors": "What is the stopping criterion of the refinement process? With refinement, there is often a problem of over-refinement, where a correct answer is refined and turned incorrect. Does training the models to perform iterative refinement address this? At test time, do models know when to stop/have you measured overrefinement?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gZJt9PgrBt", "reviewer": "Reviewer_2FHw", "summary": "This paper proposes a new reinforcement learning algorithm, DPSDP, to enhance the mathematical reasoning capabilities of large language models using a multi-agent approach involving an actor and a critic. The method instantiates two LLMs as actors and critics to perform self-reflection-style reasoning, collecting preference data by sampling from the models. This process consists of at most two rounds: posing a question, generating an initial response, providing feedback, and generating a revised response. The preference is estimated by rolling out the policy.\n\nThe DPSDP algorithm is evaluated on four benchmarks, covering both in-distribution and out-of-distribution settings, using two base models: Mistral-8B-It and Llama-3.1-8B-It. The results show improvements in accuracy across benchmarks and settings, outperforming baselines such as STaR and STaR-DPO. The authors also conduct an ablation study to analyze the impact of single-agent versus multi-agent approaches, Markovian versus non-Markovian formulations, and generative versus non-generative critics.\n\n## Response\nI maintain my view that this paper is suitable for acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper!\n\n# Claims And Evidence\n\n**Q: Analysis is not highly related to the practical algorithm**\n\nWe provide further analysis on how approximation in the practical algorithm affects the theoretical results in reply to Reviewer Hay1. \n\n\n**Q: More detailed metrics and failure pattern analysis**\n\nWe adopt the metrics p1@t1, m1@t5, and p1@t5 in line with prior work [1], and provide additional evaluation details for a more comprehensive analysis.\n\nFirst, we scale up the number of test-time refinement iterations, as shown in reply to Reviewer jmss, Table 3.\n\nNext, we analyze the dynamics of accuracy over the course of refinement. Using models based on Ministral-8B-Instruct as a representative example, we plot the changes in accuracy across iterations in Figure 3. For each refinement step, we define $\\Delta^{c \\rightarrow i}$ as the proportion of problems that change from correct to incorrect after refinement, and $\\Delta^{i \\rightarrow c}$ as the proportion that transition from incorrect to correct. \n\n|Iteration|$\\Delta^{i\\rightarrow c}$|$\\Delta^{c\\rightarrow i}$|\n|-|-|-|\n|t1→t2|7.8|4.0|\n|t2→t3|4.0|3.6|\n|t3→t4|3.4|2.8|\n|t4→t5|3.0|2.8|\n|t5→t6|1.8|1.6|\n|t6→t7|2.0|2.4|\n|t7→t8|1.4|1.0|\n\nThe table illustrates two key observations:\n\n1. $\\Delta^{i \\rightarrow c}$ consistently exceeds $\\Delta^{c \\rightarrow i}$, indicating that the refinement process is generally beneficial.\n\n2. Both $\\Delta^{i \\rightarrow c}$ and $\\Delta^{c \\rightarrow i}$ decrease as the number of iterations increases, suggesting an initial exploratory phase followed by stabilization in later refinement steps.\n\nWe conducted the same analysis on models based on Llama and Qwen and observed similar patterns. Due to space constraints, we omit those results here.\n\nIn addition to the qualitative analysis presented in Appendix E.6, we identified several notable failure patterns:\n\n- Answer enumeration: The critic repeatedly provides negative feedback, prompting the actor to cycle through different answers at each turn—effectively enumerating possible solutions.\n\n- Answer degradation: The critic incorrectly assigns negative feedback, leading the actor to progressively degrade a previously correct answer. However, this over-refinement issue is relatively rare (as evidenced by small $\\Delta^{c \\rightarrow i}$) and can be mitigated by later refinement steps.\n\n- Incorrect feedback tolerance: Occasionally, a correct answer is incorrectly revised due to faulty feedback. Yet, subsequent iterations can recover the correct answer and lead to correct final answer by majority voting across all turns, helping to mitigate the effects of over-refinement.\n\n# Experimental Designs Or Analyses\n\n**Q: diverse base models**\n\nWe conducted DPSDP on Qwen2.5-3B and the results are presented in reply to Reviewer Hay1, Table 1. \n\n**Q: Investigate whether models can self-improve**\n\nWhile earlier work suggested that models are unable to self-improve, recent studies—such as SCoRe and RISE—have demonstrated that large language models (LLMs) can develop self-improvement capabilities when properly trained. To further explore this, we conducted an ablation study in which a single model served as both the actor and critic. The results are shown in Table 1 under the row labeled Single-Agent. A detailed comparison between the single-agent and multi-agent setups is provided in Section 4.3, under the paragraph titled *Single-Agent vs. Multi-Agent*.\n\nOur findings are consistent with those of SCoRe and RISE: a single model can indeed self-improve. However, its performance is generally weaker than that of the multi-agent system, particularly on more challenging benchmarks. We replicated the single-agent setup using Llama-based models and presented the results in reply to Reviewer jmss, Table 1. Our conclusion remains the same. \n\n\n\n**Q: SFT baseline**\n\nOne of our baselines, STaR, serves as the SFT counterpart to our algorithm. To ensure a fair comparison, STaR was implemented using the same SFTed models and trained on the identical prompt set. However, as discussed in Section 4.2, STaR fails to enable effective self-improvement in models.\n\n# Questions For Authors\n\n**Q: Difference between Markovian and non-Markovian**\n\nIn Section 3.3, under the paragraph titled Optimizing Iterative Refinement with Reduced Complexity, we define the transition function $\\delta(s_h, a_h)$, which reflects a Markovian setting. This design includes only the most recent answer and feedback in the prompt, removing all prior conversational history. The motivation behind this choice is the heuristic that recent context is more informative and relevant than earlier interactions.\n\nIn contrast, an alternative approach includes the entire conversation history—i.e., all previous responses and feedback—in the prompt. This setting diverges from the Markov Decision Process (MDP) we defined and is therefore referred to as non-Markovian.\n\n[1] Recursive Introspection: Teaching Language Model Agents How to Self-Improve", "claims_and_evidence": "In general, I think the theoretical analysis is not highly relevant to the practical algorithm because extensive modifications are made to ensure feasibility. Aside from this point, the other claims are reasonable.", "methods_and_evaluation_criteria": "This method makes sense to me. However, I think the evaluation metrics need more validation. Why are m1@p5 and p1@t5 appropriate metrics? Have you considered using 10 instead? Additionally, for the accuracy of m1@p5 and p1@t5, could you analyze the failure patterns in more detail? I would appreciate the metrics and analysis being as well-explained as in the paper \"Self-Rewarding Correction for Mathematical Reasoning\".", "theoretical_claims": "I didn't check the proofs.", "experimental_designs_or_analyses": "My main concern is the experimental section. The study primarily focuses on Ministerial-8B-It, with only a few experiments on Llama-3.1-8B-It. As a result, the experiments for Llama-3.1-8B-It seem incomplete. Additionally, Ministerial-8B-It is not a strong mathematical model. Many models, such as Qwen , perform better, and there are also models fine-tuned on mathematical datasets like deepseek-math and Qwen-math. These models would provide a more reasonable baseline for improvement. Furthermore, some papers (e.g., https://arxiv.org/abs/2310.01798) suggest that large language models cannot self-improve. An ablation study should be conducted to address this concern. (I am also unsure how supervised fine-tuning (SFT) was tested in the experiments—perhaps this concern has already been addressed.)", "relation_to_broader_scientific_literature": "This contribute to the self-improvement and self-correction topic of LLM. It also contribute a new preference training method.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "If the author could address my concern about the experiment part, I would be happy to increase the score.", "questions_for_authors": "I am a bit unclear about the difference between Markovian and non-Markovian cases. What is the context for each?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5vxhqvDyYO", "reviewer": "Reviewer_X1o3", "summary": "This paper introduces DPSDP (Direct Policy Search by Dynamic Programming), a algorithm designed to enhance the reasoning abilities of large language models by utilizing a multi-agent system. This paper concludes that DPSDP provides a robust solution for refining reasoning tasks in LLMs, allowing them to generate more accurate responses through iterative refinement and effective collaboration between multiple agents.\n\n## update after rebuttal\nGiven the rebuttal and discussions with the authors, my main concern of the fair comparison in the experiments is not fully addressed. I still think this is a borderline work, so I will keep my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper!\n\n# Weaknesses\n\n**Q: DPSDP does not resemble an RL algorithm**\n\nWe adapt our algorithm from PSDP, a classic reinforcement learning method, and formulate iterative refinement as a standard MDP (Section 2). As shown in Algorithm 1, we optimize each step in reverse via policy rollout and update, following the PSDP framework.\n\nA key insight is our estimation of ground-truth Q-values, which removes inter-iteration dependencies and simplifies implementation. Experiments validate this approach, and we compare the theoretical and practical versions in Appendix E.5.\n\n**Q: Other SOTA models baselines**\n\nWhile many works aim to improve LLM responses, few share our settings. For instance, SCoRe targets one-shot self-improvement without external feedback. The closest is RISE with an oracle, which assumes access to ground-truth correctness at test-time. \n\nWe include this variant as a baseline, re-implementing it by first training an SFTed Ministral-8B-It model (Section 3.4), followed by reinforcement learning on the same problem set used for our main models. \n\nTask|Model|pass@1|maj@t3|maj@t5|pass@t5\n-|-|-|-|-|-\nMATH|DPSDP|58.2|61.8|63.2|70.0\n||Oracle-RISE|59.2|64.2|65.4|65.8\nGSM8K|DPSDP|87.8|89.1|89.1|92.7\n||Oracle-RISE|88.9|92.1|92.6|92.9\nMMLU Pro Math|DPSDP|53.1|53.0|54.2|64.3\n||Oracle-RISE|52.8|59.8|61.3|62.4\nOlympiad-Bench|DPSDP|25.8|27.2|27.0|32.9\n||Oracle-RISE|25.8|30.3|30.6|30.9\n\nOur results show that our models achieve maj@t5 accuracies comparable to RISE on challenging benchmarks such as MATH and Olympiad. Our models consistently outperform RISE on pass@t5, indicating that the actor—guided by critic feedback—explores the solution space more actively rather than sticking to initial responses. \n\n\n**Q: Unfair comparison with SFT**\n\nIt is worthnoting that the rows labeled +SFT in Table 1 are not SFT baselines but intermediate results from preliminary training (Section 3.4), which our DPSDP models build upon. For a fair comparison with standard SFT, we evaluated against STaR baselines (Section 4.2), which used the same SFTed base models, problem set, number of trajectories, and training epochs. Results show that STaR fails to enable self-improvement, underscoring the effectiveness of our approach. \n\n**Q: Comparison with self-critique**\n\nTo highlight the benefits of the multi-agent setup, we replicated the full training—preliminary training and DPSDP—using a single model as both actor and critic. As shown in Table 1 (Single-Agent row) and discussed in Section 4.3, this setup consistently underperforms the multi-agent system, especially on harder benchmarks. We confirmed this across LLaMA-based models (see reply to Reviewer jmss, Table 1), and our conclusion holds.\n\n**Q: Concerns with Q-value estimation with Monte Carlo**\n\nQ-values reflect expected cumulative rewards rather than the immediate correctness of individual actions like feedback $a_1$. The correctness of $a_2 \\sim \\pi(\\cdot \\mid s_2)$ provides an unbiased estimate of the Q-value. A related example is DeepSeek-R1-Zero, which shows reduced readability during reasoning but achieves strong final performance.\n\n**Q: The soundness of using DPO as optimization method**\n\nOur core contribution is introducing PSDP for multi-agent LLM refinement. While we use DPO for optimization, it’s a modular component that can be replaced (e.g., with rejection sampling or KTO) without altering the overall approach.\n\n# Questions For Authors\n\n**Q: Fair comparison with SFT baseline**\n\nPlease see the response above. \n\n**Q: Single-Agent experiment with Llama-based models**\n\nWe present single-agent results using Llama-based in reply to Reviewer jmss, Table 1, and our findings consistently demonstrate the general superiority of the multi-agent system over the single-agent setup.\n\n**Q: Necessity of Preliminary Training (SFT)**\n\nSee the reply to Reviewer Hay1, Table 2.\n\n**Q: Additional analytical experiments to explore impact of multi-round critiques**\n\n1. We visualize the accuracy dynamics in Figure 3 and provide results for more refinement steps in reply to Reviewer jmss, Table 3. Further detailed metrics are also presented in reply to Reviewer 2FHw, Table 1.\n\n1. In reply to Reviewer 2FHw, we identify several failure patterns and illustrate how iterative refinement helps mitigate issues related to over-refinement.\n\n1. In Appendix E.1, we compare maj1@t5 and maj5@t1, showing that the performance gains stem from the refinement process itself rather than from a simple increase in test-time computation.\n\n**Q: Data processing**\n\nOur data filtering is intentionally simple and transparent to attribute performance gains to our algorithm rather than data curation. We remove duplicates and randomly sample problems for unbiased coverage. For the SFT dataset, we use an oracle model to generate refined answers and filter feedback (Section 3.4). To ensure diverse reasoning styles, we include trajectories from different model famalies as explained in Appendix D.1.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. The proof of Theorem 1 is reasonable in its approach, but the assumptions it relies upon may not hold in the context of LLMs, and some of the relaxations used in the proof may be too loose. Therefore, the theoretical guarantee provided by Theorem 1 may not be strong. More detailed analysis and experiments are needed to verify the actual performance of the DPSDP algorithm.", "experimental_designs_or_analyses": "Yes, I checked the soundness and validity of several experimental designs and analyses, focusing primarily on those presented in Sections 4.2 (Main Results) and 4.3 (Ablation Study), as well as some from Appendix E.  The experimental designs are generally sound and well-justified, with thorough ablations and relevant comparisons. The issues are primarily areas for potential more detailed analysis, rather than fundamental flaws. The paper provides good evidence for the effectiveness of DPSDP, given the constraints of the chosen problem domain.", "relation_to_broader_scientific_literature": "The paper's key contributions relate to several strands of existing scientific literature, building upon and extending prior work in specific ways:\n\n1. Intrinsic and External Self-Correction: Previous research explores both intrinsic self-correction (LLMs refining outputs without external help) and self-correction with external feedback. Intrinsic methods often involve prompting LLMs to reflect and revise, but some unrealistically assume access to correct answers. Others try training models for self-correction, finding supervised fine-tuning insufficient and exploring RL approaches, sometimes focusing on single-turn refinement. Multi-turn refinement has also been explored. However, LLMs often struggle with purely intrinsic self-correction. Research with external feedback frequently uses code generation scenarios with feedback from tests or compilers, or incorporates external tools. Some use feedback from other models, but typically treat the answer generator and feedback provider as separate entities, relying on fixed feedback or training a separate corrector. DPSDP addresses multi-turn refinement, but with a different approach and a theoretical guarantee. The LLM critic provides a flexible feedback space, unlike the restricted feedback in some previous work.\n\n2. Multi-Agent Systems in LLMs:  The paper builds on the growing interest in multi-agent LLM systems (Guo et al., 2024b; Motwani et al., 2024). It cites examples of both competitive (debate-style) and cooperative multi-agent systems. It specifically mentions works that use multi-agent systems for reasoning improvement. DPSDP is a multi-agent system with actor and critic. It differs from some prior work by having a joint training process using DPO. It also contrasts with Motwani et al. (2024), which uses a three-model system and focuses on a 2-turn refinement, by providing a theoretical guarantee and enabling multi-turn refinement.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "## Strengths：\n1. This paper establishes a strong theoretical foundation for DPSDP, providing a formal proof that, under specific conditions, the algorithm's performance can match any comparator policy within the training distribution. The proof is presented rigorously and appears sound.\n2. The experimental evaluation is adequate, utilizing multiple metrics (pass1@turn1, maj1@turn5, pass1@turn5) across two base models. This provides a comprehensive assessment of DPSDP's performance. The ablation studies in the Appendix further validate the effectiveness of key design choices.\n3. The paper demonstrates DPSDP's ability to improve LLM performance not only on in-distribution benchmarks but also on OOD data. This highlights the algorithm's generalization capabilities, particularly on challenging reasoning problems.\n\n## Weaknesses：\n1. While framed within a reinforcement learning context, DPSDP's two-stage training process (SFT + DPO) more closely resembles a supervised learning approach with refined training data. Crucially, the optimized model cannot be used for on/off-policy data generation and further iterative improvement, a hallmark of many RL algorithms. \n2. Although the paper compares DPSDP against several baselines, the comparison to recent SOTA methods is limited. While Appendix A mentions related works, a direct comparison with a contemporary, high-performing approach is missing.\n3. The paper demonstrates DPSDP's superiority over SFT. However, the SFT baseline is trained for only one epoch, likely preventing it from reaching convergence. DPSDP, in contrast, undergoes more extensive training. This discrepancy in training duration makes the comparison between DPSDP and SFT potentially unfair, as the observed performance gains might be attributed to the difference in training extent rather than the inherent advantages of the DPSDP strategy.\n4. From Table 1 to Table 4, I noticed that the authors did not conduct self-critique experiments for comparison. Given that the open-source community has recently demonstrated the significant effectiveness of self-criticism and reflection capabilities in single-agent systems, I suggest the authors consider including such experiments in their comparisons. While the multi-agent approach shows advantages, the potential of single-agent methods should not be overlooked.\n5. Regarding the method described in Sec 3.3, where Monte Carlo sampling of $a_2$'s accuracy is used to estimate the expected return of $a_1$, I have the following concerns: This estimation method may lead to an underestimation of $a_1$ Q-value. More critically, it might induce model hallucinations where $a_1$ generates meaningless content, yet $a_2$ still provides correct answers. I recommend the authors thoroughly investigate this potential issue and discuss possible solutions.\n6. In terms of optimization algorithm selection, while the introduction of DPO is reasonable, it lacks novelty. To enhance the soundness of using DPO as an optimization method, I suggest the authors include comparisons with classic reinforcement learning baseline methods in Table 1, such as reject sampling with finetuning, and Independent PPO. This would not only highlight the advantages of DPO but also provide readers with a more comprehensive evaluation of methods.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. The paper highlights the use of DPO as a key component of DPSDP. To rigorously demonstrate the advantage of DPO over SFT, could the authors provide experimental results comparing models trained with DPO and SFT under identical conditions? Specifically, this comparison should control for either the number of training steps or the amount of training data used, ensuring a fair evaluation of the two training paradigms. This would help isolate the specific contribution of DPO.\n\n2. The results in Table 1 show that the single-agent approach outperforms the multi-agent DPSDP on several metrics using the Mistral-8B-It base model. To further validate the effectiveness of the multi-agent design in DPSDP, could the authors provide corresponding results for the single-agent approach using the Llama-3.1-8B-It base model? This would allow for a more consistent comparison across different base models and strengthen the claims regarding the benefits of the multi-agent setup.\n\n3. Section 3.4 indicates that the initial SFT phase utilizes high-quality feedback and refined answers produced by a more capable model. Is DPSDP's performance necessarily dependent upon this reliance on data from a stronger model? If not necessarily, it could offer valuable insights into the algorithm's self-improvement capabilities and lessen its dependence on external resources.\n\n4. Regarding the extension of MDP policy in multi-round critique models, it is recommended that the authors conduct the following additional analytical experiments to explore in-depth the impact of multi-round critiques on model performance:\na) Plot the performance curve of the model as the number of critique rounds increases. This will help understand the evolution trend of model performance, including whether there are performance peaks or saturation points.\nb) Evaluate the impact of multi-round critiques on the quality of generated content. Pay particular attention to whether there are issues such as quality degradation, content repetition, or inconsistency.\nc) Under the same computational resource constraints, compare the performance differences between the baseline method (using self-consistency answer filtering) and the multi-round critique method. \nThis will help to more fairly assess the effectiveness of the MDP strategy.\nIt is suggested to present the results of these analytical experiments separately from Table 1 to provide a more comprehensive performance evaluation.\n\n5. Regarding the data filtering process for initializing actor and critic models, it is recommended to provide detailed explanations of the data filtering process in Appendix D.1, including but not limited to the following aspects:\na) Specific steps and criteria for data cleaning (such as removing duplicates, low-quality, or irrelevant data), and any specific language models or algorithms used to assist in the filtering process.\nb) How to ensure that the filtered dataset is representative and diverse for initializing actor and critic models.\nProviding these details will help readers better understand the experimental setup and improve the reproducibility of the research.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce DPSDP (Direct Policy Search by Dynamic Programming), a novel reinforcement learning algorithm that trains a multi-agent LLM system (actor and critic) to iteratively refine responses for reasoning tasks. They formulate the multi-turn refinement process as a Markov Decision Process and use direct preference learning on self-generated data to jointly optimize the actor and critic. A practical adaptation allows the model to generalize to out-of-distribution refinement horizons at test time. Theoretical analysis suggests DPSDP can achieve performance comparable to any comparator policy within the training distribution. Experiments on mathematical reasoning benchmarks demonstrate that DPSDP improves accuracy with multiple answer attempts. Reviewers highlight the theoretically grounded yet practical algorithm for multi-agent response refinement, the effective adaptation of PSDP to LLM agent training, and the consistent improvements across different models and benchmarks.

Reviewers identify several limitations. Some believe the theoretical analysis is not highly relevant to the practical algorithm due to modifications for feasibility. The baseline comparisons are sometimes limited, with baselines not implemented on all models. The method's intuition could be clearer, particularly regarding pairwise data collection. The paper primarily focuses on mathematical reasoning tasks, limiting the assessment of its capabilities in other domains. While the paper compares against SFT-based baselines, one reviewer suggests a more direct comparison with a fully trained SFT model under controlled training conditions.

For the final version of the paper, please: 
- Provide further analysis on how approximations in the practical algorithm affect the theoretical results.
- Include baseline implementations across all evaluated models for a more comprehensive comparison.
- Offer a clearer explanation of the method, especially the pairwise data collection and preference labeling process.
- Further investigate the stopping criterion for the refinement process to mitigate potential over-refinement.
- Analyze failure patterns in more detail to understand the types of errors the method effectively corrects and where it struggles.

Overall though the strengths outweigh the weaknesses and I recommend acceptance. The paper presents a novel and theoretically grounded approach to improving LLM reasoning through a multi-agent reinforcement learning framework. The empirical results demonstrate promising performance gains on challenging mathematical reasoning tasks, showcasing the potential of the method. The algorithm's ability to generalize to out-of-distribution refinement iterations is a significant strength. Furthermore, the reviewers generally acknowledge the sound methodology and the meaningful contribution to the field of LLM reasoning and self-improvement. The authors' efforts to address reviewer concerns in the rebuttal, including providing additional experimental results and clarifications, further strengthen the case for acceptance. It is important to improve the paper either way, incorporating the suggestions from reviewers.
Decision: Accept (poster)